Geometry, Morphology, and Computational Imaging

Lecture Notes in Computer Science Edited by G. Goos, J. Hartmanis, and J. van Leeuwen 2616 3 Berlin Heidelberg New Y...

Author: Tetsuo Asano | Reinhard Klette | Christian Ronse

63 downloads 1934 Views 9MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Computer Science Edited by G. Goos, J. Hartmanis, and J. van Leeuwen

2616

3

Berlin Heidelberg New York Barcelona Hong Kong London Milan Paris Tokyo

Tetsuo Asano Reinhard Klette Christian Ronse (Eds.)

Geometry, Morphology, and Computational Imaging 11th International Workshop on Theoretical Foundations of Computer Vision Dagstuhl Castle, Germany, April 7-12, 2002 Revised Papers

13

Series Editors Gerhard Goos, Karlsruhe University, Germany Juris Hartmanis, Cornell University, NY, USA Jan van Leeuwen, Utrecht University, The Netherlands Volume Editors Tetsuo Asano JAIST, School of Information Science 1-1 Asahidai, Tatsunokuchi, Ishikawa, 923-1202, Japan E-mail: [email protected] Reinhard Klette University of Auckland, Computer Science Dept. and CITR Tamaki Campus, Glen Innes, Auckland 1005, New Zealand E-mail: [email protected] Chrisitan Ronse LSIIT UMR 7005 CNRS-ULP Parc d’Innovation, Boulevard Sebastien Brant, BP 10413 67412 Illkirch, France E-mail: [email protected]

Cataloging-in-Publication Data applied for A catalog record for this book is available from the Library of Congress Bibliographic information published by Die Deutsche Bibliothek Die Deutsche Bibliothek lists this publication in the Deutsche Nationalbibliografie; detailed bibliographic data is available in the Internet at .

CR Subject Classification (1998): I.4, I.3.5, I.3.7 ISSN 0302-9743 ISBN 3-540-00916-7 Springer-Verlag Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecution under the German Copyright Law. Springer-Verlag Berlin Heidelberg New York a member of BertelsmannSpringer Science+Business Media GmbH http://www.springer.de © Springer-Verlag Berlin Heidelberg 2003 Printed in Germany Typesetting: Camera-ready by author, data conversion DA-TeX Gerd Blumenstein Printed on acid-free paper SPIN 10872912 06/3142 543210

Preface

There are works which wait, and which one does not understand for a long time; the reason is that they bring answers to questions which have not yet been raised; for the question often arrives a terribly long time after the answer. – Oscar Wilde This edited volume is based on talks and discussions at the 11th Workshop on Theoretical Foundations of Computer Vision, entitled “Geometry, Morphology, and Computational Imaging,” which took place April 07–12, 2002, at Schloss Dagstuhl in Germany. The central subject of all contributions is the digital picture, representing real objects in a discrete or digitized form, which have to be printed, visualized, processed, analyzed, transformed, etc. The aspect of computational efficiency is always of great relevance due to the massive amount of data in a single digital picture. Therefore we have decided to focus the workshop and the ensuing volume on issues pertaining to the geometry and shape of objects in images, and efficient computational structures and algorithms for extracting and representing them. The contributions are of relevance for computer graphics, computer vision, image analysis, visualization, etc. — digital imaging, in short. Researchers in digital geometry, mathematical morphology and computational geometry came together at this meeting, and the finally published papers were influenced not only by this meeting but also by a careful reviewing process. The editors decided to group all chapters into the following four parts. Part I: Geometry: Models and Algorithms The first part of this book is on geometric treatments of images. Kokichi Sugihara reconsiders in Chap. 1 the Minkowski algebra for figures and proposes a new formulation under the name of “hyperfigures” together with a physical interpretation. Peter Veelaert studies in Chap. 2 uncertain geometry in the plane where points are replaced by uncertainty regions, and proposes an algorithm for reestablishing global consistency of uncertain geometric relations in digital images. In Chap. 3, Atsushi Imiya et al. define a medial axis of a random point set in low dimensions and develop an algorithm for extracting curve and surface skeletons. Mirela T˘ anase and Remco Veltkamp propose in Chap. 4 a novel type of decomposition for polygonal shapes based on the straight line skeleton. Patrick Baker shows in Chap. 5 how lines rather than points can be used as the basic atoms of computer vision based on a prismatic line constraint based on reconstruction of local shape using line measurements and rotation only. The last two chapters in Part I consider combinatorial aspects of digital halftoning. Tetsuo Asano et al. discuss several problems related to combinatorial and computational geometry and provide some algorithms. Kunihiko Sadakane

VI

Preface

et al. propose an efficient evaluation of a halftoned image based on a discrepancy measure. They also propose algorithms to minimize the discrepancy measure. Part II: Property Measurements in the Grid and on Finite Samples The second part combines two papers on triangulated surfaces, two on length and surface area estimation (with special attention given to convergence rates), and two papers on approximating borders of sets of grid points in 2D or 3D space. Lars Linsen and Hartmut Prautzsch consider in Chap. 8 local triangulations (fan clouds) as an efficient alternative to more costly triangular mesh generations for real-time rendering of complex three-dimensional scenes. Emanuele Danovaro et al. discuss in Chap. 9 discrete gradient fields defined on triangular meshes in 3D space, which leads to decompositions of scalar fields defined on simplicial complexes generalizing the watershed transform originally introduced for 2D morphological image segmentations. Chapter 10 by David Coeurjolly et al. shows that global estimates of discrete normals support a method of multigrid-convergent surface area estimation for 3D objects given as subsets of the regular orthogonal grid (e.g., voxel data in 3D image analysis). Ryszard Kozera et al. assume discrete sampling of curves in Chap. 11, and study piecewise Lagrange interpolation with respect to the convergence rate of length estimators (and also of trajectories) in dependence of the distribution of samples on the given curve. Akihiko Torii et al. discuss in Chap. 12 non-linear approximations of borders of connected sets of points in 2D or 3D regular orthogonal grids based on curvature analysis: the approximated shapes also allow a digitization at a higher resolution than that given in the original image data, and experiments illustrate such a resolution conversion. Isabelle Sivignon and David Coeurjolly approximate in Chap. 13 a connected set of points in the 3D regular orthogonal grid by a Euclidean polyhedron, following the optimality criterion of minimizing the number of vertices. Part III: Features, Shape and Morphology The third part of this book is devoted to the analysis of features and shapes in images. Such concerns can be categorized by a general name, morphology, which deals with various notions intermediate between the well-known worlds of topology and geometry, and in some sense it contains both. A special branch of this field of research is the highly formalized theory of mathematical morphology, with its algebraic structure of operators based on orders and lattices, which is represented here by four papers. Chapter 14 by Christian Ronse deals with the so-called “flat operators” for grey-level images, and generalizes them to images where the numerically ordered grey-levels can be replaced by arbitrarily ordered elements, like colors, region labels, etc. In Chap. 15, Isabelle Bloch applies morphological operators to models in modal logic, and shows how new logical operations can be built in this way, with possible applications in spatial reasoning. Chapter 16 by Jos Roerdink proposes a model of multiresolution pyramids based on morphological operations, which

Preface

VII

generalizes previous constructions. In Chap. 17, Ulrich Eckhardt studies the invariants of the median filter (which is in fact a “flat” morphological operator) by a combination of lattice-theoretical and topological methods; in this way he extends and corrects previous studies on this subject. Mathematical morphology is not only a theory, it has many concrete applications. Akira Asano et al. devote Chap. 18 to the analysis of textures by the interaction of the image with a primitive shape taken at various sizes (the socalled “granulometry” or “morphological size distribution”); the shape is chosen by simulated annealing. Another approach to texture analysis is probabilistic, based on Gibbs random fields. Chapter 19 by Georgy Gimel’farb et al. belongs to this approach, it shows how to extract basic tiles forming texels (texture elements) from such a random field. The topological structuration of features like edges is important both in image analysis and synthesis. In Chap. 20, Ullrich K¨ othe does this by introducing a combinatorial topological structure called the XPMap (extended planar map), which he applies to existing segmentation algorithms (edge detection, watershed, etc.). Part IV: Computer Vision and Scene Analysis The final part in this volume starts with two papers “at signal level,” discussing edge enhancement and Gaussian noise in an image. Corners in 2D images and line segments in 3D images are the subject of the next two chapters, followed by two contributions studying motion in a video sequence. The final chapter specifies a calibration method for panoramic cameras based on distributions of line segments in 3D space. PeiFeng Zeng and Tomiro Hirata study in Chap. 21 the application of distance transforms for image enhancement: after calculating “transient areas” (which are defined by distances to edge segments), image values are adjusted for the purpose of contrast enhancement. Chapter 22 by Lyle Noakes and Ryszard Kozera discusses the impact of Gaussian image noise on shape recovery, assuming that a photometric stereo method is applied. Hidekata Hontani and Kichiro Deguchi identify in Chap. 23 “dominant corners” in 2D polygonal contour figures, applying a scale-space method based on curvature flow. Chapter 24 by Yasushi Hirano et al. proposes two Voronoidiagram-based features (degree of concentration, and the difference of density) allowing us to analyze spatial distributions of line segments in 3D space (e.g., occurring in the context of 3D medical imaging). In Chap. 25, Daniel Keren et al. present a Bayes-based maximum-likelihood estimate for motion recovery between two successive frames of a video sequence, where special attention is paid to the recovery of the focus-of-expansion. The proposed global method leads to improved results, but incurs an increase in computational complexity. John Barron et al. consider a calculated optical flow field for successive frames, and apply in Chap. 26 a Kalman filter framework to depth recovery if camera translation can be assumed. Chapter 27 is devoted to a recently developed architecture of panoramic cameras: a line-sensor rotates around an axis of revolution (radius R), with a

VIII

Preface

constant viewing angle ω. It is shown that triples of line segments in a scene can be used for calibrating R and ω. The editors thank all reviewers for their detailed responses and the authors for efficient collaboration in ensuring a high-quality publication. Beside the editors, the reviewers were: Akira Asano John Barron Isabelle Bloch Gunilla Borgefors David Coeurjolly Alain Daurat Koichiro Deguchi Ulrich Eckhardt Georgy Gimel’farb Yasushi Hirano Tomio Hirata Atsushi Imiya Naoki Katoh Yukiko Kenmochi Daniel Keren Nahum Kiryati

Ulrich K¨ othe Yung Kong Birgit Loch Jan Neumann Naoya Ohta Helene Reiter Jos Roerdink Isabelle Sivignon Gerald Sommer Kokichi Sugihara Mohamed Tajine Takeshi Tokuyama Peter Veelaert Remco Veltcamp Michael Werman Guy Windreich

The editors also thank the International Conference and Research Center for Computer Science at Dagstuhl for providing an excellent location and service for discussing science, and the staff of the Springer-Verlag LNCS office for cooperation in finalizing this edited volume.

December 2002

Tetsuo Asano, Reinhard Klette, Christian Ronse

Table of Contents

Flat Morphological Operators on Arbitrary Power Lattices . . . . . . . . . . . . . . . . . . 1 Christian Ronse Quantification of the Spatial Distribution of Line Segments with Applications to CAD of Chest X-Ray CT Images . . . . . . . . . . . . . . . . . . . . . 22 Yasushi Hirano, Yoshito Mekada, Jun-ichi Hasegawa, and Junichiro Toriwaki Fan Clouds – An Alternative to Meshes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Lars Linsen and Hartmut Prautzsch Combinatorial and Geometric Problems Related to Digital Halftoning . . . . . . 58 Tetsuo Asano, Naoki Katoh, Koji Obokata, and Takeshi Tokuyama All Points Considered: A Maximum Likelihood Method for Motion Recovery . . . . . . . . . . . . . . . . . . . . . . 72 Daniel Keren, Ilan Shimshoni, Liran Goshen, and Michael Werman Distance Map Based Enhancement for Interpolated Images . . . . . . . . . . . . . . . . .86 PeiFeng Zeng and Tomio Hirata Multigrid Convergence and Surface Area Estimation . . . . . . . . . . . . . . . . . . . . . . 101 David Coeurjolly, Frédéric Flin, Olivier Teytaud, and Laure Tougne Geometric Structure and Randomness in Texture Analysis and Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .120 Georgy Gimel’farb, Linjiang Yu, and Dongxiao Zhou Morphological Texture Analysis Using Optimization of Structuring Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 Akira Asano, Miho Miyagawa, and Mitsuhiko Fujio Unifying Quantitative, Semi-quantitative and Qualitative Spatial Relation Knowledge Representations Using Mathematical Morphology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .153 Isabelle Bloch A New Class of Morphological Pyramids for Multiresolution Image Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 Jos B.T.M. Roerdink Root Images of Median Filters – Semi-topological Approach . . . . . . . . . . . . . . 176 Ulrich Eckhardt Medial Set, Boundary, and Topology of Random Point Sets . . . . . . . . . . . . . . . 196 A. Imiya, H. Ootani, and K. Tatara

X

Table of Contents

Scale-Based Corner Extraction of a Contour Figure Using a Crystalline Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .218 Hidekata Hontani and Koichiro Deguchi Hyperfigures and Their Interpretations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 Kokichi Sugihara Polygon Decomposition Based on the Straight Line Skeleton . . . . . . . . . . . . . . 247 Mirela T˘ anase and Remco C. Veltkamp Reestablishing Consistency of Uncertain Geometric Relations in Digital Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268 Peter Veelaert Inverse Quantization for Resolution Conversion . . . . . . . . . . . . . . . . . . . . . . . . . . . 282 A. Torii, T. Ichinose, Y. Wakazono, and A. Imiya Discrepancy-Based Digital Halftoning: Automatic Evaluation and Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 Kunihiko Sadakane, Nadia Takki Chebihi, and Takeshi Tokuyama Deriving Topological Representations from Edge Images . . . . . . . . . . . . . . . . . . 320 Ullrich K¨ othe Calibration of Panoramic Cameras Using 3D Scene Information . . . . . . . . . . . 335 Fay Huang, Shou Kang Wei, and Reinhard Klette Quantitative Depth Recovery from Time-Varying Optical Flow in a Kalman Filter Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346 John Barron, Wang Kay Jacky Ngai, and Hagen Spies From Digital Plane Segmentation to Polyhedral Representation . . . . . . . . . . . 356 Isabelle Sivignon and David Coeurjolly Lines as the Fundamental Unit of Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368 Patrick Baker Topological Analysis and Characterization of Discrete Scalar Fields . . . . . . . 386 Emanuele Danovaro, Leila De Floriani, and Mohammed Mostefa Mesmoudi External versus Internal Parameterizations for Lengths of Curves with Nonuniform Samplings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 Ryszard Kozera, Lyle Noakes, and Reinhard Klette Denoising Images: Non-linear Leap-Frog for Shape and Light-Source Recovery . . . . . . . . . . . . . . . 419 Lyle Noakes and Ryszard Kozera Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .437

Flat Morphological Operators on Arbitrary Power Lattices Christian Ronse LSIIT UMR 7005 CNRS-ULP Boulevard Sébastien Brant, 67400 Illkirch, France [email protected] http://picabia.u-strasbg.fr/lsiit/

Abstract. We give here the basis for a general theory of flat morphological operators for functions defined on a space E and taking their values in an arbitrary complete lattice V . Contrarily to Heijmans [4, 6], we make no assumption on the complete lattice V , and in contrast with Serra [18], we rely exclusively on the usual construction of flat operators by thresholding and stacking. Some known properties of flat operators for numerical functions (V = Z or R) extend to this general framework: flat dilations and erosions, flat extension of a union of operators or of a composition of an operator by a dilation. Others don’t, unless V is completely distributive: flat extension of an intersection or of a composition of operators; for these we give counterexamples with V being the non-distributive lattice of labels. In another paper [15], we will consider the commutation of flat operators with anamorphoses (contrast functions) and thresholdings, duality by inversion, as well as related questions of continuity.

1

Introduction

Most morphological operators on grey-level images are flat operators. This means [4] that they are grey-level extensions of operators for binary images, and they can be obtained by: a) thresholding the grey-level image for all threshold values, b) applying the binary operator to each thresholded image set, and c) superposing the resulting sets. They have been studied in detail [4, 5, 6, 11] in the case of grey-level images considered as numerical functions E → T where T = R = R ∪{+∞, −∞} or Z = Z∪{+∞, −∞}, or more generally T is a closed subset of R (for example, T = [a, b] for a, b ∈ R). Several mathematical descriptions have been provided for them, for example in terms of operators relying only on grey-level comparisons [13], and an alternate construction was given for them [19] (in the case where T = {0, . . . , n}), using arithmetical sum instead of superposition in step c) above. Flat operators share some fundamental properties [6], however it is known that there are variations between the two cases where the set of grey-levels is discrete (T ⊆ Z), and where it is analog (T ⊆ R). Indeed, for the commutation of flat operators with increasing grey-level transformations, some continuity T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 1–21, 2003. c Springer-Verlag Berlin Heidelberg 2003

2

Christian Ronse

constraints (either on the grey-level transformation or on the originating binary operator) must be introduced in the case of analog grey-levels. We will discuss this problem in a general context in [15]. More problems arise when we consider other types of images: The first extension is towards colour images, where the colour is coded in 3 RGB, and the space of colours is thus considered as a subset of R . Flat operators can then be defined in the usual way; in fact they can be implemented by applying the usual (grey-level) flat operator to each of the three R, G and B components of the image (this follows from Proposition 1 below). However, as pointed out by Serra [18], we lose some properties, even for discrete RGB coulours, in particular the commutation with thresholding. Second, in order to process video sequences, Keshet (Kresch) introduced [9, 10] the reference order on grey-level images. Take a closed grey-level set T ⊆ R. Choosing a reference grey-level r, we define the reference order ≤r w.r.t. r as follows: for two grey-levels a, b we have a ≤r b if a is between r and b, i.e., if either r ≤ a ≤ bor r ≥ a ≥ b (numerically); then w.r.t. that reference r order, the infimum i∈I ai of a family ai (i ∈ I) of grey-levels will be equal to the numerical median of their numerical infimum inf i∈I ai , their numerical supremum supi∈I ai , and r. This ordering on the set T of grey-levels turns it into a complete inf-semilattice, in other words every nonvoid family of grey-levels has an infimum, but not necessarily a supremum. Given a space E of points, the set T E of functions E → T can be ordered with reference to a fixed function R ∈ T E : for every point p ∈ E, grey-levels F (p), G(p) of functions F, G ∈ T E are ordered with reference to R(p), and so T E is a complete inf-semilattice, called the reference semi-lattice. This semilattice has been studied in detail in [7]. By adding a greatest element ∞ to T , T = T ∪{∞} becomes a complete lattice (and E then T is also a complete lattice with reference to R). Note that the complete E lattices T and T are not distributive. Can we define flat operators on T E or E on T , and in this case what are their properties? Third, in an algorithm for segmenting video sequences, Agnus [1, 2] defined “object-oriented” variants of flat erosion and geodesical reconstruction, where grey-levels are considered as labels of objects, and all image portions having distinct grey-levels are processed independently. In fact, as we will explain in [16], these “object-oriented” variants of flat morphological operators are simply the standard flat operators in the case where grey-levels belong to a lattice U of labels without ordering between them, except for a least element ⊥ (meaning no label) and a greatest element (meaning conflicting labels), see Fig. 1. We will briefly discuss this lattice U later, because it will appear in several counterexamples. These three examples show the need of a general theory of flat morphological operators on the power lattice V E of functions E → V , where E is a space of points and V is an arbitrary complete lattice of image values. However, with the notable exception of [18] and Chap. 10 of [6], the theory of flat morphological operators on non-binary images has been restricted to the case of grey-level images having grey-level values in a subset of Z or R.

Flat Morphological Operators on Arbitrary Power Lattices

3

In Chap. 10 and 11 of [6], Heijmans defines flat operators through the usual operations of thresholding and stacking. Let E be the space of points and let V be a complete lattice of image values. Consider a function F : E → V . For every v ∈ V , we define the threshold set Xv (F ) = {p ∈ E | F (p) ≥ v} .

(1)

Then, given an operator ψ : P(E) → P(E) on sets, the flat operator corresponding to ψ is defined as the operator ψ V : V E → V E on functions defined by setting for any function F and point p: (2) ψ V (F )(p) = {v ∈ V | p ∈ ψ (Xv (F ))} . However Heijmans makes some drastic assumptions on the lattice V of values (see in [6] Definition 10.4 of an admissible and of a strongly admissible complete lattice). We will provide a detailed analysis of these assumptions in [15], but we can already say a few words here: it is easy to show that every admissible complete lattice is completely distributive, and in fact all examples of admissible lattices given by Heijmans are products of complete chains; moreover, a product of two or more complete chains (e.g., the lattice of RGB colours) is never strongly admissible. Thus all results of [6] are restricted to a very particular case for the lattice V of values; in practice, V will have to be a complete chain. On the other hand, in an attempt to obtain in a general framework the fundamental property that flat operators commute with continuous increasing transformations on the lattice of values, Serra [18] takes an arbitrary lattice V of values, but in the definition (1) of thresholding he replaces the order ≥ by the negation ≤ of the dual order ≤: ≤ v} . Yv (F ) = {p ∈ E | F (p) When V is a chain (i.e., totally ordered), which is the case for subsets of R or Z, the relation ≤ means >. With such a definition of thresholding, the above definition (2) of the flat operator ψ V must also be modified. In this paper, we will initiate a general theory of flat operators on functions E → V , where V is an arbitrary lattice of values, using the classical construction of (1,2). As we will see, some known results for grey-level functions in R or Z extend to the general case, while some others require assumptions on the lattice V of values, like infinite or complete distributivity. The paper is organized as follows. Section 2 is a reminder of advanced latticetheoretical concepts and terminology for mathematical morphology; it introduces also our notation. Section 3 defines flat operators and gives their main properties. The Conclusion summarizes our results and links them to further perspectives. Due to lack of space, we do not deal here with the commutation of flat operators with contrast functions (i.e., increasing mappings V → V ), nor with duality under an inversion (anti-isomorphism) of V . This will be the subject of [15]. We will see there that this requires the contrast function to be “continuous” in some sense, or the set operator to be “semi-continuous” in a sense which depends on the structure of V .

4

2

Christian Ronse

Lattices, Distributivity and Images

We assume that the reader is familiarized with the notions of partial order, poset (partially ordered set), lattice, complete lattice, and power lattice, used in mathematical morphology. See Subsection 1.3 of [8] for a brief overview. Chapter 2 of [6] gives a broader exposition. A standard reference on posets and lattices is [3]. We will recall here some classical concepts not dealt with in [8] (but which are described in [3, 6]), and give our notation. We will write L for an abstract lattice, ≤ for its order, and a, b, c, . . . for its elements. The order ≤ is total if for every a, b ∈ L, we have always a ≤ b or b ≤ a; we say then that L is a chain, and we call a complete chain a totally ordered complete lattice. We will use later the following standard identity: =⇒ Xi = Xi . (3) Xi ∈ P(L) (i ∈ I) i∈I

i∈I

A lattice L is distributive [3, 6] if for any elements a, b, c of L we have a ∧ (b ∨ c) = (a ∧ b) ∨ (a ∧ c) . It is known [3] that this is equivalent to requiring for any elements x, y, z of L: x ∨ (y ∧ z) = (x ∨ y) ∧ (x ∨ z) . There are several generalizations of distributivity for complete lattices [3, 6]. Two of them are often used in morphology [6], namely infinite supremum distributivity (in brief, ISD): a∧ bi = (a ∧ bi ) , (4) i∈I

i∈I

for any nonempty index set I, and its dual, infinite infimum distributivity (in brief, IID): a∨ bi = (a ∨ bi ) . (5) i∈I

i∈I

Note that ISD implies that for any integer n > 1 we have n i=1 j∈Ji

ai,j =

a1,j1 ∧ · · · ∧ an,jn =

(j1 ,...,jn ) ∈J1 ×···×Jn

n

(j1 ,...,jn ) ∈J1 ×···×Jn

i=1

ai,ji .

(6)

The dual formula (with ∨ and ∧ interverted) holds for IID. There is a stronger distributivity law called extended distributivity or complete distributivity [3, 6]; it deals with distributivity between infinite suprema and infinite infima. The complete lattice L is completely distributive if given an index set I, a family of index sets Ji for each i ∈ I, and ai,j ∈ L (i ∈ I, j ∈ Ji ), we have ai,j = ai,ϕ(i) , (7) i∈I j∈Ji

ϕ∈Φ(I) i∈I


5

where Φ(I) is the set of choice maps associating to each i ∈ I an element of Ji . It has been shown in [12] to be equivalent to requiring bi,j = bi,ϕ(i) (8) i∈I j∈Ji

ϕ∈Φ(I) i∈I

for all bi,j ∈ L. There is an alternative way of expressing (7,8) without index sets. Given a family X of parts of L, a choice map on X is a map ϕ : X → X such that for any X ∈ X we must have ϕ(X) ∈ X; let Φ(X ) be the set of choice maps on X ; then (7) can be written as ϕ(X) , (9) X= ∀X ⊆ P(L), X∈X

ϕ∈Φ(X ) X∈X

while (8) becomes

∀Y ⊆ P(L),

Y ∈Y

Y =

ϕ(Y ) .

(10)

ϕ∈Φ(Y) Y ∈Y

It is known [3] that every complete chain is completely distributive. The following will be used in the next section: Lemma 1. Let L be a completely distributive complete lattice. Then for every a ∈ L, there exists a nonempty Ga ⊆ L such that

1. a = Ga ; and

2. for every nonempty Y ⊆ L such that a = Y , and for every g ∈ Ga , there is some y ∈ Y such that g ≤ y.

Proof. Set Xa = {Y ∈ P(L) \ {∅} | Y = a}; as {a} ∈ Xa , Xa = ∅, and Y =∅ for all Y ∈ Xa . Let Φa be the set of

choice functions ϕ on Xa , associating to each Y ∈ Xa some ϕ(Y ) ∈ Y . As a = Y for Y ∈ Xa , the extended distributivity law (7) gives Y = ϕ(Y ) . a = Y ∈Xa

ϕ∈Φa Y ∈Xa

For

each ϕ ∈ Φa , let gϕ = Y ∈Xa ϕ(Y ), and let Ga = {gϕ | ϕ ∈ Φa }. Then a = Ga , that is item 1. Let Y ∈ Xa ; for any ϕ ∈ Φa , we have gϕ ≤ ϕ(Y ) ∈ Y , that is item 2. Consider a space E of points (which can in fact be any set); the parts of E are written X, Y, Z . . ., they form the complete lattice P(E). We consider an arbitrary complete lattice V of image values, whose elements are written v, w, . . ., except the least and greatest elements, which will be written ⊥ and respectively. Images will then be functions E → V , we call such functions V -images, and we usually write them F, G, H, . . .. The power lattice V E is the complete lattice of V -images E → V , with the ordering F ≤ G iff F (x) ≤ G(x) for all x ∈ E. Note that V E satisfies the same distributivity laws as V ; in particular if V is

6

Christian Ronse

a complete chain, then V E is completely distributive. Image processing transformations are called operators; in the case of binary images, they correspond to maps P(E) → P(E), while in the case of V -images, they are considered as maps V E → V E ; they are written by lower-case Greek letters ψ, ξ, etc., except for the identity operator written id. We reserve the symbols T and U for two peculiar lattices of image values, which we describe now. Here T will denote a subset of R = R∪{−∞, +∞} which is closed under the supremum and infimum operations (or equivalently, T is topologically closed). In mathematical morphology [6], grey-level images are usually considered as numerical functions E → T , where E = Rn , Zn or a bounded part of them, and the complete lattice T of grey-levels is generally taken to be R, [a, b] = {x ∈ R | a ≤ x ≤ b} (with a, b ∈ R and a < b), Z = Z ∪ {−∞, +∞}, or [a, b] ∩ Z. Note that the binary lattice 2 = {0, 1} and the numerical lattice T of greylevels are complete chains; hence their power lattices 2E (isomorphic to P(E)) and T E will be completely distributive. Consider a nonvoid family U∗ of symbols, which are supposed to represent labels or tags for image regions. Elements of U∗ are called proper labels. We add to them two dummy labels ⊥ and whose semantic meanings are respectively “no label” and “conflicting labels”. The lattice of labels is then built as the set U = U∗ ∪ {⊥, } on which we define an order relation ≤ as follows: ∀u ∈ U, ∀u ∈ U∗ , ∀u ∈ U∗ ,

u ≤ u ⊥≤ ⊥≤ u u ≤

; ; ; .

(11)

In other words two distinct elements of U∗ (proper labels) are not comparable for the order: we never have u < u for u, u ∈ U∗ . It is easily seen that U is a complete lattice for the partial order relation ≤. We can graphically represent this order by its Hasse diagram [3, 6], namely a directed graph whose vertices are the elements of U and whose arrows represent the covering [3] relation (a b if a > b but there is no m such that a > m > b). See Fig. 1. Note that for two distinct u, u ∈ U∗ , we have always u ∨ u = and u ∧ u = ⊥. Clearly: – if |U∗ | = 1, U is a chain; – if |U∗ | = 2, U is Boolean; – if |U∗ | > 2, U is not distributive. In the sequel, we will always assume that |U∗ | > 2. As we will see later in some examples, the application of flat morphological operators to an U -image often amounts to processing independently all zones having a given proper label. See [16] for further details. Let us now introduce some general notation. For every v ∈ V , write Cv for the function E → V with constant value v: ∀p ∈ E,

Cv (p) = v .


7

Fig. 1. Graph of the covering relation on U For any B ⊆ E and v ∈ V , the cylinder of base B and level v is the function CB,v defined by v if p ∈ B, ∀p ∈ E, CB,v (p) = (12) ⊥ if p ∈ / B. Note in particular that Cv = CE,v . Also, for h ∈ E and v ∈ V , the impulse ih,v is the cylinder C{h},v , thus v if p = h, ∀p ∈ E, ih,v = (13) ⊥ if p = h.

3

Flat Operators

In this Section, we will show how the construction of increasing flat operators on grey-level images given in [4] and Chap. 10 and 11 of [6] (see (1,2) above), can be extended to V -images, and retain some of their fundamental properties. These properties will be obtained without any requirement on the complete lattice V of values, so they are valid for colour images, label images, images with values in the completed reference lattice, etc. On the other hand, some properties do not hold unless we assume the lattice V to be infinitely or completely distributive (which is true for sets, numerical and multivalued functions, i.e., for binary, grey-level and RGB colour images). Throughout this section, we restrict ourselves to increasing set operators (and the flat operators derived from them will also be increasing). Indeed, the standard construction for flat operators does not work correctly for non-increasing operators (see [4]). We consider V -images E → V , where E is a space of points and V is an arbitrary complete lattice of image values. We introduce now our main concepts.

8

Christian Ronse

Definition 1. Let W ⊆ V . 1. A stack on W is a decreasing map Ξ : W → P(E), in other words it associates to every v ∈ W a subset Ξ(v) of E, and for v, w ∈ W with v < w we have Ξ(w) ⊆ Ξ(v). 2. Given a stack Ξ on W and an increasing operator ψ on P(E), ψΞ is the stack on W defined by ψΞ(v) = ψ (Ξ(v)) 3. Given a stack Ξ on W , the superposition of Ξ is the V -image ΣΞ defined by CΞ(v),v ; (14) ΣΞ = v∈W

in other words, for every point p ∈ E we have ΣΞ(p) = {v ∈ W | p ∈ Ξ(v)} .

(15)

For a V -image F (a function F : E → V ) and a value v ∈ V , we define the threshold set Xv (F ) as in (1) above: ∀v ∈ V,

Xv (F ) = {p ∈ E | F (p) ≥ v} .

(16)

It is easily seen that Xv (F ) increases with F (that is, F ≤ G implies Xv (F ) ≤ Xv (G)), but decreases with v: v ≤ w =⇒ Xw (F ) ⊆ Xv (F ) . This equation can be generalized as follows: for W ⊆ V , w= W =⇒ Xw (F ) = Xv (F ) .

(17)

(18)

v∈W

By (17), the threshold sets will make a stack. We can now construct flat operators on V E as in [4] (see (2) above): Definition 2. Let ψ be an increasing operator on P(E). 1. Given a V -image F , the threshold stack of F is the stack Θ[F ] given by ∀v ∈ V,

Θ[F ](v) = Xv (F ) ;

for W ⊆ V , the threshold stack of F on W is the restriction Θ[F ]W of Θ[F ] to W . 2. The flat operator corresponding to ψ is the operator ψ V : V E → V E on V images, defined by setting for any V -image F : ψ V (F ) = ΣψΘ[F ] ;

(19)


9

in other words (see (14)), ψ V (F ) =

Cψ(Xv (F )),v ,

(20)

v∈V

so that for every point p ∈ E we have by (15): ψ V (F )(p) = {v ∈ V | p ∈ ψ (Xv (F ))} .

(21)

This latter equation (21) is (up to a change of notation) the same formula as in Sect. 5 of [4]. Intuitively, this means that we apply ψ to each horizontal cross-section of the umbra U (F ) = {(h, v) | h ∈ E, v ∈ V, v ≤ F (h)} of the function F , and take the upper envelope of the modified umbra. We illustrate this here in Fig. 2 for a two-valued function. In Definition 2 we have taken threshold sets Xv (F ) for all values v ∈ V . In fact, some values of v are not necessary for this purpose: – The value v = ⊥ is redundant in all the above formulas. – For T = R or Z, it has been stressed in [14] that we should not take into account infinite grey-levels t = ±∞ for the umbra U (F ) = {(h, t) | t ≤ F (h)}, the impulses ih,t (see (13)), and the threshold sets Xt (F ) (see (16)). Thus we can restrict such formulas to t ∈ T = T \ {±∞}. Indeed, ⊥ = −∞ is redundant anyway, but we have also +∞ = sup T , so that X+∞ (F ) =

t∈T Xt (F ) by (18). Now grey-level computations are easier without ±∞. – Similarly in the lattice U of labels, we can consider that the dummy labels ⊥ and are redundant in the formulas, since ⊥ = sup ∅ and = sup U∗ , and consider thresholds Xu (F ) only for u ∈ U∗ . – For RGB colour images, the flat extension of a set operator can be constructed by applying the grey-level flat operator to each of the red, green,

T

T

(a)

a1

a1 F

a0

(b) ψ (O) / ψ (A)

A

a0

E

ψ V (F)

ψ (E) E

Fig. 2. (a) The function F = Ca0 ∨ CA,a1 , where a0 < a1 . (b) The function ψ V (F ) is obtained by applying ψ to the horizontal cross-sections of the umbra of F (shown in light grey), and superposing the results. We get ψ V (F ) = Cψ(E),a0 ∨ Cψ(A),a1 ∨ Cψ(∅),

10

Christian Ronse

and blue components of the image, and joining the results. This amounts to considering only red (r, ⊥, ⊥), green (⊥, g, ⊥), and blue (⊥, ⊥, b) threshold values v for the threshold sets Xv (F ). We select thus a sup-generating subset W of V . This means that every v ∈ V is the supremum of some subset Wv of W ; in fact v = sup{w ∈ W | w ≤ v}. For example: – for V = T = R, W = T = R (we can even take W = Q); – for V = T = Z, W = T = Z; – for V = U , W = U∗ . To validate this choice, we show that if we take thresholds for values in W instead of V , we get the same result: Proposition 1. Let W be a sup-generating subset of V . Then for any increasing operator ψ on P(E) and any V -image F , ψ V (F ) = ΣψΘ[F ]W , in other words (see (20)) Cψ(Xw (F )),w , (22) ψ V (F ) = w∈W

so that for every point p ∈ E we have (see (21)) ψ V (F )(p) = {w ∈ W | p ∈ ψ (Xw (F ))} .

(23)

Proof. By (20), ψ V (F ) = v∈V Cψ(Xv (F )),v . Let G = w∈W Cψ(Xw (F )),w . As W ⊆ V , we have G ≤ ψ V (F ). Now

let v ∈ V ; if v = ⊥ we have Cψ(Xv (F )),v = = ⊥, we have v = i∈I wi for a nonvoid family wi , i ∈ I; we C⊥ ≤ G. For v have then for each i ∈ I: v ≥ wi , so that by (17) we get Xv (F ) ⊆ Xwi (F ), and as ψ is increasing, ψ (Xv (F )) ⊆ ψ (Xwi (F )); hence Cψ(Xv (F )),wi ≤ Cψ(Xw (F )),wi ≤ G . Cψ(Xv (F )),v = i i∈I

i∈I

Therefore ψ V (F ) ≤ G, and we deduce the equality (22), which means precisely that ψ V (F ) = ΣψΘ[F ]W , and we derive then (23). We will now give the properties of flat operators on V -images. It is well-known in the case of images with discrete grey-levels (V ⊆ Z) that for an increasing set operator ψ verifying ψ(∅) = ∅ and ψ(E) = E, the flat operator ψ V will not add new grey-levels in an image; on the other hand, for continuous grey-levels, it can introduce as new grey-levels only limits of existing grey-levels in the original image. When ψ(E) = E (resp., ψ(∅) = ∅), the flat operator introduces in the image the value ⊥ (resp., ). Here in the general case, new values in ψ V (F ) can be obtained by combinations of suprema and infima of existing values F (q) (including the empty supremum ⊥ and the empty infimum ). We rely on the following preliminary result:


11

Lemma 2. Let F ∈ V E . For any v ∈ V , let s(v, F ) = {F (p) | p ∈ Xv (F )} . Then s(v, F ) ≥ v and Xs(v,F ) (F ) = Xv (F ). Proof. If Xv (F ) = ∅, then s(v, F ) = ∅ = ≥ v, and by (17), X (F ) ⊆ Xv (F ) = ∅, in other words Xs(v,F ) (F ) = Xv (F ). = ∅, as F (p) ≥ v for all p ∈ Xv (F ), and s(v, F ) is the infimum of all If Xv (F ) such F (p), we get s(v, F ) ≥ v. As s(v, F ) ≥ v, by (17), Xs(v,F ) (F ) ⊆ Xv (F ). Now for every p ∈ Xv (F ), we have F (p) ≥ s(v, F ) (because s(v, F ) is the infimum of all such F (p)), so p ∈ Xs(v,F ) (F ). We deduce the equality Xs(v,F ) (F ) = Xv (F ). A complete sublattice of V is a subset W of V which is a complete lattice with the same supremum and infimum operations as in V , in other words, which is closed under the supremum and

infimum operations. In particular, it must have the same empty supremum ∅ = ⊥ and infimum ∅ = as V . We show now that for an image with values in a complete sublattive W , a flat operator defined for V E gives the same result as that operator defined for W E : Proposition 2. Let W be a complete sublattice of V and let F be a W -image. Let ψ be an increasing operator on P(E). Then ψ V (F ) = ψ W (F ). Proof. By (20) we have ψ V (F ) = Cψ(Xv (F )),v v∈V

and

ψ W (F ) =

Cψ(Xw (F )),w .

w∈W

As W ⊆ V , we get ψ W (F ) ≤ ψ V (F ). For any v ∈ V , as s(v, F ) ≥ v and Xs(v,F ) (F ) = Xv (F ), we derive that Cψ(Xv (F )),v ≤ Cψ(Xs(v,F ) (F )),s(v,F ) ;

but F (p) ∈ W for all p ∈ E, so that s(v, F ) ∈ W , and we deduce that ψ V (F ) ≤ ψ W (F ). The equality follows. It means in particular that for any V -image F , the values of ψ V (F ) will belong to the complete sublattice W of V generated by the values of F . As an illustration of this result, we can describe the behaviour of a flat operator on a constant image or an image having two increasing values. The following is well-known for V = T : Proposition 3. Let ψ be an increasing operator on P(E). 1. If ψ(∅) = ∅, then for any a ∈ V and A ⊆ E, ψ V (CA,a ) = Cψ(A),a . 2. If ψ(∅) = ∅ and ψ(E) = E, then for any a ∈ V , ψ V (Ca ) = Ca . 3. Let a0 , a1 ∈ V such that a0 < a1 , let A ⊆ E, and let F = Ca0 ∨ CA,a1 , in other words: a1 if p ∈ A, ∀p ∈ E, F (p) = a0 if p ∈ / A. If ψ(∅) = ∅ and ψ(E) = E, then ψ V (F ) = Ca0 ∨ Cψ(A),a1 , that is: a1 if p ∈ ψ(A), V ∀p ∈ E, ψ (F )(p) = / ψ(A). a0 if p ∈

12

Christian Ronse

Proof. By Proposition 2, we have just to apply the formulas in the complete sublattice W generated by the values of F , and by Proposition 1, we can restrict the values in formulas to the sup-generating subset W = W \ {⊥}. Consider item 1: we have W = {⊥, a, } and W = {a, }. If a < , (22) gives ψ V (CA,a ) = Cψ(Xa (CA,a )),a ∨ Cψ(X (CA,a )), = Cψ(A),a ∨ Cψ(∅), = Cψ(A),a , while if a = it gives ψ V (CA,a ) = Cψ(Xa (CA,a )),a = Cψ(A),a . Consider item 2: we have also W = {⊥, a, } and W = {a, }. If a < , (22) gives ψ V (Ca ) = Cψ(Xa (Ca )),a ∨ Cψ(X (Ca )), = Cψ(E),a ∨ Cψ(∅), = CE,a = Ca , while if a = it gives ψ V (Ca ) = Cψ(Xa (Ca )),a = Cψ(E),a = CE,a = Ca . In item 3, we have W = {⊥, a0 , a1 , } and W = {a0 , a1 , }. If a1 < , (22) gives ψ V (F ) = Cψ(Xa (F )),a0 ∨ Cψ(Xa (F )),a1 ∨ Cψ(X (F )), 0 1 = Cψ(E),a0 ∨ Cψ(A),a1 ∨ Cψ(∅), = CE,a0 ∨ Cψ(A),a1 = Ca0 ∨ Cψ(A),a1 , while if a1 = it gives ψ V (F ) = Cψ(Xa (F )),a0 ∨ Cψ(Xa (F )),a1 = Cψ(E),a0 ∨ Cψ(A),a1 0 1 = CE,a0 ∨ Cψ(A),a1 = Ca0 ∨ Cψ(A),a1 . Thus when ψ satisfies the conditions ψ(∅) = ∅ and ψ(E) = E, ψ V preserves constant images, and transforms a binary image (with values a0 < a1 ) into a binary image (having the same values), in the same way as one knows for grey-level images. When these conditions ψ(∅) = ∅ and ψ(E) = E are not satisfied, the behaviour of ψ on a two-valued image is more complicated, as seen = E, then above in Fig. 2. It is easily shown that given any F ∈ V E , if ψ(E) ψ V (F )(p) = ⊥ for p ∈ E \ ψ(E), and if ψ(∅) = ∅, then ψ V (F )(p) = for p ∈ ψ(∅). The following result is an immediate consequence of the fact that Xv (F ) increases with F : Proposition 4. For an increasing operator ψ on P(E), ψ V is an increasing operator on V E . The following is also elementary: Proposition 5. idV is the identity on V E .


13

Proof. For F ∈ V E and p ∈ E, let Vp = {v ∈ V | p ∈ Xv (F )}; here (21) gives

idV (F )(p) = Vp . Now for v ∈ Vp , we have F (p) ≥ v, so we get F (p) ≥ Vp . On the other hand p ∈ Xv (F ) for v = F (p), that is F (p) ∈ Vp , so we get

F (p) ≤ Vp . Therefore F (p) = Vp = idV (F )(p). Let us now consider dilations, erosions, and adjunctions. We recall from [6, 8] that given a complet lattice L, a dilation on L is an operator δ : L → L which distributes the supremum, while an erosion is an operator ε : L → L which distributes the infimum: ∀xi ∈ L (i ∈ I), δ xi = δ(xi ) and ε xi = ε(xi ) . i∈I

i∈I

i∈I

i∈I

In particular δ fixes the least element of L (equal to the empty supremum), while ε fixes the greatest element of L (equal to the empty infimum). An adjunction is an ordered pair (ε, δ) of operators L → L such that ∀x, y ∈ L,

δ(x) ≤ y ⇐⇒ x ≤ ε(y) .

We have the following: – In an adjunction (ε, δ), ε is an erosion and δ is a dilation. – Given a dilation δ, there is a unique erosion ε such that (ε, δ) is an adjunction; ε is called the upper adjoint of δ, it is defined by setting for any y ∈ L: ε(y) is the greatest x ∈ L such that δ(x) ≤ y. – Given an erosion ε, there is a unique dilation δ such that (ε, δ) is an adjunction; δ is called the lower adjoint of ε, it is defined by setting for any x ∈ L: δ(x) is the least y ∈ L such that x ≤ ε(y). Let us deal now with the particular case of the lattice P(E) of parts of the space E. A neighbourhood function (or windowing function) is a map N : E → P(E) associating to every point p ∈ E a neighbourhood (or window) N (p) ⊆ E. defined by Given a windowing function N , its dual is the windowing function N ∀p, q ∈ E,

(p) ⇐⇒ p ∈ N (q) . q∈N

The dilation by N and the erosion by N are the operators δN and εN on P(E) defined by setting for Z ⊆ E: δN (Z) = z∈Z N (z), (24) εN (Z) = {p ∈ E | N (p) ⊆ Z}. Note that

(p) ∩ Z = ∅} , δN (Z) = {p ∈ E | N c

c

and that (δN (Z c )) = εN (Z) and (εN (Z c )) = δN (Z). A “folk theorem” states that – (εN , δN ) is an adjunction on P(E), and

14

Christian Ronse

– conversely, given an adjunction (ε, δ) on P(E), there is a unique neighbourhood function N such that ε = εN and δ = δN ; for every p ∈ E, we have N (p) = δ({p}). Finally, we have always δN (∅) = ∅ and εN (E) = E. On the other hand, δN (E) = (p) = ∅. E iff for every p ∈ E, N = ∅, while εN (∅) = ∅ iff for every p ∈ E, N (p) We can now consider the flat operators corresponding to dilations and erosions on P(E). We see that we obtain the same formulas as in the usual case of numerical functions [4, 5]: Theorem 1. Let δN and εN be the dilation and erosion (on P(E)) by a neighboorhood function N . Then: 1. For F ∈ V E and p ∈ E, we have:

V δN (F )(p) = q∈N(p) F (q), εVN (F )(p) = q∈N (p) F (q).

(25)

V ) is an adjunction on V E . 2. (εVN , δN

Proof. (i) Applying successively (21), (24) and (3), we get:

V (p) ∩ Xv (F ) δN (F )(p) = {v ∈ V | p ∈ δN (Xv (F ))} = {v ∈ V | N = ∅} ,

= {v ∈ V | ∃q ∈ N (p), F (q) ≥ v} = (p) {v ∈ V | F (q) ≥ v} , q∈N

F {v ∈ V | F (q) ≥ v} = = q∈N (p) (p) (q) . q∈N Therefore the first equality of (25) is proved. By (21,24) we get:

∈ V | N (p) ⊆ Xv (F )} , εVN (F )(p) = {v ∈ V | p ∈ εN (Xv (F ))} = {v = {v ∈ V | ∀q ∈ N (p), F (q) ≥ v} = q∈N (p) F (q) . In the last equality we used, for the set of F (q) for q ∈ N (p), the fact that the infimum of a set, being its greatest lower bound, is the supremum of its lower bounds. Therefore the second equality of (25) is proved. V ) is an adjunction, in other words that for (ii) We have to prove that (εVN , δN V (F ) ≤ G ⇐⇒ F ≤ εVN (G), that is: every F, G ∈ V E , δN V ∀p ∈ E, δN (F )(p) ≤ G(p) ⇐⇒ ∀q ∈ E, F (q) ≤ εVN (G)(q) We have indeed the following equivalences: V (F )(p) ≤ G(p) δ N (p) F (q) ≤ G(p) q∈N ∀p ∈ E, ∀q ∈ N (p), F (q) ≤ G(p) ∀q ∈ E, ∀p ∈ N (q) F (q) ≤ G(p) ∀q ∈ E, F (q) ≤ p∈N (q) F (p) ∀q ∈ E, F (q) ≤ εVN (G)(q) .

∀p ∈ E, ⇐⇒ ∀p ∈ E, ⇐⇒ ⇐⇒ ⇐⇒ ⇐⇒

Flat Morphological Operators on Arbitrary Power Lattices T

15

(b)

(a) F

t2 t1

o Z1

T

Z2

E

B

T

(c)

E

(d)

F

F

FΘ B

Z1

Z2

Z1

E

Z2

E

Fig. 3. (a) A function F : R → R. (b) A structuring element B containing the R origin o. (c) The erosion F B in R with the numerical ordering on R. (d) R The same erosion in R with the label ordering on R

In Fig. 3, we show the erosion of a grey-level function by a structuring element containing the origin (this erosion is thus antiextensive), in two cases: first, the grey-levels are numerically ordered; second they are considered as labels. The function F consists of two neighbouring flat zones Z1 and Z2 ; with the numerical ordering, the grey-level t1 of Z1 spills into Z2 ; on the other hand, with the label ordering, Z1 and Z2 are eroded separately, keeping the grey-levels t1 and t2 on Z1 and Z2 respectively. Now we give some results and counterexamples concerning the flat extension of a union, intersection, and composition of set operators. The flat extension of a union of set operators is the join of their individual flat extensions: Proposition 6. Given a family of increasing operators ψi (i ∈ I) on P(E),

ψi

V

=

i∈I

ψiV .

i∈I

Proof. For F ∈ V E and p ∈ E, (21) gives i∈I

ψi

V

(F )(p) = {v ∈ V | p ∈ i∈I ψi (Xv (F ))} , = {v , ∈ V | ∃i ∈ I, p ∈ ψi (Xv (F ))} = {v ∈ V | p ∈ ψ (X (F ))} , i v i∈I

(26)

16

Christian Ronse

and

ψiV (F )(p) =

i∈I

{v ∈ V | p ∈ ψi (Xv (F ))} . i∈I

The equality follows from (3). Corollary 1. Given two increasing operators η, ζ on P(E), η ⊆ ζ implies that ηV ≤ ζ V . Proof. As η ⊆ ζ, we have ζ = η ∪ ζ, so ζ V = η V ∨ ζ V , and hence η V ≤ ζ V . For the dual of Proposition 6, we need the following prerequisite: Lemma 3. Given a family of cylinders CBi ,vi (i ∈ I), CBi ,vi = C∩i∈I Bi ,∧i∈I vi .

(27)

i∈I

Proof. For p ∈ i∈I Bi , each i ∈ I gives p ∈ Bi and so CBi ,vi (p) = vi , hence

∈ / i∈I Bi , there is some i∈I CBi ,vi (p) = i∈I vi ; on the other hand for p ⊥. Now i ∈ I such that / Bi and so i (p) = ⊥, hence i∈I CBi ,vi (p) = p∈ CBi ,v

we find that C∩i∈I Bi ,∧i∈I vi (p) = i∈I vi if p ∈ i∈I Bi and ⊥ if p ∈ / i∈I Bi . Hence the equality (27) holds. Now the flat extension of an intersection of set operators does not always coincide with the meet of their individual flat extensions: Proposition 7. Given a family of increasing operators ψi (i ∈ I) on P(E), i∈I

ψi

V

≤

ψiV .

(28)

i∈I

and the equality holds: a) when the lattice V is completely distributive; b) when the family of operators is finite (|I| < ∞) and the lattice V is infinite supremum distributive. V

≤ ψjV by Proof. For any j ∈ I we have i∈I ψi ⊆ ψj , so we get i∈I ψi Corollary 1. Taking the infimum on all j ∈ I, (28) follows. Suppose that V is completely distributive. For F ∈ V E , (20) gives ψiV (F ) = Cψi (Xv (F )),v , so that ψiV (F ) = Cψi (Xv (F )),v . v∈V

i∈I

i∈I v∈V

We apply the extended distributivity law (7); here the choice functions are all maps I → V ; in combination with (27), this gives: ψiV (F ) = Cψi (Xϕ(i) (F )),ϕ(i) = C∩i∈I ψi (Xϕ(i) (F )),∧i∈I ϕ(i) . i∈I

ϕ∈V I i∈I

ϕ∈V I


17

For every choice function ϕ ∈ V I , set vϕ = i∈I ϕ(i); then for each i ∈ I we have (F ) ⊆ Xvϕ (F ), and ψ is increasing, ϕ(i) ≥ vϕ , so by (17) we get Xϕ(i) as

i ψi Xϕ(i) (F ) ⊆ ψi Xvϕ (F ) . Hence i∈I ψi Xϕ(i) (F ) ⊆ i∈I ψi Xvϕ (F ) , and the previous equation gives: ψiV (F ) ≤ C∩i∈I ψi (Xv (F )),vϕ ≤ C∩i∈I ψi (Xv (F )),v . ϕ i∈I

v∈V

ϕ∈V I

But (20) gives i∈I

ψi

V

(F ) =

v∈V

C(∩i∈I ψi )(Xv (F )),v =

C∩i∈I ψi (Xv (F )),v .

v∈V

V V V (F ) ≤ (F ), hence i∈I ψiV ≤ , which is Thus i∈I ψi i∈I ψi i∈I ψi the converse of inequality (28). Therefore the equality holds. If V is ISD (4), we have (6), which represents the extended distributivity law (7) for a finite set I, and the above argument applies then with a finite I. The usual lattices of image values (corresponding to sets, or to numerical or multivalued functions) are completely distributive, and so is any finite distributive lattice. The above equality is then verified. However it does not always hold when V is not distributive. Taking for V the non-distributive lattice U of labels V (with |U∗ | ≥ 3), we show in Fig. 4 an example where (δ1 ∩ δ2 ) < δ1V ∧ δ2V for two dilations δ1 and δ2 . Finally, our last two results show that the flat extension of a composition of set operators is the composition of their individual flat extensions, provided that: the last operator is a dilation, or V is completely distributive: Proposition 8. Consider an increasing operator ψ and a dilation δ on P(E). Then (δψ)V = δ V ψ V . Proof. We have δ = δN for some neighbourhood function N . For F ∈ V E and p ∈ E, (21) gives:

(δψ)V (F )(p) = {v ∈ V | p ∈ δψ (Xv (F ))} ,

(p) ∩ ψ (Xv (F )) = {v ∈ V | N = ∅} ,

= {v ∈ V | ∃q ∈ N (p), q ∈ ψ (Xv (F ))} ,

= {v ∈ V | q ∈ ψ (X (F ))} . v (p) q∈N V On the other hand (25) for δN gives with (21) for ψ V : δ V ψ V (F )(p) = ψ V (F )(q) = {v ∈ V | q ∈ ψ (Xv (F ))} . q∈N (p) q∈N (p)

(p) in place of I. The equality follows from (3) with N

18

Christian Ronse

a

o b a o

b

a a

c a

c b

a

a

a

a

c

(a)

(b)

(c)

Fig. 4. E = Z2 and V = U with |U∗ | ≥ 3. (a) The two structuring elements A1 and A2 , centered about the origin o; let δ1 and δ2 be the dilations by A1 and A2 respectively (for sets). (b) The function F ∈ U E , where a, b, c are pairwise distinct proper labels, and with value ⊥ outside the support shown here. (c) Top: U U δ1 ∧ δ2U )(F ), and bottom: (δ1 ∩ δ2 ) (F ), which has a smaller value on the central pixel Theorem 2. Given two increasing operators ψ, ξ on P(E), (ξψ)V ≤ ξ V ψ V . If V is completely distributive, then (ξψ)V = ξ V ψ V .

Proof. Let F ∈ V E . For any p ∈ E, (21) gives ψ V (F )(p) = {v ∈ V | p ∈ ψ (Xv (F ))}. Thus for any ψV (F )(p) ≥ v, V v ∈ V , p ∈ ψ (Xv (F )) implies that V in other words p ∈ Xv ψ (F ) . Henceψ (Xv (F )) ⊆ Xv ψ (F ) , and as ξ is increasing, ξψ (Xv (F )) ⊆ ξ Xv ψ V (F ) . Then (20) gives: Cξψ(Xv (F )),v ≤ Cξ(Xv (ψV (F ))),v = ξ V ψ V (F ) . (ξψ)V (F ) = v∈V V

v∈V V

V

Therefore (ξψ) (F ) ≤ ξ ψ (F ) for all F ∈ V E . Suppose that V is completely distributive. Let v ∈ V \ {⊥}. Take

p ∈ Xv ψ V (F ) . Set W = {w ∈ V | p ∈ ψ (Xw (F ))}. By (21) we have W =

ψ V (F )(p), and as p ∈ Xv ψ V (F

) , ψ V (F )(p) ≥ v. Thus W ≥ v, and so W = ∅; now ISD (4) gives v = w∈W (v ∧ w). Apply Lemma 1 with a = v and Y = {v ∧ w | w ∈ W }: for any x ∈ Gv there is some w ∈ W such that x ≤ v ∧ w. So x ≤ w, and (17) gives Xw (F ) ⊆ Xx (F ), and as ψ is increasing, ψ (Xw (F )) ⊆ ψ (Xx (F )), and Vas p ∈ ψ (Xw (F )), we get p ∈ ψ (Xx (F )). As this holds for any p ∈ X shown that for every v ψ (F ) , we have V Vthus x ∈ Gv , Xv ψ (F ) ⊆ ψ (Xx (F )), so that ξ Xv ψ (F ) ⊆ ξψ (Xx (F )). V ) } and Bp = {x ∈ V | Take any p ∈ E. Set Ap = {v ∈ V | p ∈ ξ X v ψ (F

V V p ∈ ξψ (Xx (F ))}. By (21) we have ξ ψ (F ) (p) = Ap and (ξψ)V (F )(p) =


o

(a)

a

b

b

a

c

c

a

a

b

b

c c

(b)

19

a

(c)

Fig. 5. E = Z and V = U with |U∗ | ≥ 3. (a) The structuring element A is a 2 × 2 square with the origin o in the bottom right corner; let δ and ε be respectively the dilation and erosion by A (for sets). (b) The function F ∈ U E , where a, b, c are pairwise distinct proper labels, and with value ⊥ outside the support shown here; we have (εδ)U (F ) = F . (c) εU δ U (F ), which has a greater value than (εδ)U (F ) on the central pixel 2

Bp . For any v ∈ Ap \ {⊥}, for p ∈ ξ Xv ψ V (F ) ⊆

any x ∈ Gv , as

ξψ (Xx (F )), we have x ∈ Bp . So Bp ≥ x∈Gv x = Gv = v; as this holds

for any v ∈ Ap \ {⊥}, we get Bp ≥ Ap , that is (ξψ)V (F )(p) ≥ ξ V ψ V (F )(p). Therefore (ξψ)V (F ) ≥ ξ V ψ V (F ) for all F ∈ V E , and the equality (ξψ)V = ξ V ψ V follows.

As for Proposition 7, the equality is verified for the usual lattices of image values (which are completely distributive), but it does not always hold when V is not distributive. We show in Fig. 5 an example where we take again for V the non-distributive lattice U (with |U∗ | ≥ 3), and (εδ)V < εV δ V for some adjunction (ε, δ). An interesting consequence of Propositions 6 and 8 is for the flat extension of an operator ψ on sets expressed as a union of erosions, or of an opening γ on sets expressed as a union of morphological openings (using the theorems of Matheron and Serra [6, 17]); from the decompositions εi and γ= δj ε j , ψ= i∈I

we deduce the decompositions εVi ψV = i∈I

4

j∈J

and

γV =

δjV εVj .

j∈J

Conclusion

We have given here the first elements of a general theory of flat morphological operators on images taking their values in an arbitrary complete lattice. It can be applied on RGB colour images, images with the reference order [7, 9, 10] (provided that we close the grey-level reference inf-lattice with a greatest element

20

Christian Ronse

∞), label images of Agnus [1, 2], etc. Our theory is “natural”, in the sense that it relies only on the usual thresholding, stack processing and superposition techniques (cfr. Definitions 1 and 2). We have shown that many known results extend in this general framework: possibility to restrict thresholds to a sup-generating subset of the lattice of values, preservation of any complete sublattice of values, same behaviour on twovalued images as the underlying set operator, formulas for flat dilation and erosion, flat extension of a union of set operators as the join of their individual flat extensions, and flat extension of a composition of a set operator by a dilation as the composition of their respective flat extensions. However complete distributivity is necessary for the flat extension of a composition of two set operators to be the composition of their respective flat extensions, as well as for the extension of an intersection of set operators to be the meet of their individual flat extensions; (but only infinite supremum distributivity is required for a finite intersection). Due to limitations on space, we have not dealt with further properties of flat operators, which have been known for some time in the case of grey-level images: duality under value inversion, commutation with anamorphoses and with thresholding. This will be the subject of a further work [15]. There we will also give a sup-inf decomposition formula for flat operators which generalizes the Matheron decomposition [6, 8, 17] of an increasing operator as a union of erosions.

Acknowledgement This work arose through the encounter of wisdom and charm. The motivation for writing it came through discussions with wise and learned people, especially Vincent Agnus and Henk Heijmans, but also the referees (for their encouraging criticism). The inspiration that made it a reality flowed from several sources, in particular the magical atmosphere of castle Dagstuhl, and the sweet personality of Kathleen.

References [1] V. Agnus, C. Ronse, F. Heitz: Segmentation spatiotemporelle morphologique de séquences d’images. In RFIA’2000: 12` eme Congrès Francophone “Reconnaissance des Formes et Intelligence Artificielle”, Paris, France (2000), Vol. 1, pp. 619–627. 2, 20 [2] V. Agnus, C. Ronse, F. Heitz: Spatio-temporal segmentation using morphological tools. Proc. 15th International Conference on Pattern Recognition, Barcelona, Spain (2000), Vol. 3, pp. 885–888. 2, 20 [3] G. Birkhoff: Lattice Theory (3rd edition), American Mathematical Society Colloquium Publications, Vol. 25, Providence, RI (1984). 4, 5, 6 [4] H. J. A. M. Heijmans: Theoretical aspects of gray-level morphology. IEEE Trans. Pattern Analysis & Machine Intelligence, Vol. 13 (1991), pp. 568–582. 1, 7, 8, 9, 14


21

[5] H. J. A. M. Heijmans: From binary to grey-level morphology. Unpublished (1991). 1, 14 [6] H. J. A. M. Heijmans: Morphological Image Operators, Acad. Press, Boston, MA (1994). 1, 2, 3, 4, 6, 7, 13, 19, 20 [7] H. J. A. M. Heijmans, R. Keshet: Inf-semilattice approach to self-dual morphology. J. Mathematical Imaging & Vision, to appear (2002). 2, 19 [8] H. J. A. M. Heijmans, C. Ronse: The algebraic basis of mathematical morphology I: dilations and erosions. Computer Vision, Graphics & Image Processing, Vol. 50, no. 3 (1990), pp. 245–295. 4, 13, 20 [9] R. Kresch: Extensions of morphological operations to complete semilattices and its applications to image and video processing. In H. Heijmans & J. Roerdink, editors, International Symposium on Mathematical Morphology 1998. Mathematical morphology and its applications to image and signal processing IV, pp. 35–42, Kluwer Academic Publishers, June 1998. 2, 19 [10] R. Keshet (Kresch): Mathematical Morphology on complete semilattices and its applications to image processing. Fundamenta Informaticae, Vol. 41 (2000), pp. 33–56. 2, 19 [11] P. Maragos, R. W. Schafer: Morphological filters – Part II: Their relations to median, order-statistics, and stack filters. IEEE Trans. Acoustics, Speech and Signal Processing, Vol. 35 (1987), pp. 1170–1184. 1 [12] G. N. Raney: Completely distributive complete lattices. Proceedings of the American Mathematical Society, Vol. 3 (1952), pp. 677–680. 5 [13] C. Ronse: Order-configuration functions: mathematical characterizations and applications to digital signal and image processing. Information Sciences, Vol. 50, no. 3 (1990), pp. 275–327. 1 [14] C. Ronse: Why mathematical morphology needs complete lattices. Signal Processing, Vol. 21, no. 2 (1990), pp. 129–154. 9 [15] C. Ronse: Anamorphoses and flat morphological operators on power lattices. In preparation. 1, 2, 3, 20 [16] C. Ronse, V. Agnus: Morphology on label images, and applications to video sequence processing. In preparation. 2, 6 [17] J. Serra: Image Analysis and Mathematical Morphology, Vol. 2: Theoretical Advances. Academic Press, London, 1988. 19, 20 [18] J. Serra: Anamorphoses and function lattices (multivalued morphology). In E. R. Dougherty, editor, Mathematical Morphology in Image Processing, pp. 483–523, Marcel Dekker, New York, 1993. 1, 2, 3 [19] P. D. Wendt, E. J. Coyle, N. C. Callagher: Stack Filters. IEEE Trans. Acoustics, Speech and Signal Processing, Vol. 34 (1986), pp. 898–911. 1

Quantification of the Spatial Distribution of Line Segments with Applications to CAD of Chest X-Ray CT Images Yasushi Hirano1 , Yoshito Mekada2 , Jun-ichi Hasegawa3, and Junichiro Toriwaki2 1

3

Information Technology Center, Nagoya University Furo-cho, Chikusa, Nagoya 464-8601, Aichi, Japan [email protected] 2 Faculty of Engineering, Nagoya University Furo-cho, Chikusa, Nagoya 464-8603, Aichi, Japan {mekada,toriwaki}@nuie.nagoya-u.ac.jp School of Computer and Cognitive Sciences, Chukyo University 101 Tokodachi, Kaizu, Toyota 470-0393, Aichi, Japan [email protected]

Abstract. We introduce two features to quantify distributions of line figures in the three-dimensional (3D) space. One of these is the Concentration index and the other is a feature based on the extended Voronoi tessellation. The former quantifies the degree of concentration, and the latter the difference of density. We explain the two features with their applications to the benign/malignant discrimination of lung tumors. The theoretical analysis is also shown.

1

Introduction

Recently, the opportunities of use of three-dimensional (3D) images are increasing. Particularly, in the field of clinical medicine, 3D X-ray CT images are used for screening and diagnosis extensively. Because recognition of objects in 3D images is difficult even for human eyes, studies of CAD(Computer Aided Diagnosis) are continued actively for 3D medical images[1, 2, 3, 4]. Recognition of the states of line figures in a 3D image is useful to recognize what exists in the image. Line figures are useful to present objects, because they can express edges of objects, medial lines of cylindrical objects and line figures themselves. Although many studies to extract line figures from 3D images(binary or gray) have been reported [5, 6, 7, 8, 9, 10, 11, 12, 13, 14], there are few reports discussing the method to quantify states of the spatial distribution of 3D line figures. Features of the spatial distribution of line segments in a 3D image have been found to be useful in the diagnosis of many diseases. Methods to analyze the 3D structure of objects have been studied in the field called stereology[15]. It was assumed, however, that only limited numbers of cross sections are available. Therefore the 3D structure of objects was estimated T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 22–38, 2003. c Springer-Verlag Berlin Heidelberg 2003

Quantification of the Spatial Distribution

23

only statistically instead of correctly analyzing the inside of objects. Examples are estimation of micro structures of living tissues and metal from microscope images of their cross sections. In the case of 3D image analysis we discuss here we can utilize a 3D array of data obtained by measuring the whole of objects directly. For example, we can know values of attenuation coefficient of X-ray in the value element of human body with the spatial resolution of 0.2mm × 0.2mm × 0.2mm by using recent X-ray CT equipment. Thus strictly deterministic analysis of the 3D structure becomes possible without employing any statistical estimation. In order to make most of the advantage to access the 3D data directly we need to develop effective methods of 3D image analysis. In this paper, we propose two features to quantify the state of the distribution of line figures from 3D images instead of cross sections themselves. One of them measures the degree of concentration of line figures to the specified point. This feature is the extended version of the 2D concentration index [16, 17, 18] to 3D space. The other is based on the extended Voronoi tessellation (Voronoi division)[19, 20]. It quantifies the density of line figures in local areas in 3D images. The two features were used for the benign/malignant discrimination for CAD[1, 21, 22]. The paper first presents the definition and explanation of the properties of the 3D concentration index in Section 2. Second we explain the feature base on the extended Voronoi tessellation in Section 3, and give in Section 4 practical examples by applying the two features to chest X-ray CT images which show the usefulness of them for the benign/malignant discrimination of lung tumors.

2

3D Concentration Index

The concentration index quantifies the degree of the concentration of line figures in 3D space toward an arbitrary point in the same 3D space. The index was first introduced to quantify the degree of concentration of line figures in 2D space[16, 17, 18]. The 2D version of the concentration index (2D concentration index) was applied to detect stomach cancer from double contrast X-ray images of stomach. It was shown that the index was useful to detect stomach cancer[18]. The 3D version of the concentration index(3D concentration index) is the extended version of the 2D concentration index to 3D space. It is used to quantify the convergence of vessels and bronchus in chest X-ray CT images for the benign/malignant discrimination of lung tumors. In the following, the 3D concentration index is simply called the concentration index. The concentration index is defined in both of continuous and discrete spaces. The theoretical analysis and basic properties of the index are shown using the continuous version, and computational results for artificial images and actual CT images are shown using the discrete version.

24

Yasushi Hirano et al.

r1 r2

R P

Used line figure Unused line figure Fig. 1. Illustration of the neighborhood region R

2.1

Neighborhood Region

The index is calculated at each point P in the 3D space using line figures in the neighborhood region of the point at which the concentration is evaluated. The spherical shell as shown in Fig.1 is used as the neighborhood region in this paper to avoid the effect of the line figures near the point P. The size of the neighborhood region R is defined by radii r1 and r2 of the inner and the outer sphere of the spherical shell (Fig.1). 2.2

Concentration Index of Continuous Version

Definitions for Continuous Images. The index of the continuous version is defined as the following. [Definition 1] When only one curve C exists in the neighborhood region R of a point P, the concentration index C(P) at the point P is defined as follows. |ds|| cos α| C(P) =

C

r |ds| C r

,

(1)

where |ds| is the length of the tangent vector ds at a point Q on the curve C, r is the length of a line segment PQ, and α is the angle between PQ and ds(Fig.2). C means the line integral along the curve C, and it is assumed that the first order derivatives of the curve C are continuous. Because Eq.(1) contains the reciprocal of the distance r, curves which exist near the point P contribute too much to the index. Therefore we exclude the neighborhood of the point P in the calculation of C(P) as shown in Fig.1. The denominator of Eq.(1) is the same as the numerator except for the term | cos α|. The denominator normalizes the index so that the index takes values between 0.0 and 1.0.


α

Q

25

ds

r P Fig. 2. Illustration of the definition of the index[Definition 1](Continuous version)

[Definition 2] When there are n curves C1 , C2 , . . ., Cn in the neighborhood region R, the concentration index is defined as |ds|| cos α| i=1 Ci r n |ds| i=1 Ci r

n C(P) =

.

(2)

It is assumed that there is no intersection and no overlapping among curves C1 , C2 , . . ., Cn and themselves. The first order derivatives of all curves are assumed to be continuous. If intersections among curves or discontinuity points of the first order derivatives exist, the curve is divided at those points into more than one curve. In the case that there is no curve in the neighborhood region R, the value of the index is defined as 0 for the convenience. If a curve is a straight line segment, the concentration index can be calculated explicitly. Assuming that the direction of the line segment AB is coincident with the direction of the line segment AP (Fig.3(a)), the index is given as the following equation. C(P) =

max ln L Lmin max ln L Lmin

= 1.0

(3)

Otherwise (Fig.3(b)), it is calculated as C(P) =

−ln cos θ 1 1+sin θ 2 ln 1−sin θ

.

(4)

If more than one straight line segments exist in R, the index can be calculated by adding all indexes of line segments in the same way as Eq.(2). Numerical Experiment. The index was calculated for n parallel lines in the neighborhood region R using Eqs.(3) and (4). All lines are parallel to x-axis and laid on xy-plane with equal intervals. The index was calculated for three different sizes of R. The parameter r2 was fixed and T(= rr21 ) was changed(Fig.1). Fig.4 shows the index for different values of T. The index changes significantly for small values of n and it approaches to 0.5 gradually with increase of n.

26


Lmax

L P

A

B

B A θ

Lmin (a)

P

H

(•• )

Fig. 3. Positional relation between a point P and a straight line AB

Fig. 4. Concentration index for parallel line patterns. T and n are the ratio of the outer radius to the inner radius of the neighborhood region, and the number of parallel lines in the neighborhood region, respectively

Considering that the set of parallel line figures contain almost all angles α in Eq.(1), it is thought that the expected value of the index is 0.5. Thus, if we want to set a threshold to decide whether concentration exists or not, the value 0.5 will be reasonable as the threshold. Basic Properties. The following properties are known concerning the concentration index. 1. 0 ≤ C(P) ≤ 1 2. If only straight lines exist in the neighborhood region R, and if they are directed to the same point P(the angle α in Eq.(1) is 0), then C(P) is equal to 1(Fig.5(a)) 3. If a circle exists in the neighborhood region R, the index at the center P of the circle is equal to 0(Fig.5(b)). The reason is that the angle α in Eq.(1) is π 2 at any point on the circle. 4. The expected value of the concentration index of a set of parallel lines is 12 . For derivation and proofs see Ref.[21].


P

P

(a )

(b)

27

Fig. 5. Basic properties of the index. (a) The index is equal to 1. (b) The index is equal to 0

2.3

Concentration Index for Digital Images

Here, the index of a digital image (discrete version) is introduced for applying digital images. The basic properties of the discrete version are the same as the continuous version except for the definition of the digital line segment. Definitions of Discrete Version. Tangent vectors of line figures in the digital curve can not be calculated. Line figures are divided into “3D line elements (We call it line element simply.)”, and the direction of the tangent vector is approximated by the direction of the line element. [Definition 3] The line element is defined at the voxel Q which satisfies the following conditions. (Condition 1) The voxel Q is located on a line figure. (Condition 2) In N 26 (Q)(= 26-neighborhood of Q), there are only two voxels Q1 and Q2 on line figures. (Condition 3) The 26-neighborhood distance between Q1 and Q2 is equal to 2. The line element is located on the voxel Q, the length of the line element is the half of the length of the line segment Q1 Q2 , and the direction is the same as that of the line segment Q1 Q2 (Fig.6). Consider, for example, three voxels Q = (x, y, z), Q1 = (x1 , y1 , z1 ) and Q2 = (x2 , y2 , z2 ) where Q1 and Q2 ∈ N 26 (Q). Then the line element on the voxel Q = (x, y, z) exists such that its direction and length are (dx, dy, dz) = 1 y2 −y1 z2 −z1 , , ) and ds = dx2 + dy 2 + dz 2 , respectively. Examples of line ( x2 −x 2 2 2 elements are shown in Fig.6. Table 1 shows the directions and lengths of all possible patterns of line elements. When dx = 1, dy = 12 , and dz = − 21 (No.17), √ √ 2 (rectangular paralthe length of the line element is 26 (cubic voxels) or 5+a 2 lelepiped voxels) from [Definition 3]. The length is the same for dx = −1, dy =

28


(b)

(a)

(d)

(c)

(e)

(f) : 3D line element

: Voxel

Fig. 6. Examples of line elements − 12 , and dz = 12 . So far as the length of an element is concerned, only the direction is significant. In fact the length of the element does not change when the sign of the component is inverted. Twelve kinds of lengths are shown for rectangular parallelepiped voxels in Table 1. For the certain values of a, parts of these become the same lengths. To take an illustration, when a = √23 , only nine kinds of lengths remain (Table 2). [Definition 4] Consider a point P in the 3D discrete space. Assume that the line element exists at the voxel Q. Then the degree of the contribution to the concentration index at P by the line element at Q is defined by ds| cos α|,

(5)

where ds is the length of the line element, and α is the angle between the line segment PQ and the line element (Fig.7). The line segment PQ is the line connecting the point P and the center of the voxel Q. The amount of Eq.(5) is called the directional component. The point P needs not be at the center of a voxel. Denoting components of the direction of a line element by (dx, dy, dz) (These are derived from [Definition 3], and in concrete terms they take one of values in Table 1), and the direction of PQ as (rx , ry , rz ), r dx + r dy + r dz x y z 2 2 2 ds| cos α| = dx + dy + dz rx2 + ry2 + rz2 dx2 + dy 2 + dz 2 =

|rx dx + ry dy + rz dz| . rx2 + ry2 + rz2


29

Table 1. All possible patterns of 3D line elements and their lengths. Here, vx, vy and vz are the voxel sizes along each axis. Sizes of a voxel are assumed to be equal in x−, y− and z−directions(cubic voxel). Because the size in the z−direction often differs from others in actual 3D images, we show also such case in the Table length

direction

No. 1

dx

dy

dz

1

0

0

vx : vy : vz = vx : vy : vz = 1:1:a 1:1:1

1

length

direction

1

No. dx

dy

dz

28

1

1

0

29

1

-1

0

30

1

0

1

31

1

0

-1

32

0

1

1

33

0

1

-1 1/2

2

0

1

0

3

0

0

1

4

1

1/2

0

5 6

1 -1/2 0 1/2 1

0

7

1/2 -1

0

34

1

1

a 5 2

8

1

0

1/2

35

1

1 -1/2

9

1

0 -1/2

36

1

-1 1/2

10

0

1

37

1

-1 -1/2

11

0

1 -1/2

38

1

1/2

1/2 -1

1/2

5 2

4+a2 2

1/2 0

1

39

1

13

1/2 0

-1

40

1 -1/2 1

41

1 -1/2 -1

1+4a2 2

14

0

1/2

15

0

1/2 -1

42

1/2

1

1

16

1

1/2 1/2

43

1/2

1

-1

17

1

1/2 -1/2

44

1/2 -1

1

18

1 -1/2 1/2

45

1/2 -1

-1

19

1 -1/2 -1/2

20

1/2 1

21

1/2 1 -1/2

22

1/2 -1 1/2

23

1/2 -1 -1/2

24

1/2 1/2

25

1/2 1/2 -1

26

1/2 -1/2 1

27

1/2 -1/2 -1

5+a2 2

1/2 6 2

1 2+4a2 2

46

1

2 2 1+a2

8+a2 2

1

12

1

vx : vy : vz = vx : vy : vz = 1:1:1 1:1:a

1

1

47

1

1

-1

48

1

-1

1

49

1

-1

-1

3 2 5+4a2 2

3

2+a2

30


Table 2. Nine kinds of lengths are possible when a = with that in Table 1 No.

length

No.

1–2 3

1

24 – 27 28 – 29

8 – 11 4–7 12 – 15 16 – 23

√2 3 √

5 √2 19 √ 2 3

√2 . 3

Each No. coincides

length

30 – 33 34 – 37 38 – 45 46 – 49

√ 22 √ 2 √3

2

7 3

√ 31 √

2 3 10 3

Using the directional component, the index is defined as the following. [Definition 5] The concentration index (discrete version) at a point P in the discrete space is defined as the following ds| cos α| r C(P) = R ds , (6) R r

where R and r mean the summation for the line elements in the neighborhood region R, and the distance between the point P and the point Q where the line element exists, respectively. If no line element exists in the neighborhood region R, the index is defined to be 0. Experiment Using Artificial Patterns. The index was calculated for discrete artificial images. The gray image which is obtained by calculating the index at each voxel of an input image is called “the concentration index image”. Fig.8 shows the experimental results using artificial patterns. Input images contain three types of artificial patterns. Fig.8(a-1), (b-1) and (c-1) are input patterns,

Q2

Q

Line element

α r

Q1

P Fig. 7. Illustration of the directional component in [Definition 4]


31

that is, a concentrating pattern toward the point P, a circular pattern and a set of parallel lines, respectively. Fig.8(a-2), (b-2) and (c-2) show the concentration index images which are cut off partially. Cross sections of the concentration index images including their center points are shown in Fig.8(a-3), (b-3) and (c-3). Profiles of cross sections along dotted lines are attached below the corresponding cross sections. In the concentration index image for the concentrating pattern, values of concentration index are higher in the center of the images, and very low in the surrounding region(Fig.8(a-3)). In concentration index images for the circular pattern – ideally, all line elements are perpendicular to the lines connecting the center of the image and the voxels at which the line elements exist –, the concentration index takes very low values at the center of the image. Thus

Fig. 8. Concentration index images for artificial patterns. (a-1), (b-1) and (c-1): Input pattern. (a-2), (b-2) and (c-2): Concentration index images. (a-3), (b-3) and (c-3): Cross sections of concentration index images

32


(a) Generators

(b) Location of generators

(c) Voronoi regions

Fig. 9. Example of extended Voronoi tessellation

it is thought that existence of concentration can be detected by the value of the concentration index.

3

A Feature Based on 3D Extended Voronoi Tessellation

Here, a feature which quantifies the difference of the density of line figures is developed. This feature is based on the 3D extended Voronoi tessellation [19, 20], and defined as the ratio of volumes of Voronoi regions at the neighborhood of the specific point to the average volume of the Voronoi region over the whole image. Definition of the Feature. First, the line figures are divided into short line segments. The method of division is appropriately selected according to each application. For instance, if line figures are the result of thinning operation for cylindrical objects, and if the thickness of cylindrical objects is related with their positions in an input image, line figures may be divided according to the thickness of 3D objects from which line figures were extracted. [Definition 6] The feature which quantifies the difference of the distribution of line figures is defined as follows. Mr =

Maver Mspr

(7)

where Maver and Mspr are the average volume of Voronoi regions over the whole image and that in the neighborhood of the specified point, respectively. Suffix r means the category obtained from the division of line figures. If Mr is larger than 1.0, line figures distribute more densely near the specified point than the average.


(a) No convergence

33

(b) Strong convergence

Fig. 10. Example of the convergence

Application to an Artificial Image. Fig.9 shows the result of the extended Voronoi tessellation for an artificial pattern. Although we will explain the properties of the feature using the 2D version for the simplicity of description, the concept of the 3D version is almost the same. Fig.9(a) shows generators (seeds) for the extended Voronoi tessellation. Generators distribute sparsely in the left of the image, and densely in the right. Each generator is located at the center of a rectangle as shown in Fig.9(b). Letting Mspr be the average area of the Voronoi region located at the upperleft corner and its three nearest Voronoi regions, and Maver be the average area of Voronoi regions in the whole image, the feature Mr was 0.51. Using the Voronoi region located at lower-right corner and its three nearest Voronoi regions, the feature Mr became 1.81. It is shown that if generators distribute sparsely Mr is smaller than 1.0, and if densely, Mr is larger than 1.0.

4

Application to Chest CT Images

In this section, two features mentioned above were applied to chest X-ray CT images to quantify the observation of medical doctors. The “convergence” is an important observation to discriminate malignant tumors from benign tumors. The convergence is a phenomenon that tumors pull tissues(vessels, bronchus, interstitium, etc.) existing around them toward them. This phenomenon is found frequently when tumors are malignant. On the other hand, it is rarely observed in benign ones. Fig.10(a) and (b) show the case with no convergence and the case with strong convergence, respectively. These figures are rendered by MIP(Maximum Intensity Projection) only in lung regions. Gray regions are vessel regions and bronchus regions, and dark gray regions are the lung regions. The gray masses at the upper-left of the lung regions are tumors. Vessels and bronchus are spread radially in Fig.10(a). In the case with the convergence, vessels and bronchus are concentrated on tumors (Fig.10(b)).

34


Fig. 11. Example of chest X-ray CT images used in this experiment

An example of a slice from a chest X-ray CT image is shown in Fig.11. The arrow indicates a malignant tumor with strong convergence. Table 3 shows the detail of cases used in experiments. The existence of the convergence and the benign/malignant discrimination were evaluated by medical doctors. Extraction of vessel and bronchus region[1], extraction of tumor region[4] and thinning operation[6] to vessel and bronchus regions were applied to these twenty-nine cases. Two features mentioned above were calculated from line figures obtained by the above procedure. To calculate the feature Mr , only line segments obtained from vessel and bronchus regions which have a certain range of thickness were used. The suffix r in Eq.(7) means the category of the thickness of vessels and bronchus. All line figures were used to calculate the concentration index. The experimental results are shown using the case with strong convergence (Fig.10(b)) as an example. Fig.12 shows a cross section of the concentration index image through the center of the tumor. It is confirmed that in the region near the tumor values of the concentration index become higher. This tendency is the same as in the result of the artificial concentrating pattern(Fig.8). Fig.13(a) shows the Voronoi diagram for the whole lung region using line segments obtained from vessel regions which have the thickness of 0.4 – 0.6mm. Fig.13(b) shows Voronoi regions obtained from generators which exist closer than 10.0mm from the border of the tumor. The extended Voronoi tessellation was calculated in the 3D space, and two cross sections through the center of the tumor are

Table 3. Details of cases used in this experiment Convergence Benign Malignant Total Not exist 6 5 11 Exist 1 17 18 Total 7 22 29


35

Fig. 12. Concentration index image for a CT image

(a) In lung region

(b) In near the tumor region

Fig. 13. Results of 3D extended Voronoi tessellation

shown in the figure. It is confirmed that the volumes of Voronoi regions near the tumor are smaller than those apart from the tumor. Fig.14(a) and (b) are scatter diagrams of the average of the concentration index inside tumors and the ratio of volumes of Voronoi regions with symbols in graphs showing the existence of the convergence. Here, Maver is the average volume of the Voronoi regions in the whole lung, and Mspr is the average volume of the Voronoi regions which exist near the tumor. It is confirmed that almost all cases with convergence have feature values larger than 0.5 in the concentration index or larger than 1.0 in Mr . The Mann-Whitney test was applied to the distributions of each feature. The null hypothesis H0 :“There is no difference between the distributions of values of the feature for the case with the convergence and without the convergence.” was assumed. For the concentration index the hypothesis was rejected with the level of significance 0.01, and for Mr it was rejected with the level of significance 0.05. It is thought that two features are useful to quantify the existence of the convergence.

36


0.7

Concentration index

Concentration index

0.7

0.6

0.5

Convergence : Not exist Exist

0.4

0.6

0.5 Benign Malignant

0.4 0.5

1.0

1.5

Mr (a) With existence of convergence

2.0

0.5

1.0

1.5

2.0

Mr (b) With discrimination result by medical doctors

Fig. 14. Scatter diagrams of two features

Next, the benign/malignant discrimination was performed using the nearestneighbor decision rule. The leave-one-out method was employed for performance evaluation. When four features – the concentration index, the complexity in the shape of the tumor region, the average of CT values in the peripheral of the tumor region, and the volume of the peripheral of the tumor region – were employed, the correct discrimination ratio was 85.9%. When three features – the ratio of volumes of Voronoi regions(Mr ), the entropy of CT values in the tumor region, and the variance of CT values in the peripheral of the tumor region – were used, the correct discrimination ratio was 82.1%.

5

Conclusion

Two features for spatial distribution of line segments were proposed in this paper. One of these quantifies the degree of the concentration of line figures. The other measures the difference of density of line figures. The properties of features were shown experimentally. It is expected that the threshold of the concentration index for determine the existence of concentration is 0.5. Furthermore, examples of applications to practical images were shown. It is confirmed that the features are useful to quantify the phenomenon called convergence in lung. The results using the cases with the convergence had the same properties as those using the artificial concentrating pattern. For the feature based on the Voronoi tessellation, the experimental result showed that the difference in the distribution density of line segments can be distinguished using the feature. These results mean that features proposed here have an ability for the benign/malignant discrimination using chest X-ray CT images and are promising as shape features for CAD.


37

Acknowledgment Authors thank Dr. K. Eguchi of Tokai University, Japan and Dr. H. Ohmatsu of National Cancer Center Hospital East, Japan for much cooperation and useful advises, and thank colleagues of Prof. Toriwaki’s Laboratory of Nagoya University for useful discussions. Parts of this research were supported by the Grantin-Aid for Scientific Research from the Ministry of Education, Culture, Sports, Science and Technology, the Grant-in-Aid for Private University High-Tech Research Center from Ministry of Education, Culture, Sports, Science and Technology and the Grant-in-Aid for Cancer Research from the Ministry of Health, Labor and Welfare, Japanese Government.

References [1] Hirano, Y., Mekada, Y., Hasegawa, J., Toriwaki, J., Ohmatsu, H. and Eguchi, K: Three Dimensional Concentration Index –A Local Feature for Analyzing Three Dimensional Digital line Patterns and Its Application to Chest X-ray CT Images–. Proc. the 14th International Conference on Pattern Recognition(ICPR) ’98 (1998) 1040 – 1043 22, 23, 34 [2] Shikata, H., Kitaoka, H., Keserci, B., Sato, Y. and Tamura, S: Quantitative Evaluation of the Spatial Distribution of Vessels Surrounding Pulmonary Nodules Proc. Computer Assisted Radiology the 14th International Congress and Exhibition (CARS 2000) (2000) 761 – 766 22 [3] Wiemker, R. and Zwartkruis, A: Optimal Thresholding for 3D Segmentation of Pulmonary Nodules in High Resolution CT. Proc. Computer Assisted Radiology the 15th International Congress and Exhibition (CARS 2001) (2001) 611 – 616 22 [4] Hirano, Y., Hasegawa, J., Toriwaki, J., Ohmatsu, H., and Eguchi, K.: Extraction of Tumor Regions Keeping Boundary Shape Information from Chest X-ray CT Images and Benign/malignant Discrimination. Proc. Computer Assisted Radiology and Surgery the 16th International Congress and Exhibition (CARS2001) (2001) 617 – 622 22, 34 [5] Ragnemalm, I: Rotation Invariant Skeletonization by Thinning Using Anchor points. Proc. 8th Scandinavian Conference on Image Analysis(II) (1993) 1015 – 1022 22 [6] Saito, T. and Toriwaki, J.: A Sequential Thinning Algorithm for Three Dimensional Digital Pictures Using the Euclidean Distance Transformation. Proc. 9th Scandinavian Conf. on Image Analysis (1995) 507 – 516 22, 34 [7] Pudney, C: Distance-ordered Homotopic Thinning: A Skeletonization Algorithm for 3D Digital Images. Computer Vision and Image Understanding 72(3) (1998) 404 – 413 22 [8] Blezek, D. J. and Robb, R. A.: Centerline Algorithm for Virtual Endoscopy Based on Chamfer Distance Transform and Dijkstra’s Single Source Shortest Path Algorithm. Proc. SPIE Medical Imaging 1999 (1999) 225 – 233 22 [9] Bitter, I., Kaufman, A. and Wax, M.: Fully Automatic Extraction of the Colon Centerline and Its Impact on a Virtual Colonoscopy System. Proc. Computer Assisted Radiology and Surgery the 16th International Congress and Exhibition (CARS 2001) (2001) 625 – 628 22

38


[10] Monga, O., and Benayoun, S.: Using Partial Derivatives of 3D Images to Extract Typical Surface features. Trans. of Computer Vision and Image Understanding 61(2) (1995) 171 – 189 22 [11] Thirion, J.-P. and Gourdon, A.: Computing the Differential Characteristics of Isointensity Surface. Trans. of Computer Vision and Image Understanding 61(2) (1995) 190 – 202 22 [12] Wink, O., Niessen, W. J., and Viergever, M. A.: Fast Delineation and Visualization of Vessels in 3-D Angiographic Images. IEEE Trans. on Medical Imaging 19(4) (2000) 337 – 346 22 [13] Yim, P., Choyke, P. and Summers, R.,: Gray-Scale Skeletonization of Small Vessels in Magnetic Resonance Angiography. IEEE Trans. on Medical Imaging 19(6) (2000) 568 – 576 22 [14] Hirano, Y., Shimizu, A., Hasegawa, J., and Toriwaki, J.: A Tracking Algorithm for Extracting Ridge Lines in Three-Dimensional Gray Images Using Curvature of Four-Dimensional Hypersurface. Systems And Computers in Japan 32(12) (2001) 25 – 37 22 [15] Weibel, R. E.: Stereological Methods, Vol.2. Theoretical Foundations. Academic Press, London, 1980 (1979) 22 [16] Hasegawa, J., Tsutsui, T. and Toriwaki, J.: Automated Extraction of Cancer Lesions with Convergent Fold Patterns in Double Contrast X-ray Images of the Stomach. Systems and Computers in Japan 22(7) (1991) 51 – 62 23 [17] Hasegawa, J. and Toriwaki, J.: A New Filter for Feature Extraction of Line Pattern Texture with Application to Cancer Detection. Proc. the 11th International Conf. on Pattern Recognition (ICPR) (1992) C352 – 355 23 [18] Mekada, Y., Hasegawa, J., Toriwaki, J. and Nawano, S.: Automated Extraction of Cancer Lesions from Double Contrast X-ray Images of Stomach. Proc. the 1st International Work shop on Computer Aided Diagnosis (1998) 407 – 412 23 [19] Mase, K., Toriwaki, J. and Fukumura, T.: Modified Digital Voronoi Diagram and Its Applications to Image Processing. IEICE Trans. J64-D(11) (1981) 1029 – 1036 (in Japanese) 23, 32 [20] Saito, T. and Toriwaki, J.: Algorithms of Three Dimensional Euclidean Distance Transformation and Extended Digital Voronoi Diagram, and Analysis of Human Liver Section Images. Journal of Institute of Image Electronics Engineers of Japan (IIEEJ) 21(5) (1992) 468 – 474 (in Japanese) 23, 32 [21] Mekada, Y., Hirano, Y., Hasegawa, J. and Toriwaki, J.: Concentration Index for Analyzing Three Dimensional Line Patterns and Its Application to X-ray CT Images. IEICE Trans. J80-A(7) (1997) 1162 – 1170 (in Japanese) 23, 26 [22] Hirano, Y., Hasegawa, J. and Toriwaki, J: Quantification of the Convergence of Tissues in Lung for CAD of Chest X-ray CT images. Proc. the 16th International Conference on Pattern Recognition(ICPR) 2002 (2002) I-751 – I-754 23

Fan Clouds – An Alternative to Meshes Lars Linsen1 and Hartmut Prautzsch2 1

Center for Image Processing and Integrated Computing (CIPIC) University of California, Davis [email protected] 2 Institut f¨ ur Betriebs- und Dialogsysteme (IBDS) Universit¨ at Karlsruhe, Germany [email protected]

Abstract. A fan cloud is a set of triangles that can be used to visualize and work with point clouds. It is fast to compute and can replace a triangular mesh representation: We discuss visualization, multiresolution reduction, refinement, and selective refinement. Algorithms for triangular meshes can also be applied to fan clouds. They become even simpler, because fans are not interrelated. This localness of fan clouds is one of their main advantages. No remeshing is necessary for local or adaptive refinement and reduction.

1

Introduction

The real-time rendering of complex three-dimensional scenes is a challenging problem in computer graphics. Multiresolution methods can be used to reduce the complexity of a scene by adapting the level of detail to the viewing parameters. In this paper, we present multiresolution methods for point clouds or, more precisely, for fan clouds. Fan clouds are sets of local triangulations and obviate a more costly triangular mesh generation. In Section 3, we present their construction. The visualization of fan clouds is straight-forward. Contrary to the point cloud rendering techniques in [1, 29, 40, 44, 47], fan clouds provide a surface representation that can also be used for surface modeling and other processing operations. Thus, for most purposes, fan clouds can substitute triangular mesh representations. However, if required, triangular meshes can be generated quickly from fan clouds as discussed in Section 4. In Section 5, we define an entropy or importance measurement for the points of a point cloud and use it for data reduction. The entropy of a point depends on its surrounding fan and is also defined for triangular meshes. Each reduction step leads to a coarser level of detail and we get a hierarchical sequence of fan cloud representations.

http://graphics.cs.ucdavis.edu http://i33www.ira.uka.de

T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 39–57, 2003. c Springer-Verlag Berlin Heidelberg 2003

40

Lars Linsen and Hartmut Prautzsch

In Section 6, we go into multiresolution for modeling purposes, where detail is stored in a local frame. In Section 7, we discuss refinement for fan clouds beyond the given resolution. Furthermore, we look at selective refinement of fan clouds for static (Section 8) as well as dynamic scenes, where the shape of the objects may change due to modeling, animation or simulation processes (Section 6). In particular, we apply selective refinement to terrain rendering in Section 8. In contrast to triangular meshes, fan clouds allow for adaptive refinement without any topological restrictions and dependencies. Since fan clouds are local triangulations, all the introduced fan cloud processing algorithms can be run in parallel as well as by using out-of-core techniques when dealing with large-scale objects.

2

Related Work

Wavelets for multiresolution methods in computer graphics are discussed in detail by Stollnitz et al. in [45] for curves and meshes with subdivision connectivity. While Eck et al. [17] remesh arbitrary meshes to generate subdivision connectivity, Kobbelt et al. [31] as well as Guskov et al. [21] develop multiresolution for arbitrary meshes. In [28], multiresolution is further generalized to non-manifolds models. A survey is given in [32]. Mesh reduction is often based on the deviation of the reduced mesh from the original one. Such a reduction approach is common, e. g., in terrain rendering (cf. [14, 16, 20, 27, 33, 38, 43, 46]). In addition, one can take into account topological aspects [24], edge lengths [24], curvatures [22, 30], normals and colors [8], or textures [10]. In all these approaches, some energy is defined and minimized. This results in reduced meshes with high fairness. Selective refinement is commonly based either on constrained or arbitrary mesh connectivity. Constrained mesh connectivity comes with regular height fields, in general. Duchaineau et al. [16] and Lindstrom et al. [33] use binary tree hierarchies of subdivided right-angled isosceles triangles. Gross et al. [20], Pajarola [38], and R¨ ottger et al. [43] prefer quadtree hierarchies and subdivide a rectangle into four equal rectangles. For the visualization, each rectangle is split into two triangles. All these approaches suffer from the same problem shown in Figure 1 for quadtrees. Not all refinement steps lead to a valid mesh. In the figure, cracks appear in the surface due to the different refinement levels of adjacent quadtree blocks. Gross et al. [20] overcome this problem by generating a look-up table of all valid refinements, while Pajarola [38] as well as Röttger et al. [43] define restricted quadtrees and preserve the restriction by extra refinement steps. Restricted quadtrees requirement means that adjacent quadtree blocks differ by at most one level in the hierarchy. This constrained mesh connectivity requires more triangles for a given accuracy than arbitrary meshes (cf. [14, 25, 26, 27, 46]). De Floriani et al. [14] use a multi-triangulation for a multiresolution representation of surfaces. They

Fan Clouds – An Alternative to Meshes

41

Fig. 1. Corresponding surface after selective refinement of height fields using a quadtree hierarchy

define local refinement steps and dependencies between them, which lead to a partial order of the refinement steps stored in a directed acyclic graph. Each subgraph that contains all parents of each of its nodes provides an arbitrary mesh representation of the surface. Hoppe [25, 26, 27] as well as Xia and Varshney [46] use the vertex split refinement operation and its inverse, shown in Figure 2, to locally modify the level of detail of an arbitrary mesh representation. For a vertex split, they require that the four triangles ∆1 , . . . , ∆4 are active. If one of them is not active, other refinement steps are needed before. By a sequence of edge collapses a fine mesh M n can be transformed into a coarsest mesh M 0 [26, 27, 46]. The vertices of M 0 , M n , and all intermediate meshes M i form a vertex hierarchy. It can be represented by a forest, in which the root nodes are the vertices of M 0 and the leaf nodes are the vertices of M n . Any intermediate mesh M i corresponds to a vertex front through the vertex hierarchy and represents a selective refinement of M 0 . An example is illustrated by the colored nodes in Figure 3. Note that there may be invalid vertex fronts due to the restrictions for vertex splits. Establishing the forest can be done off-line in a preprocessing, which reduces the computations for real-time applications during runtime. Point clouds can be rendered without generating a triangular mesh: One can use splatting methods [1, 29, 40, 44, 47] (also called point-based rendering), which require dense point clouds and need much preprocessing, or fan clouds [34, 35].

∆1 ∆2

vertex split p

∆3

∆1

p1

∆ 2 p2

∆4 edge collapse

∆3

∆4

Fig. 2. A vertex split is executed only under certain preconditions

42


M0

15

Mi

10

Mn 1

14 2

11 3

9

13 6

12

5

7

8

4

Fig. 3. A selectively refined mesh M in the vertex hierarchy, which is represented by a forest

In [39], point clouds are approximated by linear B-spline patches to perform modeling operations.

3

Fan Clouds

An object can be represented by a sufficiently dense set of points on its surface. Fan clouds are simple, very local structures to work with point clouds. They have been introduced and discussed in [34]. To prepare for the following, we briefly recall their construction. For each point p of a point cloud, one computes a k-neighborhood consisting of k pointers to points p1 , . . . , pk of the cloud close to p as described further below. The neighbors are determined such that the k triangles ppi pi+1 form a fan that approximates the neighborhood of p on the surface represented by the point cloud. The set of all triangle fans is what is called a fan cloud. To determine a k-neighborhood of a point p, one determines the k nearest neighbors p1 , . . . , pk , computes the plane P with the least sum of squared distances to p, p1 , . . . , pk , and projects all points into P . Then one sorts, i. e., permutes the indices of p1 , . . . , pk , such that the projections qi of pi lead to an increasing sequence of angles ϕi = q1 qqi , where q is the projection of p. In this order, the points pi form a triangle fan or k-neighborhood of p, see Figure 4.

pi p2 p

ϕ

i

p1

Fig. 4. A k-neighborhood for k = 5


43

best fitting plane p

pi

Fig. 5. Best fitting plane for an edge point viewed along the plane If the point density varies sharply around p, then the neighborhood may not enclose p. Therefore, if ∇ϕi = ϕi − ϕi−1 > 90◦ , one replaces pk by the (k + 1)st neighbor and if necessary by further next neighbors till the angle criterion ∇ϕi ≤ 90◦ is met, or a certain threshold number of replacements has been reached. Along sharp edges, the best fitting plane may be normal to the surface, see Figure 5. Therefore, if the angle criterion cannot be satisfied, we rotate the fitting plane around the axis qi−1 qi by 90◦ and try again to build the neighborhood. If the angle criterion can still not be met, we assume that pi−1 , p, pi lie on the boundary of the surface. Note that a triangular mesh with n vertices has about 2n triangles, whereas a fan cloud consists of kn triangles, where k = 6 is a typical number we have used. However, storage costs are not higher for fan clouds. For each point p, we store a list of pointers to its neighbors p1 , . . . , pk . This is also the most efficient way to store a triangular mesh.

4

Triangular Mesh Generation from Fan Clouds

Triangular meshes are commonly used in Computer Graphics to represent surfaces. Therefore, in the nineties, various approaches were presented to generate triangular meshes from point clouds. The algorithms are based on spatial subdivision (e.g. [2, 4, 13, 23]), Delaunay tetrahedrization (e.g. [3, 7, 18]), distance functions (e.g. [13, 23]), warping (e.g. [2]), and incremental surface-increase (e.g. [5, 7, 12, 19, 36]). A survey is given in [37]. From their construction, we cannot expect fan clouds to provide a continuous surface representation. However, in all our experiments, we found that fan clouds are very much like triangular meshes, see Figure 6(a). In fact, they contain the triangles of a connected triangular mesh as a subset. Many triangles in our fan cloud are identical. Without duplicates, the fan clouds have about 2.5n triangles. Further, there are quadrilateral regions covered by three or four triangles of a fan cloud, i.e., by one or two superfluous triangles. Removing these superfluous triangles reduces the number of triangles to about 2.1n, see Figure 6(b). We observed that the reduced fan clouds are already triangular meshes with regions that are covered by several different triangulations. To obtain a 2D mesh manifold from a reduced fan cloud, we simply grew a triangular mesh by successively adding on triangles. The result is illustrated in Figure 6(c). It has

44


(a)

(b)

(c)

Fig. 6. The fan cloud in (a) is reduced to 2.1n triangles in (b) and further reduced to a triangular mesh in (c)

few (0.01%) self-overlaps, since we neglected any geometric aspects and based the construction only on topological aspects. Since by construction there are no edges with three ore more coincident triangles, the overlaps correspond to holes that fold back onto themselves as illustrated in Figure 7, where we have a quadrilateral hole marked by heavy lines. It is possible to avoid self-overlaps and to prove, then, the correctness of the triangulation [41]. Shaded versions of the objects in Figures 6(a) and 6(c) are shown in Figures 8(a) and 8(b), respectively. This triangulation method is fast. The construction is similar to the approach given by [12], but uses no additional information such as point classification. The computation times of triangular mesh generations via fan clouds (incl. fan cloud generation) are given in Table 1. Since the neighborhoods for typical point clouds can be computed in linear time using spatial subdivision, fan clouds can be generated in linear time. For spatial subdivision, we use a 3D-cell rasterization,


45

Fig. 7. Self-overlapping four sided hole (heavy lines)

(a)

(b)

Fig. 8. Comparing fan cloud visualization (a) with triangular mesh visualization (b)

such that for each neighborhood estimation we only have to search in a constant number of cells containing a constant number of points.

5

Reduction

Mesh reduction means to approximate a given mesh by a coarser mesh within some tolerance. Often the underlying surface is assumed to be fair and reduction is guided by the notion that it helps to minimize some fairness energy [30]. With point clouds the reduction principle is simpler. We only remove points without constructing some approximating surface. Hence, we try to find a small subset of a given point cloud that still is a good representation for the object. Since fan cloud generation is fast, the reduction scheme can be applied to large-scale data sets. Exploiting the local data structure of fan clouds, the reduc-

46


Table 1. Computation times for fan cloud generation and global triangulation via fan cloud method #points computer Fan cloud generation 47109 SGI Indigo2 Extreme Sun Ultra30 PC with Athlon K7 800MHz 100001 160940 Global triangulation 20021 PC with Athlon K7 800MHz via fan cloud 35948 160940

time 45 s 13 s 5s 17 s 25 s 4s 8s 41 s

tion scheme can also make use of out-of-core techniques and distributed computing. For fan cloud reduction, we introduce an entropy that encodes for each point, how much information it contributes to the geometric information of the surface. Similar to common fairing energies the entropy is based on point distances, color information, curvature estimates, and change in curvature. The main difference is that we try to keep the entropy high rather than to minimize it. An object like a sphere has an homogenous shape characteristic. In such situations a point that is close to its neighbors is less important than a point with large distances dj := qj − p2 to its neighbors q1 , . . . , qk . Thus, we define the measure k 1 Mdist (p) := dj k j=1 for each point p of the point or fan cloud. n −n Further, we estimate the curvature at p in the direction of any qj by jdj 2 , where n and nj are the normals at p and qj , respectively. Note that, actually, √ this is a (in dj ) linear approximation of κ2 + τ 2 = n 2 , where κ is the normal curvature and τ the geodesic torsion at p in the direction qj − p, see e. g. [6]. Hence, this estimate also measures the torsion or non-planarity of geodesics on the surface. Averaging these terms for all neighbors of p leads to the measure Mcurv (p) :=

k 1 nj − n2 k j=1 dj

of the curvature and geodesics torsion at p. Defining the change of curvature at p by Mcc(p) :=

k 1 |Mcurv (qj ) − Mcurv (p)| k j=1 dj


47

and the change of color at p by Mcol (p) :=

k 1 cj − c2 , k j=1 dj

where c and cj contain the RGB-encoded color information at p and qj , we get the entropy M (p) at p, M (p) := α1 Mdist (p) + α2 Mcurv (p) + α3 Mcc (p) + α4 Mcol (p) , with adjustable weights α1 , . . . , α4 . To make the terms of the entropy scale-invariant, we define the average distance between neighbored points d :=

k 1 dj , k·n j=1

p∈P

where P is the point cloud and n its size, and divide the term Mdist (p) by d, multiply the terms Mcurv(p) and Mcol (p) by d and multiply the term Mcc (pi ) by d2 . The fan cloud rendered in Figure 9(a) is reduced to 42% using the above entropy with α1 , α2 , α3 , α4 = 1, 0, 0, 0 in 9(b) and α1 , α2 , α3 , α4 = 1, 1, 50, 0 in 9(c). We iteratively remove a point p with lowest entropy and recompute the entropy of all points that had p as a neighbor. For Figures 9(d), 9(e), and 9(f), we zoom into one region of Figures 9(a), 9(b), and 9(c). Clearly the loss of surface features is dramatic in 9(e) and negligible in 9(f). Figure 10 gives a comparison of reductions applied to a colored fan cloud. The fan cloud in (a) shows part of the Grand Canyon. In (b), (c), and (d), it is reduced to 17.6%. In (b), the approximation error is minimized (see Section 8). In (c) and (d), we preserved the entropy as much as possible, where α1 , α2 , α3 , α4 = 1, 0, 0, 0 in (c), which results in an almost equal distribution of the points, and α1 , α2 , α3 , α4 = 1, 1, 50, 1 in (d). Again, preserving the entropy maintains salient curvature and color features best. Note that the entropy-based reduction can also be applied to triangular meshes. Furthermore, recently developed point-based rendering techniques [1, 29, 40, 44, 47] as well as hybrid (point-based/triangle-based) rendering approaches [9, 11] can be combined with this fan cloud reduction scheme.

6

Level of Detail

Successively removing points from the original fan cloud and adjusting the triangle fans, that contained the removed point, leads to a hierarchical sequence of fan clouds. We can switch back to any finer resolution in this hierarchy by reinserting points that have been removed. To accommodate for shape modifications at some lower resolution, we use for each point that we remove local rather than global object coordinates as described in [21, 31] for triangular meshes.

48


(a)

(b)

(c)

(d)

(e)

(f)

Fig. 9. Reducing a fan cloud (a),(d) to 42% only by considering distances (b),(e) or by considering distances and surface features (c),(f)

Examples and applications of the level-of-detail control combined with various modeling operations are given in [35]. Furthermore, to reduce complexity of a large object, one can speed up the visualization by a selective refinement, i. e., we extract and visualize the point cloud with a fine resolution only in the region of interest. We proceed as above, i. e., we update the positions of all points in the reverse order of their removal, but we only activate, i. e., reinsert the points within the region of interest. In Figure 11, we give an example. To distinctly illustrate the selective refinement, the points are visualized, too. The point cloud in (a) consists of 14379 points and is reduced to 2156 points as shown in (b). In (c), we see the point cloud after a global smoothing. Finally, the face of the represented bunny is selectively refined. The result in (d) consists of only 3031 points, but in the region of interest full detail information is available for further modeling purposes.

7

Refinement

For some applications in modeling and animation, it can be necessary to refine a point cloud beyond its given resolution, or in other words to extend the hierarchical sequence above beyond its finest level.


(a)

(b)

(c)

(d)

49

Fig. 10. Fan cloud (a) reduction based on approximation error (b), distances (c), and entropy (d) To insert a new point r, we determine the point p with highest entropy M (p), its neighbor, say q1 , with highest entropy, and the predecessor or successor of q1 , say q2 , in the neighborhood of p with highest entropy. We compute the weights ω :=

M(p) M(p)+M(q1 )+M(q2 )

ω1 :=

M(q1 ) M(p)+M(q1 )+M(q2 )

ω2 :=

M(q2 ) M(p)+M(q1 )+M(q2 )

and define the initial position of the new point by r := ωp + ω1 q1 + ω2 q2 . In general, r does not lie on the surface represented by the point cloud. Therefore, we move r in the direction of the surface normal. In [15], the mean curvature normal vector is approximated by nr =

1 ∂A , 2A ∂r

where A is the area of the fan of r. One can easily show that this is an affine combination of the form nr = α0 r − αi ri , i

50


(a)

(b)

(c)

(d)

Fig. 11. Selective refinement (d) of a point cloud (a) after reduction (b) and modeling (c) where ri are the neighbors of r. We set r

and r = r :=

=r−

αi ri α0 i

αi r ri + ρ , α 0 r i 2

where p2 + ω1 q 1 2 + ω2 q2 2 . ρ = ω

Since the αi do not change when r moves (not too much) in the normal direction, we get r ≈ ρ. If c, c1 , and c2 are the colors of p, q1 and q2 , the new point r gets the color ωc + ω1 c1 + ω2 c2 .


51

(b)

(a)

Fig. 12. Fan cloud (a) and iterated reduction/refinement (b)

To validate our approach we have applied a reduction-refinement-cycle to the point cloud shown in Figure 12(a) ten times. The reduction-refinement-cycle switches between the levels 100% and 50% without storing detail information. The result in Figure 12(b) illustrates, that the shape of the object remains almost given in [34, 35] to produce unaffected. Note that we used the definition of these Figures. This definition is different but very similar to the above.

8

Selective Refinement

The reduction described in Section 5 is only based on local object information. However, it is also possible to base a reduction decision on global information. In particular, for large objects, it is useful to consider positions relative to the viewer and use a lower resolution for invisible or distant parts of an object. Such a view-dependent reduction or refinement is considered in [26, 27], where Hoppe proposes a view-dependent progressive mesh (VDPM) framework to render large terrain models. Using fan clouds instead of meshes, we present a few improvements to Hoppe’s idea and speak of a view-dependent fan cloud (VDFC). As described in Section 2, Hoppe [26, 27] builds (in a preprocessing step) a forest from a given mesh by successive half-edge collapses. We also build such a forest. However, rather than only using half-edge collapses, we also tried to successively merge the point p with lowest entropy M (p) with its neighbor q with lowest entropy M (q) into the new point r=

M (p)p + M (q)q . M (p) + M (q)

The neighbors of r are the neighbors of p and q except for p and q. With this merge, we obtained better results than with the half-edge collapse, which is also used in [42, 46]. Figure 13 shows an example where we reduced the

52


(a)

(b)

Fig. 13. Reducing the terrain of Figure 10(a) by applying half edge collapses (a) or point merges (b)

(a)

(b)

Fig. 14. Triangulation before (a) and after (b) a point merge

point cloud of Figure 10(a) to 17.6% by half-edge collapses (a) and by the point merges above (b). Note that, here, different points may become differently many neighbors. As mentioned in Section 2, Hoppe allows a vertex split only if certain neighbors exist as during the collapse. Here, there are no such conditions. If, during visualization, a neighbor q of an active point p is not active, we use the active ancestor of q in the forest hierarchy rather than executing further refinement steps. We do not know if fans can become invalid neighborhoods. Figure 14 depicts such a theoretical case for a triangular mesh. Hoppe does not mention this problem and we did not encounter it in all our experiments with fan clouds. The view-dependent refinement of a terrain is based on a selective refinement due to visibility, distance, and viewing direction. Points outside the view frustum are not active and the resolution increases toward the viewpoint. To control the resolution or level of detail, we estimate for each point r in the forest the local deviation of its current triangle fan T from the original fan cloud. When r is obtained by a point merge, we compute the maximum deviation from all its descendants to T and denote it by e(r). Later, when we run the viewdependent visualization, r becomes part of the active point cloud, if it lies in the view frustum and if d(r) emax , e(r) > dmax


53

with d(r) the distance of r from the viewpoint, dmax the range of sight, and emax the maximum approximation error. Note that the triangle fan T used to compute e(r) may differ from the active triangle fan of r used to render the object. However, we used T to be able to compute e(r) in the preprocessing step. Hoppe [26] as well as De Floriani et al. [14] and Xia and Varshney [46] work with triangular meshes, which they consider as piecewise linear functions over R2 . Consequently, they define e(r) to be the maximum functional difference between T and the relevant part of the initial fine mesh, i. e., between two approximations. Also, they compute e(r) during the visualization. Large terrains are usually partitioned into blocks such that adjacent blocks share the points on their common boundary. Under a so called dynamic scene management, only blocks within the range of sight are considered. For stitching blocks together, the same points on their common boundary have to be active. Thus, in the preprocessing step, we exclude boundary points from point merges. Then, after having established the forests for the single blocks, we unite the blocks and perform further point merges including the boundary points. Without these further merges, the boundaries become visible in the shaded visualization. Figure 15 shows a view-dependent refinement of a terrain in point cloud representation. In (a), we show the point cloud and in (b) its shaded visualization. The viewing points used for the view-dependent refinement and the visualization in Figure 15 are different. In Figure 16 these viewing points are the same.

9

Conclusion

In this paper, we have shown – that multiresolution techniques, well-known for triangular meshes, can also be applied to fan clouds, – that entropy-based reduction of fan clouds or triangular meshes maintains characteristic features better than energy-based reduction, – that selective refinement underlies no restrictions with fan clouds, and we have introduced a refinement algorithm to refine a point cloud beyond its initial resolution. Fan clouds can be computed efficiently and faster than triangular meshes. Although, in general, they do not form a 2D manifold, they can be stripped down to triangular 2D mesh manifolds. The number of triangles in a fan cloud is higher than in a triangular mesh. However, storage costs are not higher, and, moreover, there are many duplicates, which we do not render. In fact, in the average, we have only 2.5 different triangles per point and if we remove single redundancies even only 2.1 triangles per point. When using the VDFC framework, the average number of triangles in a selectively refined scene is even smaller, namely approximately 2.0 triangles per point when we start with k = 6. This is about the same as for triangular meshes.

54


(a)

(b)

Fig. 15. View-dependent refinement of terrain using VDFC representation

Fig. 16. A selectively refined terrain from the viewpoint


55

One of the main advantages of fan clouds is their complete localness. Therefore, they are perfectly suited for local refinement, i. e., there is no remeshing necessary in the transition areas between refined and unrefined parts of a point cloud. In particular, we can split any point at any time and need not check its neighbors as in [26]. Also different from [26], we compute the approximation error in a preprocess. Altogether, this implies that we can split points at almost no extra cost during runtime, whereas in [26] the evaluation and preparation costs for a split are four times higher than for its execution. Moreover, since we have no restriction as to any mesh connectivity, we can reduce the number of points to its minimum given by the error tolerance. Another advantage of the VDFC representation above is the ease, with which objects can be added to the scene. The VDFC representation of monuments, places of interest, etc. can simply be added as further separate trees to the forest of the terrain representation. Since neither the VDFC representation nor the error estimation are based on a height field over some parameter plane, these objects (and the terrain as well) can be of arbitrary shape, e. g. it is possible to represent mountain ledges. Thus, the VDFC representation provides a framework for all features of a three-dimensional geographical information system.

References [1] Marc Alexa, Johannes Behr, Daniel Cohen-Or, Shachar Fleishman, David Levin, Claudio T. Silva: Point Set Surfaces. Proceedings of IEEE Conference on Visualization ’01, 21–28, 2001. 39, 41, 47 [2] Maria-Elena Algorri, Francis Schmitt: Surface reconstruction from unstructured 3d data. Computer Graphics Forum, Vol. 15 (1), 47–60, 1996. 43 [3] Marco Attene, Michela Spagnuolo: Automatic surface reconstruction from point sets in space. Computer Graphics Forum, Vol. 19 (3), 457–466, 2000. 43 [4] Chandrajit Bajaj, Fausto Bernardini, Guoliang Xu: Automatic Reconstruction of Surfaces and Scalar Fields from 3D Scans. Proceedings of SIGGRAPH ’95, 109–118, 1995. 43 [5] Fausto Bernardini, Joshua Mittleman, Holly Rushmeier, Cl` audio Silva, Gabriel Taubin: The Ball-Pivoting Algorithm for Surface Reconstruction. IEEE Transactions on Visualization and Computer Graphics, Vol. 5 (4), 349–359, 1999. 43 [6] Wolfgang Boehm, Hartmut Prautzsch: Geometric Concepts for Geometric Design. AK Peters, Wellesley, 1994. 46 [7] Jean-Daniel Boissonat: Geometric Structures for Three-Dimensional Shape Representation. ACM Transactions on Graphics, 266–286, 1984. 43 [8] Swen Campagna, Hans-Peter Seidel: Generating and Displaying Progressive Meshes. Proceedings of 3D Image Analysis and Synthesis, Erlangen, 35–42, 1997. 40 [9] Baoquin Chen, Minh Xuan Nguyen: POP: A Hybrid Point and Polygon Rendering System for Large Data Proceedings of IEEE Conference on Visualization ’01, 45– 52, 2001. 47 [10] Jonathan D. Cohen, Marc Olano, Dinesh Manocha: Appearance-Preserving Simplification. Proceedings of SIGGRAPH ’98, 115–122, 1998. 40

56


[11] Jonathan D. Cohen, Daniel G. Aliaga, Weiqiang Zhang: Hybrid Simplification: Combining Multi-resolution Polygon and Point Rendering Proceedings of IEEE Conference on Visualization ’01, 37–44, 2001. 47 [12] Patricia Crossno, Edward Angel: Spiraling Edge: Fast Surface Reconstruction from Partially Organized Sample Points Proceedings of IEEE Conference on Visualization ’99, 1999. 43, 44 [13] Brian Curless, Marc Levoy: A Volumetric Method for Building Complex Models from Range Images. Proceedings of SIGGRAPH ’96, New Orleans, LA, 4-9 August 1996. 43 [14] L. De Floriani, P. Magillo, E. Puppo: VARIANT: A System for Terrain Modeling at Variable Resolution. Geoinformatica, Vol. 4(3), 287–315, 2000. 40, 53 [15] Mathieu Desbrun, Mark Meyer, Peter Schr¨ oder, Alan Barr: Implicit Fairing of Irregular Meshes using Diffusion and Curvature Flow. Proceedings of SIGGRAPH ’99, 1999. 49 [16] M. Duchaineau, M. Wolinsky, D. Sigeti, M. Miller, C. Aldrich, M. MineevWeinstein: ROAMing terrain: real-time optimally adapting meshes. Proceedings of IEEE Visualization ’97, 81–88, 1997. 40 [17] Matthias Eck, Tony DeRose, Tom Duchamp, Hugues Hoppe, Michael Lounsbery, Werner Stuetzle: Multiresolution Analysis of Arbitrary Meshes. Proceedings of SIGGRAPH ’95, 1995. 40 [18] H. Edelsbrunner, E. P. M¨ ucke: Threedimensional alpha shapes. ACM Transactions on Computer Graphics, Vol. 13 (1), 43–72, 1994. 43 [19] M. Gopi, S. Krishnan, C. T. Silva: Surface Reconstruction based on Lower Dimensional Localized Delaunay Triangulation. Computer Graphics Forum, Vol. 19 (3), 2000. 43 [20] M. Gross, O. Staadt, R. Gatti: Efficient Triangular Surface Approximations using Wavelets and Quadtree Data Structures. IEEE Transactions on Visualization and Computer Graphics, Vol. 2 (2), 130–143, 1996. 40 [21] Igor Guskov, Wim Sweldens, Peter Schr¨ oder: Multiresolution Signal Processing for Meshes. Proceedings of SIGGRAPH ’99, 1999. 40, 47 [22] Bernd Hamann: A data reduction scheme for triangulated surfaces. Computer Aided Geometric Design, Vol. 11, 197–214, 1994. 40 [23] Hugues Hoppe, Tony DeRose, Tom Duchamp, John McDonald, Werner Stuetzle: Surface Reconstruction from Unorganized Points. Computer Graphics, Vol. 26, 71–78, 1992. 43 [24] Hugues Hoppe, Tony DeRose, Tom Duchamp, John McDonald, Werner Stuetzle: Mesh Optimization. Computer Graphics Proceedings, Annual Conference Series, Vol. 7, 19–26, 1993. 40 [25] Hugues Hoppe: Progressive meshes. Proceedings of SIGGRAPH ’96, 99–108, 1996. 40, 41 [26] Hugues Hoppe: View-dependent refinement of progressive meshes. Proceedings of SIGGRAPH ’97, 189–198, 1997. 40, 41, 51, 53, 55 [27] Hugues Hoppe: Smooth View-Dependent Level-of-Detail Control and its Application to Terrain Rendering. IEEE Visualization, 35–42., 1998. 40, 41, 51 [28] Andreas Hubeli, Markus Gross: Multiresolution Methods for Non-Manifold Models. IEEE Transaction on Visualization and Computer Graphics, 2001. 40 [29] Aravind Kalaiah, Amitabh Varshney: Differential Point Rendering. Rendering Techniques ’01 , S. J. Gortler and K. Myszkowski (edts.), Springer-Verlag, 139– 150, 2001 39, 41, 47 [30] Leif Kobbelt, Swen Campagna, Hans-Peter Seidel: Mesh Reduction Revisited. Universit”at Erlangen, 1997. 40, 45


57

[31] Leif Kobbelt, Swen Campagna, Jens Vorsatz, Hans-Peter Seidel: Interactive MultiResolution Modeling on Arbitrary Meshes. Proceedings of SIGGRAPH ’98, 1998. 40, 47 [32] Leif Kobbelt: Multiresolution techniques. To appear in: Farin, Hoschek, Kim (Edts.), ”‘Handbook of Computer Aided Geometric Design”’, Elsevier. 40 [33] P. Lindstrom, D. Koller, W. Ribarsky, L. Hodges, N. Faust, G. Turner: Real-time, continuous level of detail rendering of height fields. Proceedings of SIGGRAPH ’96, 109–118, 1996. 40 [34] Lars Linsen, Hartmut Prautzsch: Local Versus Global Triangulations. Proceedings of Eurographics ’01, Short Presentations, Manchester, 257–263, 2001. 41, 42, 51 [35] Lars Linsen: Oberfl¨ achenrepr¨ asentation durch Punktwolken. Dissertation, Universit¨ at Karlsruhe, Verlag Shaker, Aachen, 2001. 41, 48, 51 [36] Robert Mencl, Heinrich M¨ uller: Graph-Based Surface Reconstruction Using Structures in Scattered Point Sets. Proceedings of Computer Graphics International ’98, Hannover, 1998. 43 [37] Robert Mencl, Heinrich M¨ uller: Interpolation and Approximation of Surfaces from Three-Dimensional Scattered Data Points. State of the Art Report for EUROGRAPHICS ’98, Lisbon, 1998. 43 [38] Renato Pajarola: Large scale Terrain Visualization using the Restricted Quadtree Triangulation. Technical Report, ETH Z¨ urich, Switzerland, 1998. 40 [39] Mark Pauly, Markus Gross: Spectral Processing of Point-Sampled Geometry. Proceedings of SIGGRAPH ’01, 2001. 42 [40] Hanspeter Pfister, Matthias Zwicker, Jeroen van Baar, Markus Gross: Surfels: Surface Elements as Rendering Primitives. Proceedings of SIGGRAPH ’00, 2000. 39, 41, 47 [41] Stephan Preuß: Von Punktwolken zu Dreiecksnetzen. M. S. thesis, Universit¨ at Karlsruhe, Germany, 2002. 44 [42] Chris Prince: Progressive Meshes for Large Models of Arbitrary Topology. M. S. thesis, University of Washington, Seattle, 2000. 51 [43] S. R¨ ottger, W. Heidrich, P. Slusallek, H.-P. Seidel: Real-Time Generation of Continuous Levels of Detail for Height Fields. Proceedings of 6th International Conference in Central Europe on Computer Graphics and Visualization ’98, 315– 322, 1998. 40 [44] Szymon Rusinkiewicz, Marc Levoy: QSplat: A Multiresolution Point Rendering System for Large Meshes. Proceedings of SIGGRAPH ’00, 2000. 39, 41, 47 [45] Eric J. Stollnitz, Tony D. DeRose, David H. Salesin: Wavelets for Computer Graphics: Theory and Applications. The Morgan Kaufmann Series in Computer Graphics and Geometric Modeling, Brian A. Barsky, Series Editor, 1996. 40 [46] Julie C. Xia, Amitabh Varshney: Dynamic View-Dependent Simplification for Polygonal Models. Proceedings of the IEEE Visualization ’96, 327–334, 1996. 40, 41, 51, 53 [47] Matthias Zwicker, Hanspeter Pfister, Jeroen van Baar, Markus Gross: Surface Splatting. Proceedings of SIGGRAPH ’01, 2001. 39, 41, 47

Combinatorial and Geometric Problems Related to Digital Halftoning Tetsuo Asano1 , Naoki Katoh2 , Koji Obokata1 , and Takeshi Tokuyama3 1 2 3

JAIST, Tatsunokuchi, Japan Kyoto University, Kyoto, Japan Tohoku Iniversity, Sendai, Japan

Abstract. Digital halftoning is a technique to convert a continuoustone image into a binary image consisting of black and white dots. It is an important technique for printing machines and printers to output an image with few intensity levels or colors which looks similar to an input image. The purposes of this paper are to reveal that there are a number of problems related to combinatorial and computational geometry and to present some solutions or clues to those problems.

1

Introduction

The quality of color printers has been drastically improved in recent years, mainly based on the development of fine control mechanism. On the other hand, there seems to be no great invention on the software side of the printing technology. What is required is a technique to convert a continuous-tone image into a binary image consisting of black and white dots that looks similar to the input image. Theoretically speaking, the problem is how to approximate an input continuoustone image by a binary-tone image. Since this is one of the central techniques in computer vision and computer graphics, a great number of algorithms have been proposed (see, e.g., [6, 8, 9, 17]) with several theoretical results by the authors [1, 2, 3, 4, 5]. The purpose of this paper is to reveal that some notions on combinatorial and computational geometry such as Voronoi diagram, discrepancy, and dispersion are related to digital halftoning. Based on those notions we shed light on digital halftoning from different directions.

2

Known Basic Algorithms

Throughout the paper we put the following assumptions to simplify the discussion. We take as an input image an N × N real-valued matrix A = (aij ), 0 ≤ aij ≤ 1 for each (i, j) and output a binary matrix B = (bij ) of the same size. Usually, black has the intensity level 0 while white has 1. For color images, we iterate the same halftoning process three times for each of R (Red), G (Green), and B (Blue) components. Since digital halftoning is a widely used technique, a great number of algorithms have been proposed so far (refer to the books [10, 17]). First we briefly T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 58–71, 2003. c Springer-Verlag Berlin Heidelberg 2003

Combinatorial and Geometric Problems Related to Digital Halftoning

59

describe several representative halftoning algorithms with their variations of algorithmic flavor. 2.1

Simple Thresholding

Given an N × N array A of real numbers between 0 and 1, we wish to construct a binary array B of the same size which looks similar to A, where entry values represent light intensity levels at corresponding locations. The most naive method for obtaining B is simply to binarize each input value by a fixed threshold, say 0.5. It is simplest, but the quality of the output image is worst since any uniform gray region becomes totally white or totally black. The most important is how to represent intermediate intensities. 2.2

Ordered Dither

Instead of using a fixed threshold over an entire image, this method uses different thresholds. A simple way of implementing this idea is as follows: We prepare an M × M matrix of integers from 1 to M 2 . This matrix (dither array) is tiled periodically to cover the image. Each pixel in the image is compared with the corresponding threshold from the dither array to decide whether a dot will be placed at that location. Fig. 1 shows the dither matrix given by Bayer [6]. 2.3

Error Diffusion

The dither algorithm is designed to preserve the average intensity level between input and output images. There is another standard algorithm called “error diffusion” that also possesses the same property by propagating the quantization errors to unprocessed neighboring pixels according to some fixed ratios. More precisely, pixels are processed in a raster order, from left to right and top to bottom. Each pixel level is compared with a fixed threshold, 0.5 and round it up if it is greater than or equal to the threshold and round it down otherwise. The quantization error caused by the rounding is diffused over the unprocessed pixels around it with fixed ratios. For example, if a pixel level is 0.7, it is rounded up

1 49 13 61 4 52 16 64

33 17 45 29 36 20 48 32

9 57 5 53 12 60 8 56

41 25 37 21 44 28 40 24

3 51 15 63 2 50 14 62

35 19 47 31 34 18 46 30

11 59 7 55 10 58 6 54

43 27 39 23 42 26 38 22

Fig. 1. 8 × 8 dither matrix by Bayer [6]

60

Tetsuo Asano et al. • 7/16 3/16 5/16 1/16

Fig. 2. Diffusion ratios in Error Diffusion by Floyd and Steinberg to 1 and the error −0.3 is diffused to the unprocessed pixels nearby. The ratios suggested by Floyd and Steinberg in their paper [8] are shown in Fig. 2: This method certainly preserves the average intensity level because the rounding error is distributed to neighboring pixels. When the process terminates, the difference between the sums of intensity levels in the input and output images is at most 0.5. This method not only preserves the average intensity level but also gives excellent image quality in many cases, but it tends to produce visible artifacts in an area of uniform intensity, which are caused by the fixed error-diffusing coefficients. Fig. 3 compares the three algorithms, simple thresholding, ordered dither and error diffusion, by their output images.

3 3.1

Variation of Known Algorithms with Related Problems Variation of Simple Thresholding

The serious drawback of the simple thresholding is poor expression of intermediate intensity due to its independent process at each pixel and its use of a fixed threshold. One of the method to improve the expression is to use random thresholds. Precisely, we generate white gaussian noise over an input image and use the noise as threshold. This method is considered as variation of ordered dither with a dither matrix defined by random numbers. Thus, theoretically speaking, the expected average intensity level of the output image is expected to be equal to that of input image. The same idea is popular in randomized algorithms under a different name, i.e., randomized rounding [14, 16], in which a real number x, 0 ≤ x ≤ 1, is

Fig. 3. Output images: Simple thresholding, ordered dither, and error diffusion (color images)


61

rounded up with probability x. It is one of the standard techniques in randomized algorithms. 3.2

Variation of Ordered Dither Algorithm

The previous subsection described a rounding algorithm using variable random numbers as thresholds to generalize the simple thresholding that uses one fixed threshold. We can use a carefully designed table of random numbers instead of generating a random number for each pixel. Dither matrix corresponds to this table of random numbers. Small table size tends to lead to disadvantage of visible artifacts. So, the largest possible table size would be better. In fact, there is an algorithm along this idea, which is known as a blue-noise mask algorithm [12, 18, 19] in general. This algorithm uses a large dither matrix (bluenoise mask) of size, say 256 × 256. Properties of Dither Matrix The performance of the ordered dither algorithm heavily depends on a dither matrix used. We have known how to construct the dither matrix. Then what is a merit to use this dither matrix? In other words, does it optimize anything? If the purpose is only to distribute numbers 1 through 22k over the 2k × 2k matrix, there are a number of different ways. Imagine an artificial image of gradually increasing intensity from left to right. During the transition from dark to bright, the number of white dots should gradually increase. This means that for any number i between 1 and 22k those entries having numbers greater than i must be as uniformly distributed as possible over the dither matrix. The uniformity can be measured in several different ways. One measure is based on the ratio between the smallest and largest diameters of empty circles containing no point in their interior but at least two points on the circles. The smallest empty circle is attained by the minimum pairwise distance. The largest one is defined by the largest circle passing through three points while containing no point in its interior with its center lying in the convex hull of the point set. Another possible measure is based on the notion of ”discrepancy” which is related to the difference between the area and the relative number of white dots. The above regular grid-like construction of the dither matrix is optimal in the former measure since it is constructed under the notion of incremental Voronoi insertion. An optimal dither matrix under the former measure is designed as follows. Before construction we have to note that dither matrix is used to cover an entire image by repeatedly arranging the matrix. First we choose an arbitrary entry, say, the upper left corner of the matrix, to assign number 1. Because of the periodicity, it means that we have placed points numbered 1 on regular grids (8i, 8j), i, j = 0, 1, . . .. The entry 2 must be placed at a grid point farthest from the points numbered 1. Such a place coincides with a Voronoi vertex of the Voronoi diagram for the set of points numbered 1. Similarly, the location of the entry 3 should be chosen among the Voronoi vertices for the Voronoi diagram of the set of points numbered 1 or 2. This strategy is called “incremental Voronoi

62

Tetsuo Asano et al.

insertion” which is rather easy to be implemented. If we resolve ties appropriately we obtain the dither matrix. Unfortunately, this dither matrix is not good enough in practice. What is wrong? The measure may be wrong. That is, the measure based on the ratio between the minimum pairwise distance and the diameter of the maximum empty circle may not be good enough to reflect the uniformity of point distribution. The latter measure based on the discrepancy suggested above seems to be more promising. In the measure we take a number of regions. If points are uniformly distributed, the point density is roughly the same in each such region. In the discrepancy measure we can take regions of arbitrary shapes. The former measure based on the minimum pairwise distance is roughly equal to the discrepancy measure for a family of circular regions. In this sense the discrepancy measure is a generalization of the former measure. To define the discrepancy measure, we introduce a family F of regions over an image. For each region R in F , let A(R) denote the area of R and card(R) denote the number of points in R. Then, we take the difference D(R) = |n · A(R) − card(R)|, as the discrepancy for the region R, assuming that the area of the whole image is 1. √ √ Consider a regular pattern in which n points are placed in a n × n grid. Take a rectangular region R defined √ by two rows of points. Then, the area of √ the the rectangle is (1/ n) × 1 = 1/ n. If we locate the rectangle so that √ two sides exactly coincide with two rows of points, we have card(R) = 2 n. √ n. Thus, Otherwise, it contains only one of rows of points, and so card(R) = √ √ √ √ we have D(R) = |n/ n − 2 n| = n in the former case and D(R) = |n/ n − √ n| = 0 in√the latter case. In fact, we can prove that the √ maximum value of D(R) is O( n). Furthermore, it is known that it remains O( n) when n points are randomly distributed. However, there are deterministic algorithms which achieves the discrepancy O(log n). Refer to the textbooks on discrepancy by Chazelle[7] and Matouˇsek[11]. Rotation of Dither Matrix Another related problem comes from the human perception. An interesting feature of human perception is that horizontal and vertical patterns are more sensitive to human eyes than skewed patterns [15]. This fact suggests us of rotating a dither matrix. Then, the problem is how to design such a rotated pattern consisting of M 2 elements. This is not so easy since a rotated pattern must be tiled to cover the entire image and the area (number of entries) is fixed. Fig. 4 illustrates how a rotated dither matrix covers the entire plane. We shall explain how to design a pattern which satisfies the following conditions:


63

(1) Area Condition: The rotated matrix must have the same number of entries (or grid points) as that of the original matrix, and those grid points form a connected cluster without any hole. (2) Tiling Condition: The rotated matrix must be tiled to cover the entire grid plane, that is, the entire plane must be tiled without any gap by repeated placements of the same pattern. (3) Angle Condition: The rotated matrix must be bounded by four digital lines segments. The angle of those segments from the axes should be close enough to a given angle. Furthermore, the angle between two such segments should be almost perpendicular. The most important observation behind the scheme for achieving a rotation is the following Pick’s theorem [13]. [Pick’s Theorem] The area of any simple polygon P in a grid (not necessarily convex) whose vertices are lattice points is given by area(P ) = Lin (P ) + Lbd (P )/2 − 1, where Lin (P ) denotes the number of grid points in the interior of the polygon P and Lbd (R) that of grid points on the boundary. Our objective is to design a rotated square region R consisting of M 2 grid points rotated approximately by an angle θ. We have four vertices denoted by A, B, C and D, as shown in Fig. 5. Among the four vertices only the vertex A is included in the rotated pattern. Since this is a tiling pattern, the other three vertices become the positions at which the A corner of the pattern R is located. The rotated square R has four sides, AB, AC, BD and CD. The grid points on the lower sides AB and AC are included into the rotated pattern R while those grid points on the upper sides BC and CD are not. Here note that by symmetry the number of grid points on the lower sides is equal to that of grid points on the upper sides. See Fig. 5 for illustration.

Fig. 4. Tiling the entire grid by a pattern

64

Tetsuo Asano et al.

D

C B d

b c

A

a

Fig. 5. A tiling pattern R = ABCD and four parameters a, b, c, d defining it Then, the number of grid points included in the rotated pattern R is given by the sum of the number of grid points lying in the interior of the rotated square R = ABCD, half the number of grid points on the four sides excluding the vertices, and 1 for the vertex A. Thus, by the Pick’s theorem, the number of grid points in the rotated square pattern R is 1 Lin (R) + (Lbd (R) − 4) + 1 2 1 = Lin (R) + Lbd (R) − 1 = area(R). 2 Now, given a size of a rotated pattern (the number of grid points) and an angle θ, we can construct such a rotated pattern as follows: Designing a Rotated Pattern (1) Find four integers a, b, c, and d such that ad + bc = M 2 , b c tan θ. a d (2) Determine a quadrangle R = ABCD such that 1. The bottom, right, left, and top vertices of R are A, B, C, and D. 2. The coordinates of the vertices B, C, D are determined by (xA +a, yA +b), (xA − c, yA + d), (xA − c + a, yA + d + b), respectively. 3. The grid points on the lower side are included in R. 4. The grid points on the upper side are not included in R. 5. Among the four vertices, only the bottom vertex A is included in R. Lemma 1. The rotated pattern R designed above satisfies the three conditions listed above.


65

Proof. Area Condition is satisfied since the area of the rotated pattern R is (a + c)(b + d) − ab − cd = ad + bc. Tiling Condition: The pattern can tile the entire grid. When we translate R so that the A corner coincides with the location of the vertex B, the side AC of the translated pattern coincides with the side BD of the pattern in the original location. By the definition of the rotated pattern, they coincide with each other and those grid points on the side are included only in the translated pattern. It is just the same for the other sides. Finally, there is no collision of vertices since we have chosen only one vertex among the four vertices. Angle Condition: We can choose the four parameters a, b, c, d so that the slopes of the sides AB and CD are roughly equal to tan θ and those of the sides AC and BD are roughly equal to tan(θ − π/2). Thus, we can choose the best possible parameter values among those values satisfying the area condition. Blue-Noise Mask: A Huge Dither Matrix One way to remove the artifact texture pattern of the Ordered dither algorithm is to rotate the dither matrix. There is another way. Just use a huge dither matrix of size, say, 256 × 256. If we carefully design such a huge dither matrix, artifact textures are not visible anymore. The problems are how to design such a huge dither matrix and the large storage requirement. Such a huge dither matrix is generally referred to as a blue-noise mask. Important is to remove periodicity. Consider a dither matrix of size 256 × 256. When we have 256 intensity levels, each number between 1 and 256 appears 256 times in the matrix. For each number p between 1 and 256, those entries numbered 1 through p should be distributed as uniformly as possible. A desired pattern is not a regular one but somewhat random-looking pattern as is explained concerning discrepancy. There are several ways to incorporate randomness. One such method is the one called “void-and-cluster” algorithm [19]. The algorithm starts with a random distribution of points and gradually tries to reform the pattern for uniform distribution. There are two factors to break uniformity: cluster parts in which many points are located closely to each other and void parts in which points are sparsely distributed. An idea to achieve uniform distribution is to move a point in a cluster to the center of a void. Such an operation is well supported in computational geometry. Given n points, the Voronoi diagram is constructed in O(n log n) time. When a point is surrounded by many points, its associated Voronoi region tends to be small. Thus, cluster parts are found by checking areas of Voronoi regions. On the other hand, void parts correspond to sparse parts. Such locations are found as centers of large empty circles in which no point is contained. A largest empty circle can be found in linear time in two ways, one based on linear programming and the other on randomization. 3.3

Dispersion Problem

Now it finds that the problem of designing a good blue-noise mask is closely related to the following combinatorial problem.

66

Tetsuo Asano et al.

Dispersion Problem We want to insert a predermined number of points one by one as much uniformly as possible in some given domain at any instance. The uniformity is measured by the maximum ratio of the maximum gap over the minimum one. When the maximum and minimum gaps after inserting k points are denoted by Gk and gk , respectively, the ratio rk is defined by rk = Gk /gk . The objective here is to minimize the maximum ratio Rn = max(r1 , r2 , . . . , rn ), which is referred to as the dispersion of the point sequence. There may be several different manners to define a gap. In the d-dimensional space we define it by the radius of a ball with its center being in the domain which contains d + 1 points on the surface but no point in its interior. The problem here is to find upper and lower bounds on the maximum ratio Rn in each dimension. General Approximation Algorithm We assume that a domain is given by a convex polyhedron and its vertices are included as an initial set of points. Then, we add points one by one. A simple way to insert points uniformly is a so-called incremental Voronoi insertion, which inserts a point at an interior vertex of a current Voronoi diagram that has the largest clearance around it. It is not so hard to check the performance of the algorithm. In 1-D, starting with a unit interval [0, 1] with an initial set {0, 1}, we put a new point at the center point of the the current largest gap [0, 1]. Thus, a sequence of points generated is {p1 = 1/2, p2 = 1/4, p3 = 3/4, p4 = 1/8, . . .}. Then, starting with a gap G0 = g0 = 1, the largest and smallest gaps when we inserted the k-th point are Gk = 1/2lg(k+1) and gk = 1/2lg k+1 , respectively. Thus, the ratio rk is 1 if k = 2i − 1 for some integer i and 2 otherwise. Thus, the maximum ratio, i.e., the dispersion of the 1-D Voronoi insertion, is 2. What about the 2-d case? We start with a unit square with an initial point set {(0, 0), (0, 1), (1, 0), (1, 1)}. Then, we do the incremental Voronoi insertion. It proceeds similarly as the one-dimensional case. That is, the ratio is either 1 or √ √ 2. Thus, the maximum ratio is 2. One-Dimensional Case Our domain here is a unit interval [0, 1]. The two extremal points 0 and 1 are assumed to be included in the set. We can show that there is a strategy better than the incremental Voronoi insertion. As an exercise, let us consider the case when we insert 3 points. Unlike the incremental Voronoi insertion, we put the first point p1 so that the unit interval is split unevenly. Then, we put the second point p2 to split the longer interval. Now we split the current longest interval into two by putting the third point p3 . This process is represented by a binary tree rooted at the unit interval. It is followed by two intervals x1 and x2 , where x1 + x2 = 1 with x1 > x2 . Then, x1 has branches to x3 and x4 with x3 + x4 = x1 and x3 ≥ x4 . The node x2 is also followed by two nodes x5 and x6 such that x5 + x6 = x2 and x5 ≥ x6 . Then, the ratios are r1 = x1 /x2 , r2 = x2 /x4 , and r3 = x3 /x6 . Since the intervals x3 , . . . , x6 are not split anymore, the partition of x1 into x3 and x4 and that of x2 into x5 and x6 should be bisections at their center points to minimize


67

the ratios, that is, x3 = x4 and x5 = x6 . Now, let us denote x5 and x3 by y1 and y2 , respectively. Then, x2 = 2y1 and x1 = 2y2 . Therefore, r2 = x2 /x4 = 2y1 /y2 and r3 = x3 /x6 = y2 /y1 . Thus, the maximum ratio R3 is minimized when r2 and r3 are equal, and it is given by √ √ 2y1 y2 = 2. (1) R3 = r2 · r3 = y2 y1 We can show that this bound is optimal, that is, there is no sequence of three points to achieve a better ratio. Assume w.l.o.g. √ that 0 < p1 ≤ 1/2. Then, √ the ratio r1 is given by (1 − p1 )/p1 , which is at least 2. Thus, 1/2 ≥ p1 ≥ 2 − 1. We have to choose p2 in the interval [p1 , 1] to have a ratio at most r1 . So, again w.o.l.g. we assume that p2 − p1 ≥ (1 − p1 )/2, and thus p2 ≥ (1 + p1 )/2. Now, the minimum gap is between √ p2 and 1, and thus the ratio√r2 is given by p1 /(1 − p2 ), p1 / 2. Combining which is bounded by 2. This √ leads to p2 ≤ 1 − √ √ √the results, √ we have p2 ≥ (1+p√1 )/2 ≥ 2/2, and p2 ≤ 1−p1 / 2 ≤ 1−( 2−1)/ 2 = 2/2. √ Thus, p2 must be 2/2. It is also seen √ that p1 must be 2 − 1. So, whatever p3 is, the ratio R3 cannot be better than 2. We can generalize the result above in the following forms. Lemma 2. There is a sequence of real numbers (p1 , p2 , . . . , pn ) in the unit interval [0, 1] with the dispersion Rn = 2n/2/(n/2+1) .

(2)

Proof. First we consider the case when n is an even number, that is, n = 2k for some integer k. The strategy to insert points is to bisect a current longest interval each time unevenly if the resulting intervals will be further partitioned and evenly otherwise. We rename the last 2k intervals as follows: x4k−1 = x4k

−→ y1 ,

x4k−3 = x4k−2 .. .. . .

−→ y2 ,

x2k+1 = x2k+2

−→ yk .

Then, the ratios are given by r2k =

x2k 2y1 2y2 2yk , r2k−1 = , r2k−2 = , · · · , rk = . y1 y2 y3 x2k

The maximum ratio Rn = max{r2k , r2k−1 , . . . , rk } is minimized when r2k = r2k−1 = · · · = rk . Hence, we have Rn = (

x2k 2y1 2y2 2yk 1/(k+1) · · ··· ) = 2k/(k+1) , y1 y2 y3 x2k

68

Tetsuo Asano et al.

which is equal to Rn = 2n/2/(n/2+1) . We have a similar proof when n is an odd number, that is, n = 2k + 1 for some k. In this case we have intervals x1 through x4k+2 . We set x4k+1 = x4k+2

−→ y1 ,

x4k−1 = x4k −→ y2 , .. .. . . x2k+1 = x2k+2 −→ yk+1 .

Then, the ratios are as follows: r2k+1 =

yk+1 2y1 2y2 2yk , r2k = , r2k−1 = , · · · , rk+1 = . y1 y2 y3 yk+1

The maximum ratio Rn = max{r2k+1 , r2k , . . . , rk+1 } is minimized when they are all equal. Hence, we have

Rn = (

yk+1 2y1 2y2 2yk 1/(k+1) · · ··· ) = 2k/(k+1) , y1 y2 y3 yk+1

which is again equal to Rn = 2n/2/(n/2+1) . 3.4

Formulation by Integer Linear Programming

The problem of optimally distributing k points over a square grid of area n can be formulated as an Integer Linear Program. The problem P (n, k) is, given a grid of area n, to choose k(≤ n) lattice points on the grid so that the minimum pairwise distance is maximized. To solve this problem we consider a slightly different problem P (n, d); Given a grid of area n and a real number d, choose as many lattice points in the grid as possible so that their minimum pairwise distance is greater than d. Since there are only O(n2 ) possible values for the pairwise distances, if we solve the problem P (n, d) for O(log n) different discrete values of d we can obtain a solution to the original problem P (n, k). To solve a problem P (n, d) using an integer linear program, we define a binary variable xi,j for each lattice point (i, j) in the grid of area n, which is 1 if we choose the corresponding lattice point and 0 otherwise. Then, the problem is to maximize the sum xi,j , the number of grid points chosen, under the constraint that there is no pair of points with distance ≤ d. The corresponding set of linear inequalities are obtained if we enumerate all possible pairs of lattice points with distance ≤ d. In the continuous plane it is rather easy to define a region that contains every possible vector of length at most d. Take two points a and b of distance d and draw two circles of radius d centered at a and b. Then, let c be one of the intersections of the two circles and draw a circle of radius d centered at c. Now,


69

c

a

b

Fig. 6. A region containing every possible vector of length at most d

c

a

b

Fig. 7. A discrete region defined by three circles

the intersection of these three disks is the region required that contains every possible vector of length at most d in it. Note that the three points form a regular triangle. See Fig.6 for illustration. In our case we want to have a region that contains all possible integer vectors of length at most d, where an integer vector is defined by a pair of lattice points. Unfortunately, the above method by three circles does not apply to the discrete case. Consider the case of d = 4. We take two points a and b of distance 4. Then, the corresponding circles intersect at a point c that is just in the middle of the two lattice. If we draw a circle of radius 4 centered at c, the resulting region does not contain a vertical integer vector of length 4 in it. See Fig.7. To resolve the difficulty we use at most two regions instead of one. Given a distance d, we take its integer part d∗ = d. Then, we take a horizontal integer vector (a, b) of length d∗ . Let Rh be a set of all lattice points in the intersection of the half plane above the line through a and b and the two disks

70

Tetsuo Asano et al.

Rh

Rv

Fig. 8. Two discrete regions containing every possible vector of length at most 4 of radius d centered at a and b. Similarly we define a set Rv by rotating Rh by 90 degrees around the point a. Fig.8 shows two such regions for d = 4. It is easy to prove that every integer vector of length at most d is contained in one of the regions. By the construction, if we take the lattice point a as an initial point, Rh contains every integer vector of angle in [0, π/3]. By symmetry, it also contains one of angle in [π, 4π/3]. If we fix the lattice point b as one of endpoints of an integer vector, Rh contains every integer vector of angle in [2π/3, π] or [5π/3, 2π]. We have the same observation for the region Rv , which covers the angular intervals [π/6, π/2], [7π/6, 3π/2], [π/2, 5π/6], and [3π/2, 11π/6].

4

Conclusion

In this paper we have shown that several problems on discrete and computational geometry are related to digital halftoning. There are a number of open problems. Many of them are related to conversion from the continuous plane to discrete plane. One such example is a discrete dispersion problem in which points must be laid at some lattice points. Many things are left unknown for such discrete versions of the problems.

Acknowledgment The first author expresses his sincere thanks to Naoki Fujikawa (former JAIST student), Tomomi Matsui (Tokyo Univ.), Hisao Tamaki (Meiji Univ.), Hiroshi Nagamochi (Toyohashi Univ. Tech.), and Nobuaki Usui (Fujitsu Lab.) who are coauthors of the author’s papers on digital halftoning. He also would like to thank Hiro Ito (Kyoto Univ.), David Mount (Maryland Univ., U.S.A.), Masashi Kiyomi (Tokyo Univ.), and Koji Nakano (JAIST). The work has been partially supported by the Scientific Grant-in-Aid by the Ministry of Education, Culture, Sports, Science and Technology of Japan and the Kayamori Foundation of Informational Science and Advancement.


71

References [1] T. Asano: “Digital Halftoning Algorithm Based on Random Space-Filling Curve,” IEICE Trans. on Fundamentals, Vol.E82-A, No.3, pp.553-556, Medgeh 1999. 58 [2] T. Asano, K. Obokata, N. Katoh, and T. Tokuyama: “Matrix rounding under the Lp -discrepancy measure and its application to digital halftoning,” Proc. ACMSIAM Symposium on Discrete Algorithms, pp.896-904, San Francisco, 2002. 58 [3] T. Asano, T. Matsui, and T. Tokuyama: “Optimal Roundings of Sequences and Matrices,” Nordic Journal of Computing, Vol.7, No.3, pp.241-256, Fall 2000. 58 [4] T. Asano and T. Tokuyama: “How to Color a Checkerboard with a Given Distribution — Matrix Rounding Achieving Low 2 × 2- Discrepancy,” Proc. ISAAC01, pp. 636-648, Christchurch, 2001. 58 [5] T. Asano, D. Ranjan and T. Roos: “Digital halftoning algorithms based on optimization criteria and their experimental evaluation,” IEICE Trans. Fundamentals, Vol. E79-A, No. 4, pp.524-532, April 1996. 58 [6] B. E. Bayer: “An optimum method for two-level rendition of continuous-tone pictures,” Conference Record, IEEE International Conference on Communications, 1, pp.(26-11)–(26-15), 1973. 58, 59 [7] B. Chazelle: “The Discrepancy Method: Randomness and Complexity,” Cambridge University Press, 2000. 62 [8] R. W. Floyd and L. Steinberg: “An adaptive algorithm for spatial gray scale,” SID 75 Digest, Society for Information Display, pp.36–37, 1975. 58, 60 [9] D. E. Knuth: “Digital halftones by dot diffusion,” ACM Trans. Graphics, 6-4, pp.245–273, 1987. 58 [10] D. L. Lau and G. R. Arce: “Modern Digital Halftoning,” Marcel Dekker, Inc., New York, 2001. 58 [11] J. Matouˇsek: “Geometric Discrepancy,” Springer, 1991. 62 [12] T. Mitsa and K. J. Parker: “Digital halftoning technique using a blue-noise mask,” J. Opt. Soc. Am., A/Vol.9, No.11, pp.1920-1929, 1992. 61 [13] R. Morelli: “Pick’s theorem and Todd class of a toric variety,” Adv. Math., 100, pp.183-231, 1993. 63 [14] R. Motwani and P. Raghavan: “Randomized Algorithms,” Cambridge University Press, 1995. 60 [15] V. Ostromoukhov, R. D. Hersh, and I. Amidror: “Rotated Dispersed Dither: a New Technique for Digital Halftoning,” Proc. of SIGGRAPH ’94, pp.123-130, 1994. 62 [16] P. Raghavan and C. D. Thompson: “Randomized rounding,” Combinatorica, 7, pp.365-374, 1987. 60 [17] R. Ulichney: Digital halftoning, MIT Press, 1987. 58 [18] R. A. Ulichney: “Dithering with blue noise,” Proc. IEEE, 76, 1, pp.56-79, 1988. 61 [19] R. Ulichney: “The void-and-cluster method for dither array generation,” IS&T/SPIE Symposium on Electronic Imaging Science and Technology, Proceedings of Conf. Human Vision, Visual Processing and Digital Display IV, (Eds. Allebach, John Wiley), SPIE vol.1913, pp.332-343, 1993. 61, 65

All Points Considered: A Maximum Likelihood Method for Motion Recovery Daniel Keren1 , Ilan Shimshoni2 , Liran Goshen2 , and Michael Werman3 1

3

Department of Computer Science, University of Haifa Haifa 31905, Israel [email protected] 2 Faculty of Industrial Engineering, Technion Technion City 32000, Israel {ilans,lirang}@ie.technion.ac.il School of Computer Science and Engineering, The Hebrew University of Jerusalem Jerusalem 91904, Israel [email protected]

Abstract. This paper addresses the problem of motion parameter recovery. A novel paradigm is offered to this problem, which computes a maximum likelihood (ML) estimate. The main novelty is that all domain-range point combinations are considered, as opposed to a single “optimal” combination. While this involves the optimization of nontrivial cost functions, the results are superior to those of the so-called algebraic and geometric methods, especially under the presence of strong noise, or when the measurement points approach a degenerate configuration.

1

Introduction

A key problem in motion analysis is the recovery of motion between two successive frames (the first of which will be referred to as the domain, and the second as the range – these names are meant to indicate that the sought transformation operates on points in the first image, and transforms them into the second image), given a set of point correspondences. Two facets of this problem are studied in this paper: 1. If the measured points in the domain and range frames are denoted {Pi }m i=1 and {Qi }m i=1 , find a transformation which “maps Pi close to Qi ”; there are two common methods for defining this notion of “closeness”. The first, which is usually referred to as the algebraic method, seeks a transformation T , restricted to be of a certain class (Euclidean, linear, affine, projective etc.) m which minimizes T (Pi ) − Qi 2 . The geometric method searches for a set i=1

of points {Pî }m i=1 and a transformation T , such that m

Pî − Pi 2 + T (Pî ) − Qi 2

i=1 T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 72–85, 2003. c Springer-Verlag Berlin Heidelberg 2003

All Points Considered: A Maximum Likelihood Method for Motion Recovery

73

ˆ i } such is minimal. One may think of a domain-range combination {Pî , Q ˆ ˆ that T (Pi ) = Qi as “legal”; the goal is then to find a “legal” combination which is closest to {Pi , Qi }. 2. The FOE (focus of expansion) problem: given are two images of a set of n 3D points, I1 and I2 , taken from a translating calibrated camera. Let P1 and P2 be the projections of the points in I1 and I2 . Every pair of corresponding points is collinear with an epipole F . Assuming that the camera is moving forward, the points in the second image will be further away from the epipole than their corresponding points in the first image. The points that have been measured in the images are not collinear with the epipole due to noise, and therefore the lines connecting all pairs of corresponding points will not intersect at a single point. For this problem one can also define the algebraic and geometric distances. The algebraic approach to determine the epipole F is to use a linear least squares algorithm which finds the point closest to all the lines passing through the pairs of points. The analogue of the geometric method is as follows. Given a candidate epipole F , for each pair of points p1i ∈ P1 and p2i ∈ P2 , compute the line through F that is closest to the two points. Now, measure the distance of the points from this line, and add the square of each of these two distances to the error function. As opposed to the algebraic approach for which there is a closed-form solution, the geometric requires non-linear optimization techniques. For a comprehensive survey of these methods as well as other motion recovery problems, see [4, 7, 12, 3, 1, 9]. In this paper a new method is introduced, which is compared to these two methods. We first study the linear transformation model, and then extend it to FOE recovery. The method is based on computing the maximum likelihood (ML) estimate for the unknown parameters [5, 6]. The major difference compared with the algebraic and geometric methods is that the ML approach seeks not only a single “good” combination of domain-range points, but looks for a combination which has a “wide support” in the vicinity of the noisy measurement points. In all experiments, it performed considerably better than both other methods, at the price of optimizing more complicated cost functions. Our approach resembles the one in [11], in which the compatibility between a model and a data point is obtained by integrating over the entire model. The differences between our method and the Bayesian approaches in [10, 2] are in that a) we integrate out the “real” domain points, and b) we do not integrate over all possible models, but find the mode of the distribution of the model. Since the full probability distribution is computed, we could also integrate over all models, but this is beyond the scope of this paper.

2

Linear Transformation

In this section the ML estimate to the parameters of a linear transformation is derived, and its performance compared to that of the algebraic and geometric

74

Daniel Keren et al.

methods. We chose to start with the 1D case as it is technically simpler, yet it captures the basic idea of the ML method, and also lends itself to a simple, intuitive explanation. 2.1

The 1D Case

Let us first analyze the ML estimate in a very simple case: a linear transformation from R to itself. As will be seen, even this case (transformation with one parameter) involves non-trivial computations and admits no explicit solution. We seek to estimate the parameter a of a mapping from R to itself, g(t) = at, given two noisy measurement points: Y (range) and X (domain). If additional pairs are given, the probability is the product over all pairs, assuming independence. The algebraic and geometric estimates for the single pair case are both Y . equal to X The ML estimate proceeds to compute the density function of every a conditioned on the measurements. Denote this density f (·) (hereafter f () will denote probability density). Denote by x, y the “true” values of the domain and range points; hence, if the noise model is n, then X = x + n, Y = y + n, and g(x) = y. Hereafter we will assume a Gaussian noise with variance σ 2 , so 2 )2 1 1 f (X|x) = √2πσ exp( −(x−X) ) and f (Y |y) = √2πσ exp( −(y−Y ). 2σ2 2σ2 Since x is not known, it has to be marginalized, or “integrated out”. The first step is to write down the expression for the joint probability density of a and x: ∂(ax, x) f (ax, x/Y, X) = |x| f (Y, X/ax, x)f (ax, x) ∝ f (a, x/Y, X) = ∂(a, x) f (Y, X) (x − X)2 + (ax − Y )2 |x|f (Y, X/ax, x) ∝ |x| exp(− ) (1) 2σ 2 We used the fact that f (Y, X) is constant given the measurement, and assumed a uniform prior on ax, x. While the latter is subject to criticism, it is not pertinent to the theme of this paper; given a better prior on ax, x, it can easily be incorporated into the ML computation. The choice of the “correct” prior lies outside the scope of this paper. Next, the expression in Eq. 1 must be integrated over x, to obtain the residual density for a. A closed-form expression can be obtained for this integral, but it is rather cumbersome and we therefore omit it. We have not been able to obtain a closed-form expression for the ML value of a, and it was found by numerical optimization (which is easy in this case, the search space being one-dimensional). Let us demonstrate how the ML estimate differs from the algebraic and geometric estimates, by studying some special cases. Fig. 1 (left) depicts a plot of f (a) as a function of a, when X = Y = σ = 1. It is readily seen that the ML estimate is rather different from 1, which is the algebraic and geometric estimate; the optimal ML value for a is about 0.467. For low levels of noise (as demonstrated in Fig. 1, right) the ML estimate converges to 1.


75

Fig. 1. Probability of the 1D transformation given by g(x) = ax, when the domain and range measurements are 1. Left: the noise is Gaussian with unit standard deviation; right: the noise is Gaussian with standard deviation 0.1. In both graphs the horizontal axis stands for a and the vertical axis for f (a) 2.2

Interpretation of the Results: Cloud-to-Cloud Match

The ML estimate for a defines a line which does not touch the measurement point (Y, X) – as opposed to the algebraic estimate, which goes through the point. This result, which may appear counter-intuitive, can be explained as follows. The ML estimate seeks to find a slope a which has the largest support, that is, such that there is a large “volume” of pairs (ax, x) that are close to the measurement point (Y, X). In other words, it does not seek an optimal pointsto-points match (like the algebraic and geometric estimates do), but an optimal cloud-to-cloud match: one cloud is the neighborhood of (Y, X), the other is the set {(ax, x)/x = −∞...∞}. That is, the ML estimate a seeks not only that aX be close to Y , but that for a large volume of points x which are close to X, ax will also be close to Y . An explanation is presented in Fig. 2. Note that correspondence still has to be asusmed: the “clouds” here do not refer to the aggregate of points, but to “probability clouds” which surround each range and domain point. We note that least square analysis is inappropriate for the “cloud-to- cloud” matching problem, as demonstrated in [11]. 2.3

Stability

One of the characteristics of the suggested ML estimate is its stability. Consider, for example, the case σ = 1. Since the geometric and algebraic estimates for a are Y , they are very unstable: since X and Y in this case are equal both equal to X to x + N (0, 1), y + N (0, 1) (N (0, 1) = normal distribution with zero mean and 1 (0,1) unit variance), the distribution of a is equal to that of the quotient y+N x+N2 (0,1) . The expectation of this estimate is equal to ∞ ∞ −∞ −∞

y + N1 N 2 + N22 )dN1 dN2 exp(− 1 x + N2 2

76

Daniel Keren et al.

Fig. 2. Left: two lines, corresponding to the geometric choice of slope, a = 1 (dotted line), and the ML choice a = 0.467 (continuous line). Both are for the same data as in Fig. 1 (left), X = Y = σ = 1. On the right, the plots for 2 )2 |x| exp(− (x−X) +(ax−Y ) for the two cases. While the solution for a = 1 passes 2 through (X, Y ) – and therefore the corresponding plot on the right has a higher peak – the total probability for a = 0.467 is larger. Plots are for the range x = −1..4 which is undefined due to the singularity when the denominator is zero (at x = −N2 ). On the other hand, the ML estimate is always well-behaved in the sense that its expectation is bounded. The proof is presented in the Appendix. 2.4

What Does the Result Mean?

As noted, the transformation a recovered by the suggested ML method may appear counter-intuitive, because the recovered transformation does not map the domain data point onto the range data point. The explanation we offer is that the quality of the result should not be measured by the proximity of the transformed measurements of the domain points to the measurements of the range points, neither by the proximity of a configuration of “legal” domain and range points (see the Introduction), to the measurements. We suggest that the quality of the results should be measured by studying the entire probability distribution over all possible sources of the measurement points. To illustrate this for the one-dimensional case, consider again the case X = Y = σ = 1. We know that there exist “real” points x, y such that x + N1 (0, 1) = X, y +N2 (0, 1) = Y , and that the “real” a satisfies ax = y. Thus, the distribution 1 (0,1) for a is given by 1+N 1+N2 (0,1) (note that N (0, 1) is symmetric around the origin). One can, in fact, compute the distribution of this expression, and prove that it is identical to the distribution derived in Section 2.1. The same considerations hold for the problems of 2D motion recovery discussed in the following sections.


2.5

77

The 2D Case

The case of a 2D linear transformation proceeds in a manner very similar to the 1D case. Given noisy measurements of two domain points, P1 and P2 , and of the corresponding range points, Q1 and Q2 , the probability of the transformation T (x, y) = (ax + by, cx + dy) can be computed in a manner similar to the 1D case, albeit more complicated, resulting in: f (a, b, c, d/P1 , P2 , Q1 , Q2 ) = ∞ ∞ ∞ ∞

(2) (x1 y2 −x2 y1 )2

−∞ −∞ −∞ −∞ 2 (xi , yi ) − Pi 2 + T (xi , yi ) − Qi 2

× exp( i=1

2σ 2

)dx1 dy1 dx2 dy2

The explicit form of the integral is too long to include here. However, there is an interesting similarity between this integral and Eq. 1. Note, especially, the weighing factor (x1 y2 − x2 y1 )2 . As in the 1D case, a higher weight is assigned to domain measurement points which are farther from the origin; however, the weight factor also penalizes two domain points which are in similar directions. This is not surprising, since when two points are in the same direction, they are linearly dependent, hence they yield less information on the transformation T . 2.6

Optimization

Given a pair of domain and range measurements, the ML linear transformation is obtained by maximizing Eq. 2. We were not able to find an explicit solution, so the maximum was recovered using a general-purpose optimization method, the Nelder-Mead simplex method with simulated annealing, as presented in [8]. When more measurement points are provided, the expression for the probability factors into a few expressions as Eq. 2, one for each combination of two domainrange pairs. The time it took to recover the motion between two frames was on the average 0.03 seconds on a Digital workstation, for 100 point pairs. 2.7

Some Results

We tested the suggested method for a few cases, and it consistently performed better than both the algebraic and geometric methods. Results for two cases are presented: a nearly degenerate configuration and a stable one. Nearly Degenerate Configuration In this experiment, four domain and four range points were chosen according to the following simple rule: for k = 1..4, the k-th range point as well as the k-th domain point is equal to (k + n, k + n),

78

Daniel Keren et al.

where n is zero mean Gaussian with 0.01 standard deviation (with different instances of n for every coordinate). We study this case in order to illustrate the stability of the method. For example, in one of the tested cases, the noisy measurements were: T (0.999, 1.009) = (1.023, 1.015), T (2.021, 1.989) = (2.015, 1.985) T (2.983, 3.009) = (3.012, 3.002), T (4.007, 4.006) = (3.996, 4.002) The geometric method results for this type of data were very unstable. Denoting as before T (x, y) = (ax + by, cx + dy), we have that a(k + n) + b(k + n) = k + n for very small n, so a + b ≈ 1, and similarly c + d ≈ 1. In order to display the results, it therefore suffices to show the values of a and c. For ten such tests, the results are shown in the scatter diagram below (Fig. 3). The results are very unstable, although the point configurations are nearly identical. The same phenomena occurs when the algebraic method is applied to these point sets. However, the ML estimate studied here gave very stable results; for all cases, the values of the transformation coefficients were between 0.49 and 0.51, that is, the recovered transformation matrix was very close to 1 1

2 1 2

2 1 2

The explanation for this result is that the transformation tries to map points which are close to (k, k) (for k = 1..4), as close as possible to (k, k) (see previous discussion about the “cloud to cloud” matching in Section 2.2). Of all linear combinations of noisy measurements of (k, k) whose expectation is k, the one with weights ( 12 , 12 ) has the smallest variance, hence it achieves the best concentration around the range point (k, k). In this case and others, the ML estimate has the property of choosing a “simple” and stable solution when the motion recovery becomes ill-posed, and this is achieved without imposing an external simplicity constraint, as in regularization or minimal description length based methods.

Fig. 3. Scatter diagram for the coefficients (a, c) of the linear transformation recovered by the geometric method in the nearly degenerate case


79

Fig. 4. Performance of the three methods for motion recovery. Left: horizontal axis stands for noise standard deviation, vertical for average l2 error in the recovery of the motion parameters. The ML estimate is depicted as a solid line, dashed line stands for the algebraic method, and dotted line for the geometric method. Right: horizontal axis stands for noise standard deviation, vertical for the ratio of experiments in which the respective method performed best

Non-degenerate Configuration In this set of experiments, the accuracy with which the three methods recover the transformation

2 1 3 2 was studied. The domain points formed a stable configuration: {(1, 1), (2, 1), (1, 2), (2, 2)}, and various levels of noise were tested. An algorithm’s error was defined as the l2 difference between the original transformation and the recovered one. We have also charted the percentage of cases in which each method performed best. The tests were run 500 times for each noise variance. Results are presented in Fig. 4. The suggested ML estimate was clearly superior to the algebraic and geometric methods. Its average error was much smaller, and it also performed best in most cases except for the smallest noise variance, in which it was very slightly (1.6%) surpassed by the geometric method (however, in this case its average error was 29% smaller than the geometric method’s error). On a side note, our experiments indicated that, although the geometric method performs better than the algebraic method over a wide range of noise variance, its average error is larger when the noise increases.

3

Focus of Expansion

An important problem in computer vision is the recovery of the focus of expansion (FOE). In the simplest instance of the problem, four points are given in the plane: P1 = (P1x , P1y ), P2 = (P2x , P2y ), Q1 = (Qx1 , Qy1 ), Q2 = (Qx2 , Qy2 ), which are

80

Daniel Keren et al.

Fig. 5. Layout of the FOE problem. The point to be recovered is F , however only {P1 , P2 , Q1 , Q2 }, which are the noised versions of {p1 , p2 , q1 , q2 }, are given. For comparison, the FOE corresponding to {P1 , P2 , Q1 , Q2 } is the point F ‘ the noisy versions of p1 = (px1 , py1 ), p2 = (px2 , py2 ), q1 = (q1x , q1y ), q2 = (q2x , q2y ). The point F = (F x , F y ) is sought that such {F, p1 , p2 } and {F, q1 , q2 } are collinear. It is common to assume that p2 respectively q2 is between F and p1 respectively q1 (Fig. 5). The problem is to find a good estimate for F , given {P1 , P2 , Q1 , Q2 } (if more than four points are given then the ML estimate maximizes the product of the densities for the subsets of size four). In order to extend the ML paradigm to the FOE recovery problem, we integrate over all the possible choices of {p1 , p2 , p1 , p2 }. Specifically, given a candidate F , we parameterize p2 , q2 by F, p1 , q1 : p2 = λ1 p1 + (1 − λ1 )F, q2 = λ2 p2 + (1 − λ2 )F (where 0 ≤ λ1 , λ2 ≤ 1). Then, to compute the density of F given {P1 , P2 , Q1 , Q2 }, we proceed as before, using Bayes’ law and an appropriate change of variables. Alas, the computation is more cumbersome. For simplicity we assume that the noise variance satisfies 2σ 2 = 1, but obviously this does not sacrifice any generality. f (F, p1 , q1 , λ1 , λ2 /P1 , P2 , Q1 , Q2 ) =

y y y y y x y x x x x f (F x , F y , px 1 , p1 , q1 , q1 , λ1 , λ2 /P1 , P1 , P2 , P2 , Q1 , Q1 , Q2 , Q2 ) = y x y y y x x x y x x y f (p1 , p , q1 , q , λ1 p1 + (1 − λ1 )F , λ1 p + (1 − λ1 )F , λ2 q1 + (1 − λ2 )F , λ2 q + (1 − λ1 )F / 1 1 1 1 y y y y x x x x P1 , P , P2 , P , Q1 , Q , Q2 , Q ) · 1 2 1 2 y y y y ∂(px , p , qx , q , λ1 px + (1 − λ1 )F x , λ1 p + (1 − λ1 )F y , λ2 qx + (1 − λ2 )F x , λ2 p + (1 − λ1 )F y ) 1 1 1 1 1 1 1 1 y y x x y x ∂(F , F , p , p , q , q , λ1 , λ2 ) 1 1 1 1

y y y x y x y y x (1 − λ1 )(1 − λ2 ) p F x − F x q − p q1 + px 1 q1 − p1 F + F q1 1 1 1 y 2 y y 2 y x x 2 x x 2 exp(−(p1 − P1 ) − (p − P ) − (q1 − Q1 ) − (q − Q ) − 1 1 1 1 y y 2 x x 2 (λ1 p1 + (1 − λ1 )f x − P2 ) − (λ1 p + (1 − λ1 )f y − P ) − 1 2 y y 2 x x 2 (λ2 q1 + (1 − λ2 )f x − Q2 ) − (λ2 q + (1 − λ2 )f y − Q ) ) 1 2

·

=

(3)


81

Next, the “nuisance parameters” px1 , py1 , q1x , q1y , λ1 , λ2 have to be integrated out. The resulting six-dimensional integral is difficult to explicitly compute because of the absolute value expression (Eq. 3). We have therefore applied numerical integration, adopted from the qgauss method [8]. 3.1

Results

As before, the results may appear counter-intuitive. For example, consider the case P1 = (10, 20), P2 = (7.5, 15), Q1 = (24, 12), Q2 = (20, 10), with noise satisfying 2σ 2 = 1. The ML result for the FOE is (3.24, 4.03), quite different from (0, 0) which is the result of the geometric and algebraic methods. Again, the intuitive explanation is that the entire probability space for p1 , p2 , q1 , q2 is sampled by adding noise to P1 , P2 , Q1 , Q2 , and each time the FOE is computed by intersecting the lines between p1 , p2 and q1 , q2 . This results in a cloud of points whose largest density is around the ML estimate. To make this empirical explanation

Fig. 6. Empirical demonstration of the ML FOE estimate. The measured points are P1 = (10, 20), P2 = (7.5, 15), Q1 = (24, 12), Q2 = (20, 10). The FOE corresponding to the measurements is (0, 0) (represented by the intersection of the two dark lines). The ML estimate, on the other hand, is the FOE with the highest probability density; this density can be sampled by randomly adding noise to {P1 , P2 , Q1 , Q2 }, thus obtaining {p1 , p2 , q1 , q2 }, and computing the FOE for {p1 , p2 , q1 , q2 }. A scatter diagram for 4,000 such FOE’s is shown, together with the measurement points and a sample {p1 , p2 , q1 , q2 }, with its corresponding FOE (represented by the intersection of the blue dashed lines). Only points in the square [−15, 15] × [−15, 15] are shown

82

Daniel Keren et al.

of the results clear, a sample of this point cloud is presented (Figs. 6,7). It can be observed that it clusters not around the origin, but around the ML estimate.

4

Conclusion and Future Research

We have presented a Bayes based ML estimate for motion recovery. The basic idea is to consider not only the noisy measurement points, but to integrate over all possible combinations of “real” domain and range points. The method yields results which are correct in the sense of obtaining a motion estimate which has the highest support among all these combinations. Due to this global property, the method is very stable even when the point configurations are nearly degenerate, or when the noise is large relative to the size of the point set. The price is the computational cost of evaluating the probability of the candidate motion parameters, which involves integrating over all possible domain configurations. Future work will address the following questions: is a ML estimate good enough, or should the “fully Bayesian” estimate be computed by averaging the

Fig. 7. Close-up of scatter diagram of Fig. 6: denser (darker) areas correspond to higher values of the FOE probability density function. It can be observed that the highest density is not around the origin. The ML estimate is (3.24, 4.03). Note that the density is nearly flat in a large area, indicating that it is not possible to produce a single highly reliable estimate for the FOE


83

motion models weighed by their probabilities? How reliable are the ML estimates (equivalently, is the probability distribution uni-modal and strongly peaked)? Also, the method will hopefully be extended to cover more difficult problems such as the recovery of the fundamental matrix. In addition, applications of the theoretical results to real problems will be studied.

References [1] A. Adam, E. Rivlin, and I. Shimshoni. Computing the sensory uncertainty field of a vision-based localization sensor. IEEE Trans. on Robotics and Automation, 17(3):258–267, June 2001. 73 [2] D. A. Forsyth, S. Ioffe, and J. Haddon. Bayesian structure from motion. In ICCV99, pages 660–665, 1999. 73 [3] R. Hartley and A. Zisserman. Multiple Views Geoemtry in Computer Vision. Cambridge University Press, 2000. 73 [4] R. I. Hartley. In defense of the eight-point algorithm. IEEE Trans. Patt. Anal. Mach. Intell., 19(6):580–593, June 1997. 73 [5] K. Kanatani. Geometric computation for machine vision. In Oxford University Press, 1993. 73 [6] K. Kanatani. Statistical-analysis of geometric computation. CVGIP, 59(3):286– 306, May 1994. 73 [7] H. C. Longuet-Higgins. A computer algorithm for reconstructing a scene from two projections. Nature, 293:133–135, 1981. 73 [8] W. Press, B. Flannery, S. Teukolsky, and W. Vetterling. Numerical Recipes in C. Cambridge University Press, 1988. 77, 81 [9] G. Speyer and M. Werman. Parameter estimates for a pencil of lines: Bounds and estimators. In ECCV, 2002. 73 [10] P. H. S. Torr and A. Zisserman. Concerning Bayesian motion segmentation, model averaging, matching and the trifocal tensor. In H. Burkharddt and B. Neumann, editors, ECCV98 Vol 1, pages 511–528. Springer, 1998. 73 [11] M. Werman and D. Keren. A Bayesian method for fitting parametric and nonparametric models to noisy data. IEEE Trans. Patt. Anal. Mach. Intell., 23(5):528– 534, May 2001. 73, 75 [12] Z. Zhang, R. Deriche, O. Faugeras, and Q. Luong. A robust technique for matching two uncalibrated images through the recovery of unknown epipolar geometry. Artificial Inteligence, 78(1-2):88–119, 1995. 73

Appendix In this appendix we will prove that the expectation of the maximum likelihood estimator for the recovery of the one-dimensional scale parameter, discussed in Section 2.1, is well-defined and finite. Although this is a simple case, it serves to demonstrate the stability and convergence properties of the suggested method. Recall (Eq. 1) that if the mapping is given by T (x) = ax, and the domain respectively range measurements are X and Y , then the probability density of a is given by ∞ |x| exp(−[(x − X)2 + (ax − Y )2 ])dx (4) f (a) = −∞

84

Daniel Keren et al.

(where we have assumed for simplicity 2σ 2 = 1). The proof will proceed by demonstrating that the integral of Eq. 4 is smaller than f (0), for a large enough a. Hence, the maximum likelihood estimate (which is obtained at the global maximum of f (a)) will be bounded, and then its expectation can be estimated by integrating over all possible Y, X. The integral of Eq. 4 equals ∞ exp(−X − Y ) 2

2

|x| exp(−[(a2 + 1)x2 − 2(X + aY )x])dx

−∞

if A > 0, ∞ |x| exp(−[Ax − Bx])dx = 2

2A3/2 +

√ 2 B √ πAB exp( B 4A )erf ( 2 A ) 2A5/2

−∞

then, noting that |erf (x)| < 1, and substituting A = a2 + 1, B = 2(X + aY ), f (a) can be bounded by X + aY (X + aY )2 1 exp(−X 2 − Y 2 ) 2 + exp( ) (5) a a3 a2 here we gave assumed that a, Y, X ≥ 0, but this only means that in the remaining part of the proof we should replace these three variables by their absolute values; to make notations simpler, we will omit the absolute values. Note that ∞ |x| exp(−[(x − X)2 + Y 2 ])dx

f (0) = −∞

= exp(−X 2 − Y 2 )[1 + X exp(X 2 )erf (X)] ≥ exp(−X 2 − Y 2 ). So, in order to bound f (a) by f (0), we can cancel out exp(−X 2 − Y 2 ). After discarding a12 in Eq. 5, as it is asymptotically negligible compared to the other expressions, it is then sufficient to bound

X+aY a3

2

) exp( (X+aY ) by 1. a2

Lemma 1. There exists a polynomially bounded expression p(Y, X) such that if a > p(Y, X) then (X + aY )2 1 X + aY exp( ) < 3/2 exp(Y 2 + 1) 3 2 a a a Proof: It is straightforward to verify that this inequality holds for a polynomial larger than max{2, 2X, 2Y, 2X 2, 4XY, 2Y 2 }. Denote p(Y, X) = max{2, 2X, 2Y, 2X 2, 4XY, 2Y 2 }. Lemma 2. If a > max{exp( 45 (Y 2 + 1)), p(Y, X)}, then f (a) < f (0).


85

Proof:

X + aY (X + aY )2 f (0) f (a) < f (0) exp( ) < 3/2 exp(Y 2 + 1) < a3 a2 a f (0) f (0) exp(Y 2 + 1) = < f (0). exp( 65 (Y 2 + 1)) exp( 15 (Y 2 + 1) Since the ML estimate is obtained at the global maximum of f (a), it is bounded by max{exp( 45 (Y 2 + 1)), p(Y, X)}, and its expectation can be bounded by integrating this bound over all Y, X pairs: Theorem 1. The expectation of the maximum likelihood estimate for a is finite. Proof: Suppose that the “real” domain and range points are x, y respectively. Then, the expectation of a is bounded above by ∞ ∞ −∞ −∞

4 exp(−(X − x)2 − (Y − y)2 ) max{p(Y, X), exp( (Y 2 + 1))}dXdY 5

and this integral is convergent, because the left exponent is dominated by exp(−X 2 − Y 2 ) as Y, X → ∞. Hence asymptotically the integrand is bounded by max{exp(−X 2 − Y 2 )p(Y, X), exp(−X 2 − 15 Y 2 )}, and the integral of this expression clearly converges.

Distance Map Based Enhancement for Interpolated Images PeiFeng Zeng and Tomio Hirata Nagoya University, Nagoya 464-8603, Japan [email protected] [email protected]

Abstract. Distance maps have many applications in computer vision, pattern recognition, morphology and robotics. In this paper, an approach of Distance Map based Image Enhancement (DMIE) is proposed for improving the quality of interpolated images. In DMIE, edge detection is performed after images are interpolated by conventional interpolation schemes. A unified linear-time algorithm for the distance transform is applied to deal with the calculation of Euclidean distance from pixels to edges in the image. The intensities of pixels that are located around edges are adjusted according to the distance to the edges. DMIE produces a visually pleasing sharpening of edges in interpolated images.

1

Introduction

Distance maps have many applications in computer vision, pattern recognition, morphology and robotics. In this paper, we apply it to visual improvement for interpolated images. In conventional interpolations, such as bilinear, bicubic, and spline interpolations, the intensities of pixels to be interpolated are calculated based on local information of images [1]. The calculation is independent of the location in the image so they are called as space-invariant schemes. In these conventional schemes, bilinear interpolation is the simplest one but the result is poor especially at edges. In contrast, spline interpolation can produce good results but the plot is rather complicated [2, 3, 4]. Bicubic interpolation is widely used because its result is better than bilinear and can meet practical requirements in most cases. Furthermore, the calculation is not complicated. When images are expanded by linear interpolations, edges in images appear to be blurry. Many researches have been dedicated to attacking such shortcomings. Space-variant calculations are applied in these schemes to realize the image enhancement. A linear space-variant approach is proposed in [5] to enhance the visual quality of images. Edges are detected from small blocks of images and classified into five different types. Appropriate interpolation methods are decided for different types of edges, a high-pass filter is then used to improve the sharpness. In [6], edges of input images are located as the best-fit continuous space step edges. A sampling lattice of higher resolution is then superimposed, using the bilevel T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 86–100, 2003. c Springer-Verlag Berlin Heidelberg 2003

Distance Map Based Enhancement for Interpolated Images

87

result of each fit as a template. A threshold is used to the output of edge operator to judge the appropriateness of the technique. In the hybrid approach proposed in [7, 8], edge enhancement is realized by adapting the interpolation based on the local covariance coefficients near edges. A Bayesian estimation technique is discussed in [9]. Maximum a posteriori estimation and a suitable random field model are used. Two gradient algorithms are proposed for interpolating both noise-free images and images corrupted by white noise. A warped distance based image enhancement proposal (WaDi) is firstly proposed in [10] by Ramponi. As we know that in classical interpolations, the intensity of newly interpolated pixel can be calculated by interpolation equation when the distance to the neighboring pixel in the low-resolution image is determined. In [10], local homogeneities of opposite sides of the newly interpolated pixels are examined. Then, the distance used in the interpolation equation is modified according to the local homogeneities. Ramponi called the modified distance a warped distance. The intensity of the newly inserted pixel is calculated using the warped distance. In [11], Zeng and Hirata proposed a nonlinear space-variant approach, called Distance Map Based Image Enhancement with Intensity Adjustment (DMIE-IA), to enhance images. In DMIE-IA, images are firstly interpolated and processed with edge detection [12]. Then, Euclidean distance values of all pixels to nearest edges in the image are calculated [13]. Such an area called a Transient Area (TA), that is located around edges determined. Intensities of pixels in the TA are adjusted so as to increase the contrast. Over-enhancement is avoided by determining the enhance range. Images can be enhanced repeatedly and edges in the images are sharpened more and more. In this paper, we introduce the concept of warped distance from WaDi into DMIE-IA to improve the computation time and propose a scheme of DMIE with warp translation, called DMIE for short from now on. Instead of intensity adjustment, in DMIE, pixels are moved towards their nearest edges by the warp calculation based upon the Euclidean distance. Thus the width of a blurred edge will be narrowed and the edge be sharpened, too. DMIE can produce the same image enhancement quality as DMIE-IA and execute the image enhancement in a speed that is 10% faster. Compared with WaDi in [10] that executes the warp calculation with the neighboring pixels, in DMIE, images are enhanced with the information of Euclidean distances [13, 14, 15] for pixels to nearest edges in the images. Thus it can produce more pleasing visual results. Indeed, the Euclidean distance transform did not find many applications when the algorithm with time complexity O(N 3 ) was known for N × N images. The time complexity was improved in [14] to O(N 2 log N ). It was further reduced to O(N 2 ) [15]. Hirata proposed a simple O(N 2 ) time algorithm for computing Euclidean distance maps [13]. Instead of raster scanning, in [13], the calculation is realized in two steps, called the column scan and the row scan. In the column scan, the distances from all pixels to the nearest edge pixels in the same column are calculated. The calculation time of the column scan is O(N 2 ).

88

PeiFeng Zeng and Tomio Hirata

While in the row scan, instead of calculating the distances for each pixel to the nearest edge pixels in all columns, we consider the function indicating the distances between the nearest edge pixel in a column and pixels in the row. The lower envelope of all these functions gives the Euclidean distances for pixels in the row. The calculation time of row scan is O(N 2 ) for all rows and also the overall time of the algorithm of [13]. The above efforts make practical applications of the Euclidean distance transform possible. When the shortest distance for a pixel is calculated, the nearest edge pixel can be located also and this information is used in this paper to speedup the processing. Both in DMIE-IA and DMIE, the image enhancement is performed with the information of Euclidean distance to nearest edge. To make this paper selfcontained, we will give some descriptions that appeared in [11] and organize the paper as follows. Section 2 describes the proposal of DMIE in 1D and extends it to image processing. In Section 3, experimental results for the proposal are provided and a comparison between our proposal and a contemporary image enhancement is also discussed. In Section 4, the performance of DMIE is analyzed in comparison with WaDi. Several conclusions are produced in Section 5.

2

Warped Distance Image Enhancement

When images are expanded by conventional interpolations, edges in the images are blurred and widened. Intensities of the images change gradually in the TA, the area that surrounds the edges. See Fig. 1. So the natural approach to improve the visual quality of interpolated images is either to increase the contrast of the TA or to narrow it. The proposal in this paper is the latter one. To simplify the description, DMIE is discussed in 1D first. 2.1

DMIE for One Dimensional Case

In the 1D situation, when signals are interpolated with conventional linear schemes, edges in the signals are smoothed. By observation, we found that the width of such widened edges are proportional to the interpolation factor m. So in this paper, we set the width of the TA be 2m, twice the interpolation factor. In Fig. 1, the TA is defined as the interval (−m, m) and the edge point is located at the position 0. Generally, there are three models that are used to describe edges. They are step, ramp, and sigmoidal model. The edges in interpolated signals appear to be smooth, so the sigmoidal is adopted for modeling edges in this paper. The sigmoidal function is expressed as f (x) =

1 . 1 + e−Cx

(1)

In (1), the parameter C controls the tangent of f (x) passing through the point 0, the edge point defined in this paper.


1 0.9

−m

0 TA

A

89

m B IB

0.8 0.7 0.6 0.5 0.4

C increases

0.3 0.2 0.1

IA

0

Fig. 1. The transient area (TA). The width of the TA is set to be 2m, where m is the interpolation factor for either dimension of image In the warp calculation, all points in the TA are moved to a new position towards the edges that are nearest to them. We denote a point P in a TA with the distance d to the edge after warp calculation. The point is moved from the position P0 that has distance d0 . The TA will be narrowed when d < d0 . In this paper, we define the warp equation, i.e., the relation between d and d0 as d0 = d + kd(m − d)/m.

(2)

In (2), the parameter m equals the maximum distance of points in the TA. Parameter k ≥ 0 is used to control the warp calculation. When k = 0, there is no warp, thus d0 = d and no points should be moved. When k increases, the warp will function and points in the TA will be moved towards edges that are located in the midpoints of the TA. The larger the value of k, the larger the enhance rate. The relation between d and d0 that is controlled by the value of k is shown in Fig. 2. The value of d0 exceeds the range of the TA in the case that k > 1. That will cause artifacts in the result of DMIE. Thus, The non-distortion DMIE is limited to k ≤ 1. Fig. 3 shows the result of applying the DMIE enhancement with different values of k. The tangent curves increases and edges get more and more sharpened when the value of k increases. In Fig. 4, DMIE enhancement with k = 1 is applied to edges with various widths. These edges are simulated by sigmoidal functions. The parameter C in (1) increases from 0.001 to 1 from the top to the bottom in Fig. 4(a). In the results of DMIE in Fig. 4(b), The TA is narrowed visually and the intensity changes smoothly. 2.2

DMIE for Interpolated Images

Since the images belong to 2D, L2 distance calculation is needed for the application of DMIE in image processing. In this paper, an algorithm in [13] is used

90


5

Displacement after warping

4 3

k=0.2 k=0.6 k=1.0 k=1.5

2 1 0 −1 −2 −3 −4 −5 −6

−4

−2

0

2

4

6

Displacement before warping

Fig. 2. Pixels in the TA are moved from P0 to new locations P by warp translation (m = 5). x-axis indicates the distance d0 and y-axis the d

to implement the calculation of Euclidean distance for images. The TA is the region that surrounds the edge. In the TA, all pixels have the distance values equal to or less than the interpolation factor m. By linking all pixels in the TA to their nearest edges, we obtain the values and the directions of the distance. For every TA pixel, set the current position, i.e., the distance to edge as d and call the pixel as P , calculate the distance d0 with (2) to find the position of pixel P0 from which the intensity is copied to pixel P . We know that the line that links the pixel P0 and the edge will hit the pixel P . But in general, P0 does not necessary to lie on the grid in the image matrix. To solve the problem, we

1

0.8

Interpolated edge k=0.5 k=1.0

0.6

0.4

0.2

0 −5

0

5

Fig. 3. Edges are sharpened after warp calculation. (m = 5)


(a)

91

(b)

Fig. 4. (a) Edges modeled by sigmoidal function with different value of C. Corresponding to edges from the top to the bottom, the value of C changes from 0.01 to 1. (b) Edges are enhanced by DMIE in (2)

use the bilinear scheme to calculate the intensity of P0 based upon four pixels that are around P0 . When edges are located too near to each other, their corresponding TAs will overlap one another. In this case, each TA is narrowed. When the warp calculation of (2) is applied to pixels in such area, P0 will have the distance to edges that is larger then the width of the TA there. The artifacts will occur in the result of DMIE. Fig. 5(a) shows such an example, there are some white pixels in the black strip in the figure. To overcome such a problem, a translate direction check is applied as follows. We denote (x0 , y0 ), (ex0 , ey0 ), (x, y), and (ex , ey ) the coordinate of P0 , the nearest edge pixel of P0 , P , and the nearest edge pixel of P , respectively. In the condition that (x0 − ex0 ) · (x − ex ) ≥ 0 , (y0 − ey0 ) · (y − ey ) ≥ 0 the intensity of P0 will be copied to pixel P . From the result of the processing in Fig. 5(b), these white dots in black strips have disappeared.

3 3.1

Experiments and Comparison with WaDi Approach Experimental Results

DMIE is applied to the image of Lena for performance testing. The bicubic interpolation is applied to the image with the expansion factor of five in both dimensions. Fig. 6(b) shows the interpolation result of Fig. 6(a), which is a part of Lena. Fig. 6(c), (d), and (e) illustrate the results of applying the DMIE with k = 0.5, k = 0.7, and k = 0.9, respectively.

92


(a)

(b)

Fig. 5. (a) Artifacts occur when the edge are located too near to each other. White dots in the black strip are the results of translation of pixels located outside the TA. (b) The artifacts are eliminated after the translate direction check is applied From the experimental results, we find that the image becomes increasingly more enhanced and edges in the image become clearer by increasing the parameter k in DMIE. The larger the factor in the interpolation is, the larger the width of edges in the interpolated images will be. In such cases, a large value of k is needed to enhance edges. When the images are interpolated with the factor of five in either directions, we find that there will be little difference in the result of DMIE when k > 0.7. In Fig. 6(d) and (e), the two results appear to be similar. To measure the processing speed of DMIE, we apply the DMIE to images with various sizes and set the interpolation factor to five. The test is performed using a personal computer (CPU: Pentium III 833MHz, RAM: 400MB). The results are shown in Fig. 7. From Fig. 7, we observe that the computation complexity of DMIE is nearly O(N 2 ) for an N × N input image. For a 256 × 256 input image, DMIE can produce a 1280 × 1280 output in 2.4 seconds. 3.2

Comparison with WaDi Approach

In [10], Ramponi proposed an image enhancement approach called as WaDi, warped distance image interpolation. WaDi can be applied to various conventional image interpolations to improve image’s visual quality, too. In WaDi, the distance between two consecutive pixels in an image to be interpolated is defined as one. When a pixel is to be inserted in the place between pixels xk and xk+1 and has a distance s to xk , a warped distance s instead of the distance s is used in the conventional linear interpolation to calculate the intensity of the newly interpolated pixel. The s is defined as s = s − kAs(s − 1),

(3)

where, the parameter A represents the local asymmetry of the new pixel. k is used to adjust the enhancement degree of WaDi.


93

Ramponi has made a comparison of WaDi with two image enhancement algorithms in [5] and [16]. From the experimental results he reached the conclusion that WaDi is better than either of the two other approaches. The shortcomings of the proposal in [5] is noise exaggeration and poor continuity of edges in the result. The proposal in [16] also produces noisy output due to its intrinsic high-pass filtering. In [17], the WaDi has been proven to be better than the two schemes. The DMIE proposed in this paper does the image enhancement based upon the detected edge and the warp calculation, so the continuity of edge is good in the result. By limiting the image enhancement in TA areas, noise is not exaggerated in our results. So we will make a comparison with WaDi scheme in this paper.

(a)

(b)

(c)

(d)

(e)

Fig. 6. (a) A part of Lena. (b) The image is processed by bicubic interpolation. It is expanded by a factor of five in both dimensions. (c) DMIE with k = 0.5 is applied. (d) DMIE with k = 0.7 is applied. (e) DMIE with k = 0.9 is applied

94


7

6

Bicubic interpolation Edge detection Distance transform Image enhancement Total time

(Sec.)

5

4

3

2

1

0 50

100

150

200

250

300

350

400

450

(N)

Fig. 7. Processing speed of DMIE for an N ×N image. The images are expanded by a factor of five in both dimensions by bicubic interpolation

Since bicubic can provide interpolation provides better results than bilinear interpolation, both DMIE and WaDi are applied to an image that is interpolated by bicubic interpolation in this paper. The Barbara is interpolated by a factor of five in both dimensions. A portion of the result is shown in Fig. 8(a). Fig. 8(b) is the result of WaDi with k = 10. Fig. 8(c) is the result of DMIE with k = 1. From the results, we find that the edges of the scarf of the woman in (b) are strongly enhanced while little enhancement is applied to the edges of the eye. This is caused by the changing of the value of A in (3). The parameter A increases in the edges of scarf because of the strong contrast. Thus intense warp is applied for edges with larger contrast. On the other hand, in DMIE, the same processing is applied for all pixels nearby edges. So all edges can receive same enhancement during the processing.

(a)

(b)

(c)

Fig. 8. (a) Barbara is interpolated by bicubic interpolation with a factor of five in either dimensions. (b) Result of WaDi with k = 10. (c) Result of DMIE with k=1


(a)

(b)

95

(c)

Fig. 9. (a) Result of WaDi with k = 1. (b) Result of WaDi with k = 3. (c) Result of of DMIE with k = 1 More over, edges in Fig. 8(b) appear to be blocking. That is caused by over enhancement. Ramponi [10] set the warped distance s to one in case that its calculated value exceeds one. In this case, the intensity of the newly interpolated pixel will be set to the value of xk+1 in (3). When there exists strong contrast in the image, the parameter A will increase greatly. In case the value k is also increased, the above case will occur and blocking appears in the edge region of the result. But in DMIE, the enhancement is processed based upon the distance value along the edges of interpolated images. It is a relatively global information so edges are smooth when they are enhanced. No blocking will occur in the result of DMIE. The WaDi enhancement is adjusted by the parameter k. By reducing the value k, the artifacts in edges will be alleviated but the enhancement is also reduced. Fig. 9(a) and (b) are results of WaDi with k = 1 and k = 3, respectively. Blocking artifacts in edges are alleviated but the enhancement is also weakened. For comparison, the result of enhancement of DMIE is shown in (c) with the enhancement rate k = 1. It can produce smooth edges with satisfactory enhancement result. Furthermore, we observe that the worst case of blocking in WaDi occurs in the case that edges are along the angle near 45◦ or 135◦ . See the edges in Fig. 9(b). While there will be no blocking for horizontal and vertical edges. WaDi is unable to provide a satisfying enhancement result, but it has a nice feature of conceptual simplicity. Therefore, its calculation time is shorter compared to DMIE. From Fig. 10, we understand that the average calculation time of DMIE equals 3.1 times that of WaDi. By comparing DMIE with the speed of DMIE-IA, we realize that the proposal of DMIE is improved by 10% with no deterioration in the results.

4

The Performance of DMIE

With DMIE, blurred edges are narrowed and images are sharpened. In order to analyze the performance of DMIE, we apply the down-sampling process to

96


8 7

DMIE−IA DMIE WaDi

6

(Sec.)

5 4 3 2 1 0 50

100

150

200

250

300

350

400

450

(N)

Fig. 10. Computation comparison among DMIE, WaDi, and DMIE-IA for an N × N image. The images are expanded by a factor of five in both dimensions by bicubic interpolation

a known sample and get an image with the size reduced by a factor of five in both dimensions. Then the reduced image is expanded and restored to the original size by bicubic interpolation. The performance of DMIE is studied by applying DMIE to the blurry interpolation result, measuring the differences of DMIE results with the original image. The results of our proposal are compared with that of WaDi. In order to measure the difference of two images, we define the distance between two N × N images F = {fi,j } and G = {gi,j } as1 ∆=

1 N2

2

(fi,j − gi,j ) .

(4)

0≤i,j≤N −1

When the ∆ in (4) equals zero, the two images F and G are identical. The sample binary image for DMIE analyzing is shown in Fig. 11(a), it is the result of down-sampling to the image in Fig. 11(b) by a factor of five in both dimensions. In the sample, there are edges with different directions. When the sample image is processed by bicubic interpolation, the edges are smoothed as shown in Fig. 12(a). The distance between the smoothed image and the image in Fig. 11(b) is ∆ = 8.2 × 10−3 . We first apply WaDi to the sample for comparison. The results are presented in Fig. 12(b) and (c) with k = 2 and 10, respectively. Fig. 13(a) shows the distance of WaDi with various k. Secondly, DMIE is applied to the sample. By increasing k, the ∆ of the enhancement result to the input image will be reduced. The minimum distance ∆ = 4.2 × 10−3 is provided by applying k = 1 for DMIE. 1

The intensity of pixels in the two images is re-scaled to the range of [0, 1].


(a)

97

(b)

Fig. 11. (a) The down-sampling test image with the demension 50 × 50. (b) The test image before down-sampling with the demension 250 × 250 However, in some cases, a clearer enhancement result is expected. This can be achieved by repeating DMIE to the previous result of DMIE. Fig. 13(b) shows the change of distance ∆ by iterating the DMIE to the sample. By repeating DMIE with k = 1, the distances of the second and third enhancement are ∆ = 1.95 × 10−3 and ∆ = 1.04 × 10−3 , respectively. Fig. 12(d), (e), and (f) show the results of DMIE processing (k = 1) once, twice, and for three times, respectively. From the results, we understand that by increasing the value of k or by repeating the enhancement, the distance of DMIE to the known sample is reduced continuously. In contrast, the WaDi fails to reduce the distance to the known goal when its k is increased. This is because the intrinsic shortcomings of WaDi that incapacitates it to produce smooth diagonal edges or locate horizontal and vertical edges correctly. From Fig. 13(a), we understand that the distance between the result and the original input is reduced for 0 < k < 2. That is because the edges are enhanced more and more when k increases. However, when k > 2, the error caused by erroneous edge location of WaDi becomes distinct. So Fig. 12(c) presents a clearer enhancement result than Fig. 12(b), but with a larger distance value.

5

Conclusions

In this paper, a warp translation is applied to distance map based image enhancement and an approach called DMIE is presented. The quality of results of linear interpolations, such as bilinear, bicubic, and cubic spline, can be further improved by DMIE. In DMIE, interpolatory edge detection [12] is applied to input images, and thus edges are more perceptible and their location accuracy is also improved compared to conventional edge operators such as Mallat [18] and Canny [19] edge operators. In order to make visually pleasant results with edge enhancement, the enhancement is processed based upon the Euclidean distance values to the nearest

98


(a)

(b)

(c)

(d)

(e)

(f)

Fig. 12. (a) The test image is interpolated by bicubic. The interpolation factor is five in either direction. (b) Result of applying WaDi with k = 2. (c) Result of applying WaDi with k = 10. (d) Result of applying DMIE once with k = 1. (e) Result of applying DMIE twice with k = 1. (f) Result of applying DMIE for three times with k = 1 edges in the TA. All pixels in the TA are moved to new locations in the direction towards nearest edges, thus the TA is narrowed to achieve a pleasant enhancement result. To speedup the processing of Euclidean calculation, a unified linear-time algorithm for the distance transform is adopted. In DMIE, after the preparations such as edge detection and TA calculations, the enhancement rate is determined in the last step. The results of different enhancement can be achieved quickly with repeating the last step of DMIE with different enhancement rate. Thus DMIE is good at adjusting the image enhancement to different degrees. By limiting the enhancement in the TA, the flat area in the results of DMIE keep the same smoothness as the ordinary interpolated results. Comparing with WaDi, DMIE can produce clearer edges and no blocking artifacts occur in the results. Moreover, DMIE is able to reduce the distance ∆ between the original image and the enhanced result while WaDi failed. In our


−3

8.4

x 10

9

8.2

8

8

7

−3

Enhance once Enhance twice Enhance for three times

6

(Error)

(Error)

7.8 7.6 7.4

5 4 3

7.2

2

7

1

6.8 0

x 10

99

1

2

3

(k)

(a)

4

5

6

0 0

0.2

0.4

(k)

0.6

0.8

1

(b)

Fig. 13. (a) Errors of applying WaDi with various k. (b) Errors of applying DMIE with various k future work, we will use real images instead of synthesized images in our future work to verify the effectiveness of DMIE to natural pictures. Compared with DMIE-IA, DMIE has a quick processing speed with no deterioration in the results.

Acknowledgment The authors thank anonymous reviewers for their critical comments, which help to improve the presentation of this paper.

References [1] Kenneth R. Castleman. Digital Image Processing. Prentice Hall, Inc., 1996. 86 [2] M. Unser, A. Aldroubi, and M. Eden. Fast B-spline transforms for continuous image representation and interpolation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(3):277–285, March 1991. 86 [3] M. Unser, A. Aldroubi, and M. Eden. B-Spline signal processing: Part I—Theory. IEEE Transactions on Signal Processing, 41(2):821–833, February 1993. 86 [4] M. Unser, A. Aldroubi, and M. Eden. B-Spline signal processing: Part II—Efficient design and applications. IEEE Transactions on Signal Processing, 41(2):834–848, February 1993. 86 [5] K. P. Hong, J. K. Paik, H. J. Kim, and C. H. Lee. An edge-preserving image interpolation system for a digital camcorder. IEEE Trans. Consumer Electron., 42(3):279–283, August 1996. 86, 93 [6] K. Jensen and D. Anastassiou. Subpixel edge localization and the interpolation of still images. IEEE Trans. IP, 4(3):285–295, March 1995. 86 [7] X. Li and M. T. Orchard. Edge-directed prediction for lossless compression of natural images. IEEE Trans. IP, 10(6):813–817, October 2001. 87

100


[8] X. Li and M. T. Orchard. New edge-directed interpolation. IEEE Trans. IP, 10(10):1521–1527, October 2001. 87 [9] R. R. Schultz and R. L. Stevenson. A bayesian approach to image expansion for improved definition. IEEE Trans. IP, 3(3):233–242, May 1994. 87 [10] G. Ramponi. Warped distance for space-variant linear image interpolation. IEEE Trans. IP, 8(5):629–639, May 1999. 87, 92, 95 [11] P. F. Zeng and T. Hirata. Distance map based image enhancement. (submitted). 87, 88 [12] P. F. Zeng and T. Hirata. Interpolatory edge detection. Machine Graphics and Vision, 10(2):175–184, 2002. 87, 97 [13] T. Hirata. A unified linear-time algorithm for computing distance maps. Information processing letters, 58:129–133, 1996. 87, 88, 89 [14] M. N. Kolountzakis and K. N. Kutulakos. Fast computation of Euclidean distance maps for binary images. Information processing letters, 43:181–184, 1992. 87 [15] L. Chen and H. Y. H. Chuang. A fast algorithm for Euclidean distance maps of a 2-D binary image. Information processing letters, 51:25–29, 1994. 87 [16] S. Thurnhofer and S. K. Mitra. Edge-enhanced image zooming. Opt. Eng., 35:1862–1869, July 1996. 93 [17] G. Ramponi. A cubic unsharp masking technique for contrast enhancement. Signal Processing, 67:211–222, 1998. 93 [18] S. Mallat. A Wavelet Tour of Signal Processing. Academic Press, 1998. 97 [19] J. Canny. A computational approach to edge detection. IEEE Transactions on PAMI, 8(6):679–698, November 1986. 97

Multigrid Convergence and Surface Area Estimation David Coeurjolly1 , Frédéric Flin2 , Olivier Teytaud3 , and Laure Tougne1 1

Laboratoire ERIC 5, avenue Pierre-Mendes-France, F-69676 Bron Cedex {dcoeurjo,ltougne}@eric.univ-lyon2.fr 2 Centre d’Etudes de la Neige 1441, rue de la Piscine, Domaine Universitaire, F-38406, Saint Martin d’Hères Cedex [email protected] 3 Artelys 215, rue Jean-Jacques Rousseau, F-92136 Issy-les-Moulineaux Cedex [email protected]

Abstract. Surface area of discrete objects is an important feature for model-based image analysis. In this article, we present a theoretical framework in order to prove multigrid convergence of surface area estimators based on discrete normal vector field integration. The paper details an algorithm which is optimal in time and multigrid convergent to estimate the surface area and a very efficient algorithm based on a local but adaptive computation.

1

Introduction

In three-dimensional shape analysis, the surface area is one of the important features. Its definition and calculation are well known in classical mathematics. Problems arise when we want to define such a measure on discrete data. Recently, many papers have proposed both theoretical analysis and algorithmic aspects for the surface area estimation of discrete surfaces. When an estimation of an Euclidean measure is proposed on the discrete model, a way to formally evaluate this estimator is to consider its multigrid convergence [17]: we assume a multigrid digitization of a family of Euclidean shapes and we prove that the proposed estimator will converge to the Euclidean measure when the grid resolution increases. An important interest of this property is the soundness of such an estimator in multi-scale object analysis processes. In the literature, two main approaches exist, the first one consists in polyhedral approximation of the discrete volume. In that approach, Klette et al. [16] use a digital plane segmentation process but no proof of multigrid convergence is given. Sloboda et al. [27] introduce the notion of relative convex hull of the discrete objet, this approach is multigrid convergent but no algorithm exists. Another approach consists in local approximations given a neighborhood. More precisely, Mullikin et al. [23] consider a finite set of voxels configurations in a given neighborhood on a discrete surface and they associate a weight to each T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 101–119, 2003. c Springer-Verlag Berlin Heidelberg 2003

102

David Coeurjolly et al.

configuration. Hence, the surface area is given by summing all weight configurations found on the discrete surface. Such a process is not multigrid convergent but allows efficient implementations. This work was also motivated by a practical application in snow sample analysis: the surface area of a sample is an essential parameter to describe snow metamorphisms and offers interesting outlooks for parametrising the fine-scale behavior of snow in avalanche risk forecasting models. First of all, we present the multigrid convergence property of Euclidean measure estimators and then we present a surface area estimation based on discrete normal vector field integration. We first prove that this estimator is multigrid convergent if and only if the normal vector estimator is convergent in direction. Then, we present a local but adaptive algorithm for the surface area estimation based on this schema. This algorithm approach allows us to control the estimation process according to our practical motivations. Finally, we detail an optimal in time and multigrid convergent algorithm to estimate this normal vector field.

2

General Framework for the Surface Area Multigrid Convergence

First of all, we present the theoretical estimator evaluation framework based on the multigrid convergence property. 2.1

Multigrid Digitization and Multigrid Convergence

We consider the Euclidean 3-space E 3 = (R3 , d2 ). We denote by S a Jordan surface in E 3 . We assume an orthogonal grid constant 0 < θ ≤ 1 in the 3-space. In other words, θ is the uniform spacing between grid points parallel to one of the coordinate axis. We denote r = 1/θ the grid resolution. We consider a digitization process denoted by digr . Such a digitization process can be the Grid Intersect Quantization scheme (GIQ for short), the Object Boundary Quantization scheme (OBQ for short) or the Background Boundary Quantization scheme (BBQ for short) (see [14] for a digitization scheme survey). We consider a digitization based surface area estimator EA . A way to evaluate such an estimator is to consider the multigrid convergence property [15]: Definition 1 Let S be a Jordan surface in E 3 , digr be a digitization process, EA be a surface area estimator based on digr (S) and A(S) be the surface area of S. EA is multigrid convergent with a convergence speed v(r) if and only if there exists r0 and r ≥ r0 such that: |EA (digr (S)) − A(S)| ≤ v(r)

and

limr→∞ v(r) = 0

(1)

In the literature, the multigrid convergence of a discrete surface area estimator is an open problem (see the Open Problems in Digital Geometry and

Multigrid Convergence and Surface Area Estimation

103

Topology web page maintained by R. Klette1 ). There exist algorithms without convergence proof [16] and convergence proof without algorithms [27]. In the following, we present a general framework to prove multigrid convergence of surface area estimators. 2.2

Discrete Integration of a Normal Vector Field

→ Let {− n } denote the normal vector field associated to the Euclidean surface S. In the continuous case, the area of S can be expressed as: → − A(S) = n (s)ds (2) S

The main idea of the discrete normal vector field integration approach is to consider a finite sum of dot products between estimated normals at each point of the discrete surface and elementary surface elements as an estimation of ds. Hence, we consider a complex cellular based discrete surface [18]. We define → the normal vector − n ∗ (s) associated to the surface element (or surfel) s as an estimation of the normal vector and we also define an elementary normal vector → − n el at a surfel as the unit vector orthogonal to the cell pointing outward. Hence, the discrete version of equation 2 is: → − → n ∗ (s).− n el (s) (3) EA (Dr (S)) = s∈S

where ’.’ denotes the scalar product. In other words, the idea of this approach is to compute the contribution of each surfel to the global surface area by projecting the surfel according to the normal vector’s direction. We call this process a discrete integration of a normal vector field. This method was originally proposed by Ellis et al. in [7] for the length estimation problem of 2D discrete curves. Independently, Lenoir [20, 19] presents a similar approach as Ellis et al.’s one and proposes an algorithm without multigrid convergence proof for the surface area estimation problem. In the following, we prove that the surface area estimation based on these technics is multigrid convergent if the normal vector estimation is also convergent. We have the following theorem : Theorem 1 Let S be a Jordan surface in E 3 with continuous derivative. The estimator EA (Dr (S)) is multigrid convergent if and only if the estimated normal → vector field converges to {− n }. Hence, we consider a compact surface S in E 3 with continuous derivative. We consider a digitization function fr which is an application from S ∈ E 3 to E 3 and such that dfr converges weakly to the identity (i.e. for any continuous function g we have S g.dfr (ds) → S gds ). We also need that dfr and fr are 1

http://www.citr.auckland.ac.nz/DIG OP/

104


bounded for θ small enough, and fr must converge (in the usual sense) to the identity on S. In other words, fr is a digitization function such that fr (S) converges to S when r increases. The weak convergence property intuitively means that we can evaluate integrals using the discretization. The interest of this formalism is that simple remarks using powerful technical lemmas from functional analysis lead to interesting results. Notice that this definition of a digitization function is not usual, but any classical digitization schemes lead to the existence of such a function (not uniquely defined). In other words, the function fr (S) can be defined by Dr (S). This point will be detailed at the end of this section. We first prove the following lemma: Lemma 1 If dfr converges weakly to the identity, as the precision increases, and → → if − n ∗ (fr (s)) converges uniformly to − n (s), then the discrete integral converges to the surface area and the convergence speed is O(). Proof: A solution for evaluating a surface in a continuous case consists in using − → → − → − S n (s)ds, with n (s) the normal to S in s. The discrete approximation described − → − → above consists in using n ∗ (s )ds (notice that whenever we use an integral fr (S)

notation, this is a finite sum as all elements are constant on a finite number of → areas in usual digitization), with − n ∗ an evaluation of the normal. We only assume → − → ∗ that n (s ) converges uniformly to − n (s) when d(s, s ) → 0. Precisely: lim

sup

→0 d(s,s )≤

→ → |− n ∗ (s ) − − n (s)| = 0

(4)

Notice that the hypothesis can be rewritten as a simple convergence condition, as we work on continuous functions on compact sets. Usual approximation results will directly lead to this formula. The discrete integral is then equal to: → − → − n ∗ (fr (s))dfr (ds) (5) S

Let us denote:

∆=|

S

→ − − → n ∗ (fr (s))dfr (ds) −

→ − − → n (s)ds|

(6)

S

→ → if 1/r is small enough to ensure that |− n ∗ (fr (s)) − − n (s)| < , then (thanks to the finiteness of dfr ) → − → − → → − ∆ = O() + | − n (s)dfr (ds) − n (s)ds| (7) S

S

− Thanks to the smoothness of → n (s) and to the weak convergence of dfr , the term |.| converges to 0 as r increases. This concludes the proof of the lemma.

To conclude the proof of theorem 1, we detail the existence of fr and the weakly convergence of dfr .


105

fr fr

S S’

S’ S

Fig. 1. Illustration of the digitization framework in 2D: (lef t) a continuous curve S, its digitization S and the function fr (vectors), (right) pathological case when S is perpendicular to S For 1/r sufficiently small (such that there is no ”turn back” in the discretiza→ n (s) (k ∈ R) and d(s, s ) tion), we define fr (s) = s such that s ∈ s + k − is minimal (Fig. 1). The existence, uniqueness, continuity, derivability of s are clear almost everywhere except if S is perpendicular to S in a neighborhood of s (Fig. 1-right). All the work consists in proving that this (almost) never occurs. → Fortunately, this occurs for − n (s) ∈ E, with E of measure 0, set of normals to normals in discretizations. E has a measure 0 because E is a finite union of sets of measure 0 (there are a finite number of possible normals in a discretization, hence a finite number of hyperplanes of normals to normals of a discretization). At this point, the proof is not completed. This proves that for a given s, the → probability of having − n (s) ∈ E is 0 ; we need this on any neighborhood of s ∈ S. → Consider now d(s) for s ∈ S the dimension of the set of − n (s ) for s in the neighborhood of s. d is an integer-valued function, constant except for a set of measure 0. Define Si for i ∈ I the different maximal connected subsets of S on which d is different of the embedding dimension. Necessarily I is countable, as each Si has a positive measure, and a finite sum is necessarily countable (easily derived from the fact that the number of Si of measure larger than a given t > 0 → is necessarily finite). Hence, the sum (over i ∈ I) of the probabilities of − n (s) ∈ E for s ∈ Si (the random variable being the choice of the angles defining the grid) is a countable sum of probabilities equal to 0; hence this probability is 0. The convergence of fr towards the identity being clear, we have to verify the weak convergence of dfr to the identity. This is indeed a simple consequence of the fact that fr converges almost everywhere to the identity, thanks to the classical result stating that the almost sure convergence implies the weak convergence of the derivative. Other proofs based upon the Green-Ostrogradsky theorem can be provided as well.

3

Normal Vector Estimation

Meshing the initial binary discrete image is the most popular approach for evaluating normal vectors; generally, it uses Marching Cubes methods [21]: the easy

106


(a)

(b)

(c)

Fig. 2. Obtaining the gradient map in 2D. Binary (a), distance map (b) and gradient map (c) images derivation of normals from facets is used in many shading algorithms [12]. Although smoothing and decimation procedures exist [11], the initial accuracy of triangle positioning is essential. Many imaging scientists have developed normal vector estimation algorithms. A synthesis of the works done until 1992 can be found in [32]. From this date, new approaches considering the 3D neighborhood of the points have appeared. A first approach consists in working on the volume of the image as in [29]. Other algorithms use the surfel decomposition of the digitized object [18], then average surfel elementary normals using convenient weightings and neighborhoods [3, 24]. Another interesting approach consists in considering two dimension slices of a 3D image [20, 28]. In this section, we propose both a local but accurate normal vector estimation based on an adaptive distance gradient filtering method and, a multigrid convergent and optimal in time algorithm. 3.1

The Adaptive Distance Gradient Filtering Method

We present here a simple algorithm to obtain accurate results on all types of shapes whatever their degree of local convexity is. A first approach, proposed in [9], was completed to obtain a fully adaptive algorithm in [10]: a discrete background distance map of the original 3D image is first constructed. For each voxel (volume element) of the object, an elementary gradient vector field is then computed. The normal vector of a surface voxel is obtained by summing each gradient vector in a neighborhood which is determined by angular and symmetry criteria. Thanks to these two criteria, the shape and the size of the working neighborhood are automatically adapted to the local geometry of the surface. This approach can be decomposed in two steps : 1. Obtaining the Gradient Map: we first construct the background distance map of the object. In other words, we label all points in the object with the distance to the closest background point. Many algorithms exist to compute such a map, some of them use Chamfer metrics to approximate the Euclidean metric [1, 30], other methods compute the exact Euclidean distance transform [13, 22]. In practise we choose a d5−7−9−11 Chamfer metric due to its simplicity. Then we compute the gradient map of the distance information using a classical Prewitt first-rank mask derivative filter (Fig. 2).


107

Fig. 3. Summation of gradient vectors in 2D. For each pixel, the gradient vector is represented by a bar in a circle. A gray angular sector, depending on β0 and nR−1 (p), is plotted in each pixel. The bar is embedded in this gray sector for the pixels verifying the angular criterion: this is the case of the point q. The pixel area that is finally retained for the determination of nR (p) is filled in gray. Note that q is not taken into account because sym(q), the symmetric of q according 0 is directly related to nR−1 (p), does not verify the angular criterion. β0 = π−α 2 to the angular width of the smoothest detail that should be accounted for in the considered problem

2. Summation on an Adaptive Shape Neighborhood: the idea is to sum gradient vectors on a spherical neighborhood in order to estimate the normal vector at each surface element. To decide whether a voxel contribution has to be counted or not, it has to be tested in terms of angular and symmetry criteria (Fig. 3). – Angular Criterion: to take the singularities on the surface into account, the gradient vectors whose angle with the normal is more than a fixed angle β0 should be ommitted. β0 is directly related to the angular width α0 of the smallest detail that should be accounted for in the considered problem. In practice we set α0 to 120◦ . This angle provides good results and is particularly relevant to snow images for physical reasons. Indeed, 120◦ corresponds to the maximal angle of hexagonal ice crystals. As these shapes are common in snow images, we wanted to process them correctly. – Symmetry Criterion: to ensure the validity of the normal vector in a point of the surface, the neighborhood should be symmetrical according to the normal in this point. Since these two criteria depend on the normal vector, the following iterative method is applied to each surface voxel p: The first estimation of the normal vector n(p) is given by its gradient vector in p. n(p) is updated by summing each relevant contribution of the R radius

108


spherical cap to the previous normal estimation. Note that to be fully adaptive on all types of images, the connectivity of the neighborhood has to be ensured at each step of the iteration. In practice, as the radius of the maximal spherical cap is limited to a few voxels to save computation time, this connectivity criterion can generally be ommitted. In [10], we prove that this algorithm√is multigrid convergent if the size of the spherical neighborhood decreases in O( θ). 3.2

Multigrid Convergent and Optimal in Time Algorithm

In this section, we describe an algorithm for surface area estimation which is multigrid convergent and optimal in time. The optimality is given by the fact that this algorithm is linear in the number of surface elements of a discrete surface. First of all, we briefly detail algorithms to estimate the normal vectors on a 2D discrete curve. Discrete Normal Vector Field in 2D The discrete tangent on a discrete curve was proposed by Vialard et al. in [31] and based on detecting Discrete Straight Segments (DSS for short). The discrete tangent at a point p of a discrete curve is the longest recognized DSS centered at p (Fig. 4). Many algorithms exist for the DSS recognition, refer to [26] for a survey. Classical DSS recognition algorithms can be used directly but the tangent computation at each point of the curve becomes in O(n2 ). However, we use an optimization proposed by Feschet et al. [8] to compute all tangents in linear time. This idea of this optimal algorithm is illustrated in Fig. 5: if we know the discrete tangent parameters at a point p of a discrete curve, the discrete tangent parameters of some given neighboring pixels are the same. More precisely, if we consider the discrete tangent at p, the first rejected point is denoted by q (Fig. 5-top). Then, if we compute a DSS from q to p, we define the point l as the first rejected point during this process (Fig. 5-middle). Finally, Feschet et al. [8]

p

Fig. 4. An example of a discrete tangent: a DSS recognition process is computed, centered at p. The gray pixels are the tangent extremities (first rejected points)


109

prove that for all pixels between p and the middle m of [ql] (Fig. 5-bottom), the discrete tangent parameters do not change. They finally propose an efficient algorithm whose complexity is linear in the number of pixels that computes the discrete tangent at each point of the discrete curve. This algorithm is based on the definition of arithmetical discrete segment proposed by Reveillès [25] and based on the Debled’s DSS recognition algorithm [6]. Based on this calculus, we define a discrete normal vector at a point of a discrete curve as the unit orthogonal vector to the discrete tangent defined above. In the following, we prove that discrete normal vectors are multigrid convergent in direction. In the sequel we assume the following: 1. Γ : [0, 1] → R2 denotes the underlying continuous curve whose curvature is bounded by C = R1 . 2. At a point pi of Γ we define a tube Ti centered in pi of diameter θ and length L(Ti ) such that Γ belongs to the tube. The θ-enlargement of a tube is the union of this tube and two parts as described in Fig. 6 (for negative values of θ,the notion of enlargement is similarly defined).

3. L(Ti ) ≥ K

θ C,

for any i < N .

This definition of centered tubes is a geometrical explanation of the discrete tangent definition. The begin and end of a tube are the intersections between the axis and the lateral faces (extremities). The pertinence of the last hypothesis is discussed later in the case of particular discretizations. Moreover, we assume that θ is small enough to avoid pathological cases such as half-turns in a tube ( θ2 < R). The goal is to bound on the error, linear in θ. Notice that the hypothesis (2) implies that Γ leaves the (−θ)-enlargements of the Ti ’s through lateral faces.

q p

point of the tangent first rejected point other point of the curve

q point of the recognized line

p 11 00

111 000 000 111 000 first rejected point 111

11 00 11 00 l

other point of the curve

m

p

11 00 11 00 l

1111 0000 0000 1111

q

Fig. 5. Illustration of the optimal time computation of the tangent of a discrete curve

110


θ

L− θ

θ /2

T

β

θ /2 R

θ

R R

Ω+ β Ω

α

Fig. 6. Left: θ-enlargement of tube T . Middle: schema for proof of lemma 2: L is the tube length, the dashed line is the tube contour, the solid line is the (−θ)-enlargement of the tube. Right: case L large in Lemma 1 Lemma 2 The angle2 between Γ and the axis of Ti in the (−θ)-enlargement θ θ of Ti is lower than B = arcsin( L−θ ) + L(Ti )/R. If α such that tan α = 2R−θ verifies R sin(α) ≤ L(Ti ), then the angle is bounded by B = α. Proof: this is the case of largest negative derivative for a projection on a plane including the direction of the tube (Fig. 6). The following holds: Ω ≤ L/R θ sin β ≤ L−θ

(8) (9)

This yields the first result. The particular case in which R sin(α) ≤ L(Ti ) for θ tan α ≤ 2R−θ is illustrated in Fig. 5 6-Right; in this case, L is large, and the derivative must be small enough to keep Γ inthe enlargement of the tube. This

θ proves that the angle is bounded by arc tan( 2R−θ ).

The lemma 2 shows that the maximum angular error between the normal vector √ defined by the tube and the normal vector of Γ in T is bounded by O( θC) under the above assumptions. The error tends to zero when r = 1/θ tends to ∞. Hence, we prove the theorem 1. We now have to study the hypothesis according to which L(Ti ) ≥ K δ/C, for any i < N . This hypothesis is true whenever the discretization is and the polygonalization satisifies that any curve which lies in a tube of radius r with the resolution r is included in a segment (this can easily be proved by considering the fact that for bounding curvature C, the minimal length of Γ before r-deviation from the tangent is Θ( r C )). The classical DSS algorithm does not satisfy this property for some pathological cases. It is likely that adding some reasonable constraints on the curve could remove this condition. For example, if Γ is convex or if Γ has a fixed number of inflection points, the DSS approach verifies the property. Further details can be found in [4]. To summarize these results, we have the following theorem: 2

This angle is the maximal one between a tangent of Γ and the tube axis.

Multigrid Convergence and Surface Area Estimatiorl

Fig. 7. Illustration of the slice decomposition: ( a ) a discrete sphere and (b) one slice along each drection Theorem 2 G w e n a n Euclzdean curve r satzsfysng the above hypotheszs, the estsmated normal vector field based o n dsscrete tangents zs multzgrzd convergent. Optimal in Time Slice Based Approach In this section, we prove both the multigrid convergence and the optimal time computation of a normal vectors estimation algorithm on a discrete surface. In [20,191, Lenoir proposes an approach based on a slice decomposition of the surface (Fig. 7): each surfel of the discrete surface belongs to exactly two slices according to its orientation. Hence, if we compute at a surfel s the two dimensional normal vectors on the two slices, denoted and we can construct the 3D normal vector at s:

z2,

+* n

(8) = -%?I

( s )x

$2

(s)

where ' x ' denotes the vector product. Tellier e t al. [28] also proposed a similar approach using the notion of DSS to estimate the 2D normal vectors. Obviously, we can use the optimal in time algorithm proposed by Feschet et al. to estimate 2D normal vectors and thus obtain an optimal in time 3D normal vector estimation algorithm. T h s algorithm is straightforward: we first compute 2D discrete tangents at each point of each slice of the surface and then we combine these vectors using the vector product according to the orientation of surfels. Finally, we have the result: Theorem 3 G w e n a Jordan surface S in E3 wzth contznuous dertvative and such t h a t the intersectzon r between S and a plane perpendicular t o a grzd arcas

112


satisfies hypothesis of theorem 2. The normal vector estimation on a discrete surface based on a slice decomposition approach is multigrid convergent and the algorithm is optimal in time. → → n ∗ on slices Proof: let us consider a surfel s, its two 2D normal vectors − n ∗ and − 1

2

defined by s and the underlying Euclidean surface S. When the grid resolution increases, the slices will converge to orthogonal curves on S. Furthermore, the → normal vector estimation on each slice is multigrid convergent. Hence, if − n ∗1 and → − → − → − ∗ n 2 converges respectively to n 1 and n 2 which are exact normal vector at s → → → on S. Then − n∗ = − n ∗1 × − n ∗2 is also multigrid convergent. The proof of the linear complexity in the number of surface elements is straightforward: each surface element belongs to exactly two slices. Since 2D computation on slices are independent and done in an optimal time, the global cost of this algorithm is linear in the number of surfel and thus optimal for the problem.

Finally, we can present the corollary: Corollary 1 The surface area estimation using the discrete normal vector field integration and based on slice decomposition approach for normal vector estimation is multigrid convergent and optimal in time. Proof: since the normal vector field is multigrid convergent and computed in a linear time in the number of surface elements, the surface area estimation is also multigrid convergent according to theorem 1. Furthermore, since the discrete normal vector field integration process is done in a linear time in the number of surfels, the global algorithm is also optimal for the problem.

4

Results

In this part are remembered some validations obtained with the Adaptive Distance Gradient Filtering (ADGF) and the voxel projection algorithms (VP) [10]. These algorithms were tested on simulated data and natural snow images processed by X-ray microtomography [2, 5] and compared to the Marching Cubes algorithm (MC for short) [21]. After a short validation of the normals obtained by the ADGF algorithm, special attention will be paid to the multigrid convergence of the surface area estimation. 4.1

Normal Vectors

To allow comparison with the ADGF algorithm, the normals obtained by the MC algorithm were allocated to the nearest center of the Marching Cubes grid and then added on the height neighboring voxels. For a voxel belonging to the surface of the object, the resulting normal vector was assumed to be the Marching Cubes normal vector. For all the tests with the ADGF algorithm, β0 was set to 30◦ . It amounts to detect angles sharper than 120◦ (see the angular criterion paragraph in section 3.1.2). The radius of the maximal spherical cap was set to 5 voxels to save computation time.


12

113

Marching Cubes - mean Marching Cubes - std deviation ADGF - mean ADGF - std deviation

10

Degree

8

6

4

2

0 20

40

60

80

100

120

140

Grid resolution

Fig. 8. Comparison of the angular error for spheres Multigrid Tests on Spheres Tests were carried out on a set of digitized spheres from 1 to 140 voxels of radius. Spheres were obtained according to the definition adopted in [24] and [28]. The theoretical normal vector is the vector starting from the sphere center through the sphere voxel centers. Errors obtained with the ADGF and MC algorithms are compared in Fig. 8. We can note that for spheres of small radius, errors are larger. This can easily be explained by the fact that a discrete sphere of small size is not a sphere but a polyhedron. Other Geometrical Shapes Here are compared ADGF and MC algorithms on other simulated data. Rendering images of the Fig. 9 point out the accuracy of ADGF method on convex, concave, rounded and sharp shapes. Snow Images Image renderings of the same melt-freeze crust snow sample obtained by X-ray microtomography at the ESRF are presented in Fig. 10. This snow sample is constituted of both rounded shapes and sharp angles and allows to check the effectiveness of the ADGF algorithm. The image processed with this algorithm is compared to the image rendering obtained with the MC algorithm. Note that some distorsions to optimal normal vector field occur when the image is one voxel wide, in this case, the object surface is not defined. 4.2

Surface Area

This section compares the surface area estimations obtained with the VP method and the Marching Cubes and Convex Hulls algorithms. Convergence Tests The convergence tests were carried out on spheres, cylinders and thick spherical caps. Relative errors of surface area estimation are

114

David Coeurjolly et al. Cylinder

Hexagonal Low-tilted prism spherical cap

Tilted cube

Voxel images

Image rendering obtained by MC algorithm

Image rendering obtained by the ADGF algorithm

Fig. 9. Comparison between the ADGF and MC algorithms on geometrical shapes plotted against grid resolution. The series of spheres are the same as defined in the first part of this article. The cylinders used verify the following conditions: -

-

+

height = 1 2 . r a d i u s the cylinder axis is colinear t o (Oz) axis

Thick spherical caps were obtained as follows: -

-

radius,,, = 2 . radiusmi, to make an easier theoretical estimation of this object, the two spheres were truncated so that their surface area is the half part of a whole sphere surface area (see voxel image of the low-tilted spherical cap in Fig. 9).

The convergence rates of both convex hull and voxel projection surface area estimators are obviously much better than the convergence rate of the MC algorithm (Fig. 11). In addition, the voxel projection estimator allows t o compute surface areas of non-convex objects. In Fig. l l ( a ) , the surface area estimator does not suffer from the ADGF normal vector estimation: curves with theoretical normal vectors or estimated vectors are similar.

Local Surface Ratio Visualization Local surface area ratio maps were obtained as follows: the ADGF normal determination algorithm was applied t o geometrical objects. Then the projection ratios were computed and plotted in gray level for each point of the discrete surface (Fig. 12).


115

(a) Whole sample processed with the ADGF algorithm.

(b) Marching Cubes.

(c) ADGF algorithm.

Fig. 10. Image renderings of a same snow sample with the two methods. Meltfreeze crust sample taken at le Col de Porte, France. Image edge: 300 voxels Surface Area Estimation on Snow Samples The comparison of surface area estimations of snow sample images are presented in table 1. For these estimations, the whole numerical object was taken into account.

116


35 30

Convex hull Marching cubes VP Theoretical normal vectors integration

10

25

Relative error in %

Relative error in %

100

Convex hull Marching cubes VP Theoretical normal vectors integration

20 15 10

1

0.1

0.01 5 0

0.001 20

40

60 80 Grid resolution

100

120

140

20

40


100

120

140

(a) Spheres (logarithmic scale on the right).

25

100

Convex hull Marching Cubes VP

Convex hull Marching Cubes VP

20 Relative error in %

Relative error in %

10 15

10

1

0.1 5

0

0.01 20

40


100

120

140

20

40


100

120

140

(b) Cylinders (logarithmic scale on the right).

35

Marching Cubes VP

Relative error in %

30 25 20 15 10 5 0 0

20

40

60

80

100

120

140

Grid resolution

(c) Thick spherical caps.

Fig. 11. Surface area estimations using different algorithms on multigrid objects


117

Table 1. Surface areas estimated on three snow samples (rounded grain, wet grain and melt-freeze crust) by the two algorithms. The VP algorithm was applied to the normals obtained by the ADGF method. The deviation between the marching cubes estimation and the VP algorithm is expressed in percent of the VP estimation

As we can expect from a qualitative estimation of the grain size, the lower the grain diameter is, the higher the surface area is. The two surface area estimators give relatively similar results. But, for the three samples, the surface area estimated by the MC algorithm is superior to the value obtained with the VP algorithm. We can see from Fig. 10(b) that the roughness of the surface described by the MC algorithm is high. This proves the smoothing ability of the ADGF algorithm.

5

Conclusion and Future Works

In this article, we have presented a multigrid convergent and optimal in time algorithm for the surface area estimation of discrete objects. We have also presented a local but adaptive algorithm to estimate the surface area that allows us to control the estimation according t o our practical motivations. We have shown experimental results of this estimator on various discrete shapes. An important future work of this approach is a comparative evaluation of the optimal in time algorithm regarding to surface area estimators in the literature.

Fig. 12. Maps of surface area ratios estimated on geometric objects: sphere (a), cube "in axes" (b) and tilted rounded cube (c)

118


References [1] G. Borgefors. Distance transformations in digital images. Comput. Vision Graphics Image Process., 34(3):344–371, Jun. 1986. 106 [2] J.-B. Brzoska, C. Coléou, B. Lesaffre, S. Borel, O. Brissaud, W. Ludwig, E. Boller, and J. Baruchel. 3D visualization of snow samples by microtomography at low temperature. ESRF Newsletter, 32:22–23, Apr. 1999. 112 [3] L. Chen, G. T. Herman, R. A. Reynolds, and J. K. Udupa. Surface shading in the Cuberille environment. IEEE Comput. Graph. Appl., 5(12):33–43, Dec. 1985. 106 [4] D. Coeurjolly, I. Debled-Rennesson, and O. Teytaud. Segmentation and length estimation of 3d discrete curves. In Digital and Image Geometry, pages 295–313. Springer Lecture Notes in Computer Science, 2243, 2001. 110 [5] C. Coléou, B. Lesaffre, J.-B. Brzoska, W. Ludwig, and E. Boller. Threedimensional snow images by X-ray microtomography. Ann. Glaciol., 32:75–81, 2001. 112 [6] I. Debled-Rennesson and J.P. Reveillès. A linear algorithm for segmentation of digital curves. In International Journal of Pattern Recognition and Artificial Intelligence, volume 9, pages 635–662, 1995. 109 [7] T.J. Ellis, D. Proffitt, D. Rosen, and W. Rutkowski. Measurement of the lengths of digitized curved lines. Computer Graphics and Image Processing, 10:333–347, 1979. 103 [8] F. Feschet and L. Tougne. Optimal time computation of the tangent of a discrete curve : Application to the curvature. In 8th International Workshop in Discrete Geometry for Computer Imagery, pages 31–40. Springer-Verlag, LNCS, 1568, 1999. 108 [9] F. Flin, J.-B. Brzoska, B. Lesaffre, C. Coléou, and P. Lamboley. Computation of normal vectors of discrete 3D objects: application to natural snow images from X-ray tomography. Image Anal. Stereol., 20:187–191, Nov. 2001. 106 [10] F. Flin, J.-B. Brzoska, B. Lesaffre, C. Coléou, P. Lamboley, D. Coeurjolly, O. Teytaud, G. Vignoles, and J.-F. Delesse. An adaptive filtering method to evaluate normal vectors and surface areas of 3D objects. Application to snow images from X-ray tomography. submitted to IEEE Trans. Image Processing. 106, 108, 112 [11] P.-L. George and H. Borouchaki. Delaunay triangulation and meshing - Application to finite elements. Hermes, Paris, 1998. in French. 106 [12] H. Gouraud. Continuous shading of surfaces. IEEE Trans. Comput., C-20(6):223– 228, Jun. 1971. 106 [13] T. Hirata. A unified linear-time algorithm for computing distance maps. Information Processing Letters, 58(3):129–133, May 1996. 106 [14] A. Jonas and N. Kiryati. Digital representation schemes for 3D curves. Pattern Recognit., 30(11):1803–1816, 1997. 102 [15] Y. Kenmochi and R. Klette. Surface area estimation for digitized regular solids. In L. J. Latecki, D. M. Mount, and A. Y. Wu, editors, Proc. Vision Geometry IX, volume 4117, pages 100–111. SPIE, Oct. 2000. 102 [16] R. Klette and H. J. Sun. Digital planar segment based polyhedrization for surface area estimation. In C. Arcelli, L. P. Cordella, and G. Sanniti di Baja, editors, International Workshop on Visual Form 4, volume 2059 of Lect. Notes Comput. Sci., pages 356–366. Springer-Verlag, 2001. 101, 103 [17] R. Klette and J. Zunic. Multigrid convergence of calculated features in image analysis. Journal of Mathematical Imaging and Vision, 13:173–191, 2000. 101


119

[18] V. A. Kovalevsky. Finite topology as applied to image analysis. Comput. Vision Graphics Image Process., 46(2):141–161, May 1989. 103, 106 [19] A. Lenoir. Des outils pour les surfaces discrètes. PhD thesis, Université de Caen, 1999. in French. 103, 111 [20] A. Lenoir, R. Malgouyres, and M. Revenu. Fast computation of the normal vector field of the surface of a 3-D discrete object. In Discrete Geometry for Computer Imagery: 6th Workshop, volume 1176 of Lect. Notes Comput. Sci., pages 101–112. Springer-Verlag, 1996. 103, 106, 111 [21] W. E. Lorensen and H. E. Cline. Marching cubes: a high resolution 3D surface construction algorithm. Computer Graphics, 21(4):163–169, 1987. SIGGRAPH’87 Conference Proceedings (Anaheim, California). 105, 112 [22] A. Meijster, J. B. T. M. Roerdink, and W. H. Hesselink. A general algorithm for computing distance transforms in linear time. In Mathematical Morphology and its Applications to Image and Signal Processing, pages 331–340. Kluwer, 2000. 106 [23] J.C. Mullikin and P.W. Verbeek. Surface area estimation of digitized planes. Bioimaging, 1:6–16, 1993. 101 [24] L. Papier and J. Fran¸con. Evaluation de la normale au bord d’un objet discret 3D. Revue de CFAO et d’informatique graphique, 13:205–226, 1998. in French. 106, 113 [25] J.P. Reveillès. Géométrie discrète, calcul en nombres entiers et algorithmique. PhD thesis, Univ. Louis Pasteur, Strasbourg, 1991. 109 [26] A. Rosenfeld and R. Klette. Digital straightness. In Sébastien Fourey, Gabor T. Herman, and T. Yung Kong, editors, International Workshop on Combinatorial Image Analysis, volume 46 of Electronic Notes in Theoretical Computer Science, Temple University , Philadelphia, Pennsylvania, U.S.A., August 2001. Elsevier Science Publishers. 108 [27] F. Sloboda and B. Zatko. On approximation of Jordan surfaces in 3D. In G. Bertrand, A. Imiya, and R. Klette, editors, Digital and Image Geometry, volume 2243 of Lect. Notes Comput. Sci., pages 365–388. Springer-Verlag, 2001. 101, 103 [28] P. Tellier and I. Debled-Renesson. 3D discrete normal vectors. In G. Bertrand, M. Couprie, and L. Perroton, editors, Discrete Geometry for Computer Imagery: 8th Workshop, volume 1568 of Lect. Notes Comput. Sci., pages 447–458. SpringerVerlag, 1999. 106, 111, 113 [29] G. Th¨ urmer and C. A. W¨ uthrich. Normal computation for discrete surfaces in 3D space. Computer Graphics Forum, 16(3):15–26, Aug. 1997. Proceedings of Eurographics’97. 106 [30] B. J. H. Verwer. Local distances for distance transformations in two and three dimensions. Pattern Recognit. Lett., 12:671–682, Nov. 1991. 106 [31] A. Vialard. Geometrical parameters extraction from discrete paths. Discrete Geometry for Computer Imagery, 1996. 108 [32] R. Yagel, D. Cohen, and A. Kaufman. Normal estimation in 3D discrete space. The visual computer, 8(5-6):278–291, Jun. 1992. 106

Geometric Structure and Randomness in Texture Analysis and Synthesis Georgy Gimel’farb, Linjiang Yu, and Dongxiao Zhou Centre for Image Technology and Robotics Department of Computer Science, Tamaki Campus University of Auckland, Private Bag 92019, Auckland, New Zealand Abstract. Gibbs random field models describe image textures in terms of geometric structure and energy of pixel interactions. The interaction means statistical interdependence of signals, the structure is given by characteristic pixel neighbourhoods, and the energy depends on signal cooccurrences over the neighbourhoods. In translation invariant textures all the neighbourhoods have the same relative geometry. The interaction structure of such a texture is reflected in a model-based interaction map (MBIM) giving spatial distribution of the interaction energies over a large neighbourhood. We show that due to scale / orientation robustness, the MBIM allows to partition a given training sample into tiles acting as structural elements, or texels. Large-size textured images can be synthesised by replicating the training texels.

1

Introduction

Image textures are typically introduced by examples such as, for instance, the Brodatz’s or MIT Media Laboratory VisTex collections [2, 15], because they lack formal definitions. Known informal definitions refer usually to repetitive areas with similar visual appearance called texels (texture elements) [10] or textons [12]. Each area is characterised by location, shape, size, and spatial distribution of signals (grey values, colours, or local features derived from them). Because of a wide variety of images perceived visually as homogeneous textures, there is no universal constructive way of defining texels, ranges of their allowable variations, and rules of spatial arrangement in a single texture. Today’s supervised synthesis of realistic textures is confined basically to spatially homogeneous images with translation invariant repetitive appearance. The goal is to generate images that are visually similar to a given small-size training sample but are of much larger size. Most notable progress has been made towards the realistic synthesis using (i) the Markov Chain Monte Carlo (MCMC) generation of samples of a Markov random field (MRF) of signals or features and (ii) the pixel-wise or block sampling of the training image [4, 5, 7, 8, 14, 16, 17, 18]. In both cases certain size-independent spatial signal statistics of the training sample are approximated with similar statistics of the synthesised images. The MCMC synthesis is performed with pixel-wise stochastic relaxation based on generative MRF models, specified with joint Gibbs probability distibutions (GPD) of image signals or features. The models belong to popular in T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 120–140, 2003. c Springer-Verlag Berlin Heidelberg 2003

Geometric Structure and Randomness in Texture Analysis and Synthesis

121

applied statistics exponential families of distributions suited to well-established optimality criteria such as maximum entropy or minimum discrimination information [1, 13]. They are also derived from the Gauss’s principle that the average values of certain signal statistics are the maximum likelihood estimators [3]. As underscored in [1], these properties of the exponential families constitute the folklore of modern statistics. In texture modelling they are sometimes derived anew as, for instance, the maximum entropy principle in [19, 20]. Generic Gibbs random field (GGRF) models in [7] describe single-resolution textures in terms of geometric structures and energies of multiple pairwise pixel interactions. The interaction structure of a particular texture is given by a generally disjoint translation invariant pixel neighbourhood formed by characteristic interacting pixel pairs. An interaction means statistical dependence of signals in neighbouring pixels. The interaction energy is a sum of Gibbs potentials for all the interacting pairs, each potential being a function of the pairwise signal co-occurrence. In addition to the above optimality properties, this exponential family yields a very simple non-parametric analytical first approximation of the potentials. The approximate potentials are directly proportional to sufficient signal statistics for the model, namely, to centred normalised grey level co-occurrence histograms (GLCH). Each GLCH is collected over the family of translation invariant pixel pairs in the training sample. Most characteristic families giving the interaction structure are estimated analytically or empirically from the training sample using the approximate potentials [7, 8, 18]. The Markov/Gibbs models can be extended to a multi-resolution image representation such as quad-trees and Gaussian or Laplacian pyramids [11]. Generally, the hierarchical model involves conditional GPDs of signals or features at the adjacent levels of the pyramid. For computational feasibility, simplifying assumptions have to be involved in most cases, e.g. inter-level statistical independence or conditional normal distributions. Multiresolution non-parametric Markov random fields in [16] accounts for conditional relative frequency distributions of the multiscale signals over small heuristically chosen pixel neighbourhoods such as squares of size 7 × 7. Assuming that all the neighbourhoods are statistically independent, the GPD is roughly estimated from the training sample using clustering techniques. Texture synthesis with stochastic relaxation is not practical in this model so that the textures are synthesised by approaching the training distributions with a direct constrained multi-resolution optimisation. The MCMC synthesis results in realistic outputs for many textures. But due to computational complexity, it usually cannot be used to synthesise large-size images. A more practical alternative way of producing textures much larger than the training sample is based on direct approximation of the pixel neighbourhoods in the training sample with similar neighbourhoods in the goal images. In most cases, the neighbourhoods are involved implicitly, e.g., through spatial features of a multi-resolution image representation such as relative frequencies of the topdown signal co-occurrences along a Laplacian or steerable complex wavelet image pyramid [4, 17]. The generated pyramid replicates, up to a certain similarity threshold, the corresponding top-down chains of the training signals.

122

Georgy Gimel’farb et al.

A single-resolution non-parametric texture sampling in [5] exploits a square pixel neighbourhood of the heuristically chosen size as an explicit structurepreserving element. To generate a texture, a small seed is taken randomly from the training sample. Then the seed is extended in a pixelwise mode to a desired size. Each new pixel is added by an equiprobable choice from among the already synthesised pixels with closely similar neighbourhoods. Unfortunately in the general case the pixelwise extrapolation accumulates local errors rupturing the desired texture. A patch-based non-parametric sampling suppresses the degradation by adding at each step not a single pixel but a small rectangular patch of a heuristically chosen size [14] or by replicating patches of the training sample and forming zigzag borders between the overlapped patches in order to minimise signal discontinuities across the borders [6]. These techniques allow to realistically synthesise different textures. But mostly they cannot produce largesize images because of large amounts of computations per pixel. The patch-based non-parametric sampling [6, 14] is fast but the quality of image synthesis depends on how adequate the heuristically chosen patches are. This paper analyses the geometric structure of pairwise pixel interactions for certain types of regular mosaics in order to relate the proper choice of signal groups picked up from the training sample to a model-based interaction map (MBIM). The MBIMs show spatial distributions of the partial energies of pairwise pixel interactions over the large rectangular pixel neighbourhood in the training sample [7, 8]. The unique pattern of the spatial energy clusters in the MBIM reflects scale, orientation, and periodicity of the texture. Figures 1 and 2 show examples of the training samples and MBIMs for natural mosaics from [2, 15]. These and similar MBIMs are robust with respect to small-scale random geometric and photometric distortions of the repetitive regions. As is shown below, the MBIMs allow for estimating tilings that split the training sample of a mosaic into such regions. The estimated tiles are considered as individual texels, and the large-size prototypes of the mosaic are formed fast by stretching the training samples by replication of either a single texel [9] or all the texels. The paper is organised as follows. Section 2 discusses in brief a generic Gibbs random field (GGRF) model with multiple pairwise pixel interactions [7] and behaviour of the MBIMs at different levels of image resolution. Estimation of tiles, or texels for a particular mosaic is considered in Section 3. Experiments in synthesising large-size prototypes of natural mosaics from the estimated tiles are described in Section 4.

2

Structure of Pairwise Pixel Interactions and MBIMs

Let (R = [i = (x, y) : x = 0, . . . , M − 1; y = 0, . . . , N − 1] and Q = {0, . . . , qmax } denote a finite rectangular arithmetic lattice and a finite signal set, respectively. Let Ca = {(i, i + a) : i, i + a ∈ R} be a family of translation invariant pixel pairs, or cliques of the neighbourhood graph, separated by the same relative inter-pixel shift a in the lattice. Let A be a finite set of the inter-pixel shifts that specifies geometric structure of translation invariant pairwise pixel interactions


D1

D6

D14

D20

D21

D34

D34,−5◦

D34,−20◦

D53

D55

D65

D83

123

a

b

a

b

a

b

Fig. 1. Training samples (a) and MBIMs (b) for the mosaics D1, D6, D14, D20, D21, D34 (original and rotated −5◦ and −20◦), D53, D55, D65, and D83 from [2]

124

Georgy Gimel’farb et al. D95

D101

D101,5◦

D101,20◦

D102

Tile7

Textile25

Fabrics8

a

b

a

b

Fig. 2. Training samples and MBIMs for the mosaics D95, D101 (original and rotated 5◦ and 20◦ ), D102 from [2] and Tile 7, Textile 25, Fabrics 8 from [15] in terms of most distinctive interacting pixel pairs (i, i + a). The structure yields the characteristic neighbourhood NA,i = {(i − a), (i + a) : a ∈ A; (i ± a) ∈ R} of each pixel i ∈ R. Let Va : Q × Q → R = (−∞, ∞) denote a bounded Gibbs potential function giving the quantitative strength of pairwise pixel interactions in the clique family Ca . All the pairwise interactions are specified by the potential V = {Va : a ∈ A}. Then the GGRF model with multiple pairwise pixel interactions involves the arbitrary structure A and potential V that both are specific for a particular spatially homogeneous texture. The model relates the probability of a digital image g : R → Q: Ea (g|Va ) (1) Pr(g|A, V) ∝ exp a∈A


125

to the partial energies Ea (g|Va ) of pixel interactions in the clique families: Ea (g|Va ) = Va (gi , gi+a ) (2) (i,i+a)∈Ca

Let Fa = {Fa (q, s|g) : q, s ∈ Q} denote the relative frequency distribution of signal co-occurrences (gi = q, gi+a = s) in the cliques of the family Ca : Fa (q, s|g) =

1 |Ca |

δ(gi − q)δ(gi+a − s)

(i,i+a)∈Ca

where δ(· · · ) is the Kronecker function: δ(z) = 1 if z = 0 and δ(z) = 0 otherwise. The partial energy of Eq. (2) is easily rewritten as the dot product of potential and frequency vectors Va = [Va (q, s) : (q, s) ∈ Q × Q] and Fa (g) = [Fa (q, s|g) : (q, s) ∈ Q × Q] to reveal that the GPD in Eq. (1) belongs to the exponential families of distributions [7]: Va (q, s) · Fa (q, s|g) Ea (g|Va ) = |Ca | (q,s)∈Q×Q

As shown in [7], the clique families specified by a large search set W of the intra-clique shifts a can be ranked by their approximate relative partial energies 1 Fa (q, s|g) Fa (q, s|g) − (3) Ea,0 (g) = |Q|2 2 (q,s)∈Q

The characteristic structure A is estimated by parallel or sequential selection of the top-rank energies of Eq. (3) [7, 8]. Spatial distribution of these energies over the large set W of the intra-clique shifts represents the MBIM that characterises uniquely the periodic pattern of the mosaic. Every spatial position a ≡ (x, y) of the MBIM indicates the relative Gibbs energy Ea,0 (g ◦ ) for the intra-clique shift a ∈ W, the diametrically opposite shifts (x, y) and (−x, −y) representing the same clique family. Scaled images 81×81 of the model-based interaction maps (MBIM) in Figs. 1 and 2 show the structure of translation invariant pairwise pixel interactions for each training sample 128 × 128 of the natural Brodatz and MIT VisTex mosaics [2, 15]. In these examples, every MBIM is supported by the set W of the relative intra-clique x- and y-shifts in the range [−40, 40] representing 3280 clique families. The partial energies are greycoded so that the larger the energy value, the darker the dot. Examples of how the MBIMs depend on image resolution (scale) at different levels of image pyramids are shown in Figs. 3–6. The MBIMs for the mosaics D53 and D102 in Figs. 3 and 4 correspond to the three bottom levels 640 × 640, 320 × 320, and 160 × 160 of the quad-tree pyramids. It is easily seen that these MBIMs are notably similar in spite of minor changes. In these two cases the training samples are of the same size 160 × 160 at each level. The MBIMs for

126

Georgy Gimel’farb et al. Level 0

Level 1

Level 2

a

b

c

Fig. 3. Training samples (a), scaled MBIMs (b), and spatial energy clusters (c) found by thresholding the MBIMs for the mosaic D53 at the three levels of the quad-tree pyramid the Level 0 (initial resolution) is formed with the set W of the intra-clique shifts corresponding to the window of size 81×81. Level 1 with twice coarser resolution and Level 2 with four times coarser resolution are formed with the windows of size W 41 × 41 and 21 × 21, respectively. The latter two MBIMs are magnified in Figs. 3 and 4 to the same scale as the MBIM for Level 0 to emphasise similarity between the interaction structures at different resolution. Similarity between the MBIMs at various levels holds for other types of image pyramids, e.g. the Gaussian and Laplacian ones as shown in Figs. 5 and 6 for the mosaics D53 and D101. In these cases the bottom-level images have the same resolution as the textures D53 and D101 at Levels 3 and 1, respectively, in Figs. 3 and 4. The windows W are of the half-image size. The repetitive details of each texture are reflected in the MBIMs to a lesser degree when the images are of lower resolution. Nonetheless, even Levels 2 and 3 of the pyramids in Figs. 5 and 6 allow to detect the periodicity of the MBIMs.

Geometric Structure and Randomness in Texture Analysis and Synthesis Level 0

Level 1

127

Level 2

a

b

c

Fig. 4. Training samples (a), scaled MBIMs (b), and spatial energy clusters (c) found by thresholding the MBIMs for the mosaic D101 at the three levels of the quad-tree pyramid

3

Rectangular Tiles to Synthesise Mosaics

As seen from Figs. 1–6, the MBIMs for the Gibbs model of Eq. (1) reflect to a large extent the scale, orientation, and repetitiveness of a particular mosaic. Until image resolution is sufficiently fine, positions of and pitches between the replicate spatial clusters of the top-rank partial interaction energies relate to rectangular or hexagonal mutual arrangement and shapes of the repetitive parts. These latter can be considered as texels, but their choice is not unique. The same mosaic yields many equivalent tilings such that the grid of tiles is shifted with respect to the image lattice. The size of the tile is bounded only below. Generally, the shape and scale of such texels as well as their photometric characteristics vary for different samples of a particular texture and even within the same sample. The central cluster in the MBIMs in Figs. 1–4 describes most energetic closerange interactions relating mainly to a uniform background. But all the peripheral clusters reflect the repetitive pattern of the texture. In principle, the shape

128

Georgy Gimel’farb et al. Level 0

Level 1

Level 2

Level 3

a

b

c

d

Fig. 5. Scaled textures (a,c) and MBIMs (b,d) at the four levels of the Gaussian (a,b) and Laplacian (c,d) image pyramids for the mosaic D53

and orientation of a mask for cutting texels from a translation invariant mosaic is governed by the central and top-rank peripheral clusters of energies. At least in the case of regular mosaics, the shape of such a mask should also facilitate dense tiling (e.g., a regular triangle, rectangle, or hexagon). For simplicity, our consideration is restricted to only rectangular texels with an arbitrary but fixed orientation and size. After the MBIM is formed by collecting the GLCHs for a given training sample g ◦ and computing the partial energies {Ea,0 (g ◦ ) : a ∈ W} of Eq. (3), such texels can be estimated with the following simple heuristic scheme: (i) Detect spatial clusters of the clique families with top energies by threshold¯ + c · σE where E ¯ and ing the MBIM using an empirical threshold: E ∗ = E σE denote the mean value and standard deviation of the relative partial energies for the training sample and c is a chosen factor.

Geometric Structure and Randomness in Texture Analysis and Synthesis Level 0

Level 1

Level 2

129

Level 3

a

b

c

d

Fig. 6. Scaled textures (a,c) and MBIMs (b,d) at the four levels of the Gaussian (a,b) and Laplacian (c,d) image pyramids for the mosaic D101 (ii) If there are no peripheral clusters in addition to the central cluster around the origin a = (0, 0), then the texture has no regular structure to be described by the texel-based tiling. (iii) Otherwise rotate the Cartesian coordinate axes around the origin of the MBIM until the peripheral clusters projected onto one of the axes and not occluded with (that is, not projected onto the projection of) the central cluster have the minimum span. (iv) Find along each coordinate axis a projected position of the clique family with the largest energy provided that the cluster containing this family is not occluded with the central cluster; if the positions along both axes are produced by the same clique family, then resolve the ambiguity by using the cluster containing the family with the second largest energy that does

130


Table 1. Rectangular tiles estimated by projecting onto the rotated coordinate axes the top-rank energy clusters detected in the MBIMs using different thresholds (Ncl.f denotes numbers of selected cliques) Factor Ncl.f Rectangle Orientation (x × y) pixels angle, ◦ c D20: Figs. 1 and 8 0.5 423 19.39 × 18.00 -0.63 260 20.32 × 17.99 -2.13 1.0 191 19.47 × 18.00 -0.75 1.5 157 19.47 × 18.00 -0.75 2.0 94 19.47 × 18.00 -0.75 3.0 D34: Figs. 1 and 8 0.0 1035 34.98 × 12.77 -2.00 664 34.98 × 12.77 -2.00 0.4 598 33.14 × 24.51 18.75 0.5 354 32.45 × 26.09 22.00 1.0 251 32.62 × 26.10 -21.25 1.5 D53: Figs. 1 and 8 0.5 891 22.00 × 15.76 -0.63 459 22.00 × 15.71 -0.75 1.0 195 22.00 × 15.71 -0.75 1.5 64 22.00 × 15.56 -1.13 2.0 33 22.00 × 15.56 -1.13 3.0 D55: Figs. 1 and 10 0.5 498 24.00 × 22.00 -0.88 293 24.00 × 22.00 -0.63 1.0 152 23.99 × 21.99 -1.25 1.5 88 23.99 × 21.99 -1.25 2.0 33 23.99 × 21.99 -1.13 3.0

Factor Ncl.f Rectangle Orientation c (x × y) pixels angle, ◦ D101: Figs. 2 and 9 0.5 301 14.54 × 13.99 -2.25 1.0 230 14.54 × 13.99 -2.25 1.5 189 14.54 × 13.99 -2.25 2.0 145 14.51 × 13.99 -2.13 3.0 97 14.24 × 14.00 -1.00 D101, 5◦ : Figs. 2 and 9 0.5 329 15.01 × 14.01 0.87 1.0 225 15.03 × 14.03 3.12 1.5 186 15.03 × 14.03 3.37 2.0 151 11.09 × 10.31 -38.63 3.0 98 15.03 × 14.03 3.25 D101, 20◦ : Figs. 2 and 9 0.5 402 14.87 × 13.93 20.25 1.0 276 14.86 × 13.93 20.50 1.5 204 14.86 × 13.92 18.62 2.0 145 14.85 × 13.90 17.37 3.0 97 14.85 × 13.90 17.37 Fabrics0008: Figs. 2 and 10 0.5 556 20.00 × 20.00 0.87 1.0 305 20.00 × 20.00 0.75 1.5 211 19.98 × 19.98 2.87 2.0 150 20.00 × 20.00 1.12 3.0 72 19.98 × 19.98 2.87

not belong to and is not occluded by the central cluster or by the cluster with the largest energy. The most important problem is to find the proper orientation of the tiling. Mosaics with rectangular patterns of the top-rank energies in the MBIM yield sufficiently accurate and stable orientation estimates at Step (iv). But the estimates are less stable for the MBIMs with a hexagonal structure such as for the texture D34 in Fig. 1. More stable estimates of the orientation angles are obtained by using only the peripheral clusters with the largest and the second largest energy [9]: (iii) Assign to each cluster the largest energy of the clique family belonging to that cluster and choose the first and the second top-rank energy cluster such that they do not occlude each other from the MBIM origin. (iv) Find the minimum rotation angle placing the centre of one of the chosen two clusters onto the Cartesian coordinate axis.


131

Table 2. Rectangular tiles estimated by using the orientation angles for the first and second top-rank energy clusters in the MBIMs with the thresholding factor c = 2.5 (F08, T07, T25 stand for Fabrics0008, Tile0007, and Textile0025, respectively); the thresholds in the range 1 ≤ c ≤ 3 yield almost the same results Texture:

D1

D6 D14 D20 D21

Rectangle x-size, pixels: 21.5 21.0 29.0 19.0 7.0 Rectangle y-size, pixels: 33.02 34.0 23.0 18.0 7.0 -1.74 0.0 0.0 0.0 0.0 Orientation angle, ◦ : Texture:

D34 D53 0◦ -5◦ -20◦ 70.0 70.26 34.47 44.0 14.0 28.24 42.15 16.0 0.0 -4.9 -22.31 0.0

D65 D83 D95

D101 D102 0◦ 5◦ 20◦ Rectangle x-size, pixels: 44.0 42.0 25.96 14.0 15.1 15.2 19.0 Rectangle y-size, pixels: 32.0 52.0 36.76 14.0 14.04 13.92 19.0 0.0 0.0 -1.64 0.0 3.81 19.65 0.0 Orientation angle, ◦ :

D55 24.0 22.0 0.0

F08 T07 T25 20.0 20.0 0.0

9.0 20.0 8.0 14.0 0.0 0.0

The first scheme yields more precise estimates of the desired angles but the second scheme yields better results for the mosaics with both rectangular and hexagonal patterns of the repetitive parts (such as, e.g., D34 or D83). In the general case the MBIMs should be processed more in detail in order to choose the shape, size, and orientation of the texel masks for each particular mosaic. The above estimation schemes show only that it is basically possible to formally derive some types of simple texels from the MBIMs.

4

Experiments in Generating Mosaics

Parameters of the tilings obtained with both estimation schemes are given in Tables 1 and 2. The factor c of the energy threshold at Step (i) in the range 0.5 ≤ c ≤ 2 for the first scheme (Table 1) and 1 ≤ c ≤ 3 for the second scheme (Table 2) gave similar results for most of the textures used in the experiments. Using the second scheme, the orientation angle of each tiling was estimated with the precision up to 0.25◦ . Figure 7 demonstrates the rectangular tilings estimated with the first scheme and overlaid onto the training samples of size 128 × 128. Examples of the generated larger textures of size 256 × 256 are shown in Figs. 8 – 10. Parameters of the tiles used in these experiments are boldfaced in Table 1. One of the two generated images of each texture is an approximate texture prototype obtained by replicating a single tile selected arbitrary as the texel. The second image results from a texel-based “rubber stretching” of the training sample performed by replicating all the tiles in accord with the size ratio between the desired and training samples. For simplicity, the stretching is performed without interpolating the successive replicas. Similar results are obtained for other mosaics shown in Figs. 1 and 2. Some texture prototypes of size 800 × 170 are presented in Figs. 11 – 13.

132

Georgy Gimel’farb et al. D1

D6

D14

D20

D21

D34

D34,−5◦

D34,−20◦

D53

D55

D65

D83

D95

D101

D101,5◦

D101,−20◦

D102

Tile7

Textile25

Fabrics8

Fig. 7. Examples of tilings of the training samples

5

Conclusions

Our experiments show that spatial patterns of most energetic pairwise pixel interactions in the MBIMs reflect basic geometric structure of specific texture types such as repetitive translation invariant mosaics. The pattern is robust with respect to random signal variations over the texture and scale / orientation transformations of the images. Quantitative analysis of the MBIMs makes it pos-


133

D20

a

b

D53

a

b

Fig. 8. Replication of a single texel (a) and texel-based stretching (b) of the training samples

sible to partition the training sample of a mosaic onto a set of similar tiles where each tile can be considered as a simple texel. In principle, the prototypes of such texels can be obtained by processing all the mutually registered tiles. The processing should enhance their spatial compatibility and reduce relative geometric and photometric deviations. Then the large-size textures can be generated fast by replicating the texels. The replication of a single texel prototype forms an idealised large-size prototype of the mosaic. More realistic large-size samples can be obtained by transforming each replica of the texel in such a way as to preserve continuous transitions between the adjacent tiles. The alternative way is to perform a texel-based “rubber stretching” of the training sample by replicating each training tile while preserving its relative position with respect to other training tiles. Such texel-based synthesis has obvious drawbacks in that any singularity of a chosen tile or a group of tiles is replicated verbatim. But relative deterministic and random deviations of the training tiles can be significantly eliminated by joint processing that produces an idealised prototype of the texel with no sin-

134


D101

a

b

D101,5◦

a

b

D101,20◦

a

b

Fig. 9. Replication of a single texel (a) and texel-based stretching (b) of the training sample


135

D55

a

b

Fabrics08

a

b

Fig. 10. Replication of a single texel (a) and texel-based stretching (b) of the training sample

gularities. The verbatim copies in a “rubber stretched” training sample can be suppressed in part by bilinear or spline interpolation of the intermediate replicas of the tiles. The above texel-based description is inadequate for stochastic (irregular) textures that have no repetitive pattern of the peripheral energy clusters. Both the deterministic interaction structure and randomness of a stochastic texture are better described in terms of translation invariant joint and conditional GPDs of signals over the characteristic pixel neighbourhoods. But the texel-based description is practical at least for various translation invariant regular mosaics. Realistic texture synthesis is based at present on direct approximation of certain heuristically selected pixel neighbourhoods in the training sample with the like neighbourhoods of the generated image. The analysis of the MBIMs for the regular mosaics results in the explicit characteristic neighbourhoods, and the use of them to estimate the proper tiles (or texels) simplifies and accelerates the synthesis of the large-size realistic images.

136


Fig. 11. Replication of a single texel: mosaics D1, D6, D14, and D21


137

Fig. 12. Replication of a single texel: mosaicsD34 (rotated 0◦ , −5◦ ; −20◦ ), and D65

138


Fig. 13. Replication of a single texel: mosaics D83, D95, Tile07, and Textile25


139

Acknowledgements This work was supported by the Royal Society of New Zealand Marsden Fund under Grant 9143/3600771 and the University of Auckland Research Committee under Grants 9343/3414113 and 9343/3600529.

References [1] Barndorff-Nielsen, O.: Information and Exponential Families in Statistical Theory. John Wiley & Sons: Chichester (1978). 121 [2] Brodatz, P.: Textures: A Photographic Album for Artists and Designers. Dover Publications: New York (1966). 120, 122, 123, 124, 125 [3] Campbell, L. L.: Equivalence of Gauss’s principle and minimum discrimination information estimation of probabilities. Annals of Mathematical Statistics 41:3 (1970) 1011–1015. 121 [4] De Bonet, J. S.: Multiresolution sampling procedure for analysis and synthesis of texture images. In: Proc. ACM Conf. Computer Graphics SIGGRAPH’97 (1997) 361–368. 120, 121 [5] Efros, A. A., Leung, T. K.: Texture synthesis by non-parametric sampling. In: Proc. IEEE Int. Conf. Computer Vision ICCV’99, Greece, Corfu, Sept. 1999, vol.2 (1999) 1033–1038. 120, 122 [6] Efros, A. A., Freeman, W. T.: Image quilting for texture synthesis and transfer. In: Proc. ACM SIGGRAPH’01, Los Angeles, Calif., August 2001, pp.341–346. 122 [7] Gimel’farb, G. L.: Image Textures and Gibbs Random Fields. Kluwer Academic: Dordrecht (1999). 120, 121, 122, 125 [8] Gimel’farb, G.: Characteristic interaction structures in Gibbs texture modeling. In: Blanc-Talon, J., Popescu, D. C. (Eds.): Imaging and Vision Systems: Theory, Assessment and Applications. Nova Science: Huntington, N.Y. (2001) 71–90. 120, 121, 122, 125 [9] Gimel’farb, G.: Estimation of texels for regular mosaics using model-based interaction maps. In: Proc. Joint IAPR Int. Workshops SSPR 2002 and SPR 2002, Windsor, Ontario, Canada, August 2002 (Lecture Notes in Computer Science 2396), Springer: Berlin (2002) 177–185. 122, 130 [10] Haralick, R. M., Shapiro, L. G.: Computer and Robot Vision, vol.2. AddisonWesley: Reading (1993). 120 [11] Heeger, D., Bergen, J.: Pyramid-based texture analysis/synthesis. In: Computer Graphics 29 (Proc. ACM SIGGRAPH’95 Conf., Los Angeles, CA.) (1995) 229– 238. 121 [12] Julesz, B.: Textons, the elements of texture perception, and their interactions. Nature, no. 290 (1981) 91–97. 120 [13] Kullback, S.: Information Theory and Statistics. John Wiley & Sons: New York (1959). 121 [14] Liang, L., Liu, C., Xu, Y., Guo, B., Shum, H. Y.: Real-Time Texture Synthesis by Patch-Based Sampling. MSR-TR-2001-40. Microsoft Research (2001). 120, 122 [15] Picard, R., Graszyk, S., Mann, S., e.a.: VisTex Database. MIT Media Lab.: Cambridge, Mass. (1995). 120, 122, 124, 125 [16] Paget, R., Longstaff, I. D.: Texture synthesis via a noncausal nonparametric multiscale Markov random field. IEEE Trans. on Image Processing 7 (1998) 925–931. 120, 121

140


[17] Portilla, J., Simoncelli, E. P.: A parametric texture model based on joint statistics of complex wavelet coefficients. Int. Journal on Computer Vision 40 (2000) 49–71. 120, 121 [18] Zalesny, A., Van Gool, L.: A compact model for viewpoint dependent texture synthesis. In: Pollefeys, M., Van Gool, L., Zisserman, A., Fitzgibbon, A. (Eds.): 3D Structure from Images (Lecture Notes in Computer Science 2018). Springer: Berlin (2001) 124–143. 120, 121 [19] Zhu, S. C., Wu, Y., Mumford, D.: Minimax entropy principle and its application to texture modeling. Neural Computation 9 (1997) 1627–1660. 121 [20] Zhu, S. C., Wu, Y., Mumford, D.: Filters, random fields and maximum entropy (FRAME): towards a unified theory for texture modeling. Int. Journal of Computer Vision 27 (1998) 107–126. 121

Morphological Texture Analysis Using Optimization of Structuring Elements Akira Asano1 , Miho Miyagawa2, and Mitsuhiko Fujio2 1

Hiroshima University, Higashi-Hiroshima, Hiroshima 739-8521 Japan [email protected] http://kuva.mis.hiroshima-u.ac.jp/ 2 Kyushu Institute of Technology, Iizuka, Fukuoka 820-8502 Japan

Abstract. This paper proposes a method of texture analysis using morphological size distribution. Our framework is based on the concept that a texture is described by estimation of primitive, size distribution of grains derived from the primitive, and spatial distribution of the grains. We concentrate on estimation of primitive using an assumption on grain size distribution. We assume a model that grains are derived from one primitive, and a uniform size distribution since we consider target textures containing grains of various sizes. Thus the structuring element used for the measurement of size distribution is optimized to obtain the most uniform size density function. The optimized structuring element is an estimate of the primitive under the assumption. Simulated annealing algorithm is employed for the optimization.

1

Introduction

Texture recognition and discrimination are important aims of image processing, as well as object shape recognition in images. A lot of texture analyzing methods have been proposed, and the texture classification and segmentation are main objectives among them. The texture classification and segmentation require characterization of textures, i. e. evaluation of features describing local or global characteristics of the target texture. According to [1], the texture characterization approaches can be divided into four categories: statistical, geometrical, model-based and signal processing. Our approach is categorized into geometrical one. Among these methods, the geometrical approach considers a texture to be composed of grains derived from primitives, and attempts to describe the shapes of primitives. Our framework is based on the concept that a texture is described by estimation of primitive [2][3], size distribution of grains derived from the primitive, and spatial distribution [4] of the grains. In this paper, we concentrate on estimation of primitive shape. We concentrate on estimation of primitive using an assumption on grain size distribution. We assume a model that grains are derived from one primitive, and a uniform size distribution since we consider target textures containing grains of various sizes. Our assumption is applicable to many natural textures except regularly T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 141–152, 2003. c Springer-Verlag Berlin Heidelberg 2003

142

Akira Asano et al.

periodic texture like textiles, since shapes of grains depend on the materials of which the entities of textures are made and sizes of grains are often affected by random process. The morphological size density function [5][6], which is a measure of size distribution of an image, of a texture relative to a structuring element, which is a small object used for morphological operations, is highly dependent on the shape of the structuring element. If we assume that the size distribution of grains is uniform, the size density function of the target texture relative to the structuring element similar to the primitive will be also uniform. Suppose that the size density function of the texture is calculated using structuring elements of various shapes. The structuring element that yields the most uniform function is regarded as an estimate of the texture under this assumption. We employ the simulated annealing for finding the optimal structuring element that makes the size density function uniform, and estimate the shape of primitive by the shape of the optimized structuring element. An application of the morphological size distribution to the texture analysis has been already proposed [7][8][9]. In [7], they prepare structuring elements of some typical shapes, for example circle, line segment, etc., and calculate size distributions relative to each structuring element. They divide the target texture into some segments each of which belongs to a class of typical shape, by measuring characteristics of each size distribution. This method is effective for segmentation of textures, however, it does not describe characteristics of textures but only classifies textures into a limited number of predefined typical shapes. In [8] and [9] the shapes of structuring elements are also predefined. On the contrary, our method estimates the primitive without any limitation of predefined typical shapes. In Section 2 we review the mathematical morphology and size distribution. We explain our method of texture primitive estimation and the experimental results of estimation for binary structuring elements in Section 3. We show an extension to gray scale structuring elements and its experimental results in Section 4. Finally we conclude our work in Section 5.

2 2.1

Morphological Size Distribution Mathematical Morphology

Mathematical morphology is a framework for modeling the processes of human recognition of visual information. Mathematical morphological operations are shift-invariant image manipulations and can be decomposed into two simple basic operations - dilation and erosion. We first explain these basic operations in terms of binary structuring elements for binary images. These operations are defined as set operations. An image is assumed to be a set of vectors that correspond to pixel positions in the interior of image objects. Let X denote a set representing an image. Let B be another set called a structuring element. The structuring element corresponds to the window

Morphological Texture Analysis Using Optimization of Structuring Elements

143

of a filter. Minkowsky set subtraction X B and addition X ⊕ B are defined as follows1 : Xb (1) X B = b∈B

X ⊕B =

Xb

(2)

b∈B

where Bx denotes translation of B by x, defined as Bx = {b + x|b ∈ B}.

(3)

ˇ and X ⊕ B, ˇ respectively, The erosion and the dilation are defined as X B ˇ is defined as where B ˇ = {−b|b ∈ B}. B

(4)

The following interpretations of erosion and dilation are well-known [10]: ˇ = {x|Bx ⊂ X} X B

(5)

ˇ = {x|Bx ∩ X = ∅} X ⊕B

(6)

ˇ is the locus of the origin According to the above expressions, erosion X B ˇ of B sweeping the whole interior of X, i. e. scraping of X by B. Dilation X ⊕ B is the locus of the origin of B sweeping not to be apart from B, i. e. expansion of X by B. The morphological operations for gray scale images are defined using maximum and minimum operations: The morphological operations for gray scale images and binary structuring elements are defined as follows: Consider a function X(x) that denotes a gray scale image, where x is a pixel position. Then ˇ and dilation X ⊕ B ˇ are defined as follows: erosion X B ˇ = min X(x + b) X B

(7)

ˇ = max X(x + b). X ⊕B

(8)

b∈B

b∈B

The morphological operations for gray scale images and gray scale structuring elements are defined as follows: Consider a function B(b) that denotes a gray scale structuring element, where b is a relative pixel position from the origin of the structuring element. Let w(B) be the extent of structuring element, i. e. ˇ and dilation X ⊕ B ˇ are B(b) is defined only if b ∈ w(B). Then erosion X B defined as follows: ˇ {X B}(x) = min {X(x + b) − B(b)} b∈w(B)

1

Another popular defnition of Minkowsky set operations are explained in [5].

(9)

144

Akira Asano et al.

ˇ {X ⊕ B}(x) = max {X(x + b) + B(b)}. b∈w(B)

(10)

Two other important and basic operations, called opening and closing, are derived from the erosion and dilation. Opening XB and closing X B are defined as follows: ˇ ⊕B XB = (X B)

(11)

ˇ B. X B = (X ⊕ B)

(12)

The following interpretation of opening is well-known: Bx . XB =

(13)

Bx ⊆X

This indicates that opening of X by B is the locus of B itself when B sweeps the whole interior of X. Opening eliminates all the portion smaller than the structuring element while preserving the residual portion of the objects. Closing is the complement: it fills up smaller spots while preserving the other portion of the background. This ability of opening and closing to quantify the size of objects is important for the definition of the size distribution. 2.2

Pattern Spectrum and Size Density

The pattern spectrum is a morphological method of measuring size distribution. The pattern spectrum of size n relative to an structuring element is defined as the pixelwise difference between the target image opened by a homothetic set of structuring element of size n and that opened by structuring element of size n + 1. Let nB be the homothetic set of a structuring element B of size n, defined as follows: nB = B ⊕ B ⊕ · · · ⊕ B (n−1) additions 0B = {0}.

(14)

Then the pattern spectrum of size n relative to the structuring element B for image X, denoted as PS(X, B, n), is defined as follows: {XnB (x) − X(n+1)B (x)} (15) PS(X, B, n) = x∈whole image where ”−” sign denotes the pixelwise difference. Since the opening removes the portion smaller than the structuring element, the difference of the images opened by the structuring elements of size n and


pattern spectrum

results of opening

structuring elements

145

difference

object

extracted shapes sizes

Fig. 1. Pattern spectrum size n + 1 contains the portion whose size is exactly n. Figure 1 illustrates the process to calculate the pattern spectrum. The original image is decomposed into sets of the structuring elements of various sizes. The normalized pattern spectrum, which is defined as the ratio of the original pattern spectrum to the sum of the pixel values over the whole original image, is often used. The value of the normalized one at each size indicates the ratio of the portions at a size to the whole image. The normalized pattern spectrum is often called the size density because of the analogy between the normalized pattern spectrum and the probability density function. The size density F(X, B, n) is defined as follows: F(X, B, n) =

PS(X, B, n)

x∈whole

3 3.1

image X(x)

.

(16)

Method of Primitive Estimation Employment of Simulated Annealing

Our method estimates the primitive of a texture based on an assumption on the size distribution of grains. We explain our idea using an example texture shown in Fig. 2. This natural texture contains various sized grains similar to a line segment from right-top to left-bottom. Figure 3 illustrates the size density functions relative to structuring elements of four-direction line segments. The

146

Akira Asano et al.

size density(%)

size density function relative to the line segment from right-top to left-bottom is almost uniform, and in this case the structuring element is similar to the primitive.

directions of line segments 30

20

10

0

Fig. 2. An example texture

0

1

2

3

4

5

6 size

Fig. 3. Size density functions relative to four line segments

This example suggests that the structuring element producing the most uniform size density function is the best estimation of the primitive in these four line segments. Here the average E[F(X, B, n)] and V[F(X, B, n)] of the size distribution are defined as follows: l

F(X, B, n) l+1 l {F(X, B, n) − E[F(X, B, n); l]}2 V[F(X, B, n); l] = n=0 l+1 E[F(X, B, n); l] =

n=0

(17)

where l denotes the upper limit of sizes for the calculation of average and variance. According to this example, we directly reach to the following idea: If we assume a natural and random texture as a target, the structuring element which estimates the primitive is obtained by seeking the structuring element that yields the smallest variance of the size density function, V[F(X, B, n); l]. This is achieved by iterating a slight modification of the structuring element and the evaluation of the size distribution. Such procedure is equivalent to the optimization process of the structuring element under the criterion of the smallest V[F(X, B, n); l], and can be realized by an algorithm known as the simulated annealing. We first calculate the size density function relative to the structuring elements of the four-direction line segments, and choose a structuring element that yields


147

Fig. 4. Modification of binary structuring element. The selected pixel, surrounded by thick line, is altered

the smallest variance. We use this structuring element as the initial one of the optimization process. The optimization process is described as the following set of the procedures: 1. Calculating the size density function relative to the initial structuring element and the variance as the evaluation function. 2. Modifying a pixel of the structuring element. For binary structuring element, inverting a pixel of the structuring element from black to white or white to black, as shown in Fig. 4. 3. Calculating the size distribution and the variance again with the modified structuring element. If the variance is smaller than that with the structuring element before modification, this modification is accepted and fixed. If the variance does not decrease, the acceptance probability of the modification is determined by a function of the increment of the variance and the number of repetition. The larger increment of the variance and the larger number of iteration cause the smaller the probability of acceptance. If the modification is not accepted, it is cancelled. 4. Iterating the procedures 2. and 3. until the modification is not accepted any more. Practically, the process is terminated when it is iterated sufficiently many times.

3.2

Experimental Results for Binary Structuring Elements

In our experiment, we use the evalution function ER that is exactly identical to the variance V[F(X, B, n); l]: ER = V[F(X, B, n); l].

(18)

We set l = 3, i.e. the range of sizes in which this value is calculated to 0, 1, 2, 3. The structuring element is restricted to the 5 × 5 binary ones. The function to determine the probability of acceptance of the modification, denoted P(∆ER), is defined as follows: 1 if ∆ER < 0 1 (19) P(∆ER) = if ∆ER ≥ 0 ∆ER 1+exp( ) Ti

148

Akira Asano et al.

where ∆ER denotes the difference of ER before and after the modification, as follows: ∆ER = ERafter − ERbefore

(20)

where ERafter and ERbefore denote ERs after and before the modification, respectively. Ti is called ”temperature,” which is a function of the number of iteration and defined here as follows:

T0 = 105 (21) Ti+1 = 0.98Ti−1 where i denotes the number of repetition of the procedures 2 and 3 explained in the previous section. In this experiment the iteration is terminated at 1000 times. Figure 5 shows example textures used for the experiments. Each of the pairs A and A , B and B , and C and C is extracted from one texture image. Figure 6 shows extracted structuring elements, or estimated primitives, of the example textures. An identical structuring element is extracted from a pair of textures from one texture image, and distinguishable structuring elements are extracted from different textures.

A

B

C

A′

B′

C′

Fig. 5. Example textures


A

B

C

A′

B′

C′

149

Fig. 6. Estimated primitive by binary structuring elements

4 4.1

Extension of the Method to Gray Scale Structuring Elements Method

Since the target texture images are gray scale ones, it is natural that the primitive is estimated more precisely by gray scale structuring elements. The morphological operations with gray scale structuring elements are defined in Eqs. 9 and 10. We get from these equations that a pixel of value zero in the extent of structuring element and a pixel out of the extent have absolutely different contributions in these operations. This means that we should optimize the extent and the pixel values separately. Thus we apply our optimization algorithm in two stages: The first stage is the same as the algorithm for binary structuring element explained in the previous chapter. This stage optimizes the extent of structuring element. The second stage optimizes the pixel values of the gray scale structuring element whose extent is optimized at the first stage. The second stage performs almost similarly to the first stage, however, pixels are modified in different manner in the procedure of simulated annealing. At this stage, the initial structuring element is defined as the gray scale one whose extent is the same as the binary structuring element optimized by the first stage and whose pixel values are all zero. The structuring element is modified by selecting one pixel and increasing the value of this pixel by one, as shown in Fig. 7. 4.2

Experimental Results

Figure 8 shows the estimated gray scale structuring elements using the same evaluation function at the second stage of optimization as used at the first stage.

150

Akira Asano et al.

1 2 0 0 3 3 1 1 1 2

2 2 0 0 3 3 1 1 1 2

Fig. 7. Modification of gray scale structuring element. The selected pixel, surrounded by thick line, is increased by 1

These structuring elements are not admissible as estimates of primitives, since every structuring element contains one high peak surrounded by smaller pixel values and completely different sets of pixel values are obtained from similar textures. The reason why such curious structuring elements are obtained is supposed as follows; The gray scale opening of an object relative to structuring elements with one high peak and its homothetic expansions of small sizes yields objects almost identical to the original one. Since the size densities at the small sizes are almost zero in this case, the evaluation function, or the variance of the size densities, is also very small. To avoid this problem, we introduce another evaluation function to solely the second stage of optimization, as follows: ER =

F(X, B, 0) . F(X, B, 1) − F(X, B, 0)

(22)

8 0

25539 48 8 0

8 0 0

A

207119

8

247 0

B

C 0

0

40 247

0 0

55215207 239 47

A′

0 55

B′

0 0

C′

Fig. 8. Estimated primitives by gray scale structuring elements using the same evaluation function as binary case


151

72 247 16

15 39 231 247 39

A

247

0 247247 247247

40

B

C 80

24 40

247 56 215 231 79

A′

247

0 247247 247247

B′

32

C′

Fig. 9. Estimated primitives by gray scale structuring elements using modified evaluation function The value F(X, B, 0) is the sum of pixel values eliminated by opening using the structuring element of size 1. The smaller ER is, the better the resultant structuring element fits the primitive. The experimental results using the new evaluation function at the second stage are shown in Fig. 9. These results indicate the brightness distribution of primitive within the extent derived at the first stage of optimization. These results also indicate that the obtained structuring elements are similar for similar textures and distinguishable for visually different textures.

5

Conclusions

We have proposed in this paper a novel method of estimating texture primitive based on the morphological size distribution. Our method optimizes the structuring element to fit the grains under the criterion of making the size density function uniform. We have applied the simulated annealing algorithm to modification of the structuring element in the process of the optimization. We have applied our method to the optimization of gray scale structuring element, as well as binary one. In the case of gray scale one, the optimization algorithm is applied in two stages: The first stage optimizes the extent of structuring element and the second optimizes the pixel value in the extent. We have concentrated in this paper on the estimation of primitive using an assumption on the size distribution of grains. We have shown the results of texture discrimination as a demonstration of reliability of the estimation. Evaluation of the discrimination ability is an open problem.

152

Akira Asano et al.

We are now working on extension to primitive estimation without any assumption on the size distribution of grains [11], and to that of multiprimitive textures [12]. Estimation of size distribution and spatial distribution of grains, and their modification for generation of other textures related to the original one, are also our future works.

Acknowledgments This research has been supported by the Grant-in-Aid for Scientific Research of the Japan Society for the Promotion of Sciences, Nos. 12750337 and 14750297.

References [1] Ojala T. and Pietik¨ ainen M., ”Texture classification” in R. B. Fisher, ed., CVonline: The Evolving, Distributed, Non-Proprietary, On-Line Compendium of Computer Vision. (http://www.dai.ed.ac.uk/CVonline/LOCAL COPIES/ OJALA1/texclas.htm) 141 [2] Asano, A.: Texture Analysis Using Morphological Pattern Spectrum and Optimization of Structuring Elements. Proc. 10th International Conference on Image Analysis and Processing (1999) 209–214 141 [3] Asano, A., Miyagawa, M., and Fujio, M.: Texture Modelling by Optimal Gray Scale Structuring Elements using Morphological Pattern Spectrum. Proc. 15th International Conference on Pattern Recognition 3 (2000) 479–482 141 [4] Gimel’farb, G.: ”Characteristic interaction structures in Gibbs texture modelling” in J. Blanc-Talon and D. Popescu, eds., Imaging and Vision Systems: Theory, Assessment and Applications. Nova Science Publishers (2001) 71–90 141 [5] Heijmans, H. J. A. M.: Morphological Image Operators. Academic Press (1994) 142, 143 [6] Maragos, P.: Pattern Spectrum and Multiscale Shape Representation. IEEE Trans. Pattern Anal. Machine Intell. 11 (1989) 701–706 142 [7] Dougherty, E. R., Newell, J. T., and Pelz, J. B.: Morphological Texture-Based Maximuml-Likelihood Pixel Classification Based on Local Granulometric Moments. Pattern Recognition 25 (1992) 1181–1198 142 [8] Sand, F. and Dougherty, E. R.: Asymptotic granulometric mixing theorem: morphological estimation of sizing parameters and mixture proportions. Pattern Recognition 31 (1998) 53–61 142 [9] Sand, F. and Dougherty, E. R.: Robustness of granulometric moments. Pattern Recognition 32 (1999) 1657–1665 142 [10] Serra, J.: Image Analysis and Mathematical Morphology Academic Press (1982) 143 [11] Asano, A., Ohkubo, T., Muneyasu, M., and Hinamoto, T.: Texture Primitive Description Using Morphological Skeleton. Proc. International Symposium on Mathematical Morphology VI (2002) 101–108 152 [12] Asano, A., Endo, J., and Muraki C.: Multiprimitive Texture Analysis Using Cluster Analysis and Size Density Function. Proc. International Symposium on Mathematical Morphology VI (2002) 109–116 152

Unifying Quantitative, Semi-quantitative and Qualitative Spatial Relation Knowledge Representations Using Mathematical Morphology Isabelle Bloch Ecole Nationale Supérieure des Télécommunications Département TSI - CNRS URA 820, 46 rue Barrault, 75013 Paris, France Tel: +33 1 45 81 64 85 - Fax: +33 1 45 81 37 94 [email protected]

Abstract. One of the powerful features of mathematical morphology lies in its strong algebraic structure, that finds equivalents in set theoretical terms, fuzzy sets theory and logics. Moreover this theory is able to deal with global and structural information since several spatial relationships can be expressed in terms of morphological operations. The aim of this paper is to show that the framework of mathematical morphology allows to represent in a unified way spatial relationships in various settings: a purely quantitative one if objects are precisely defined, a semiquantitative one if objects are imprecise and represented as spatial fuzzy sets, and a qualitative one, for reasoning in a logical framework about space. Keywords: mathematical morphology, spatial relationships, spatial reasoning.

1

Introduction

One of the powerful features of mathematical morphology lies in its strong algebraic structure, that finds equivalents in set theoretical terms, fuzzy sets theory and logics. Moreover this theory is able to deal with local information, based on the concept of structuring element, but also with more global and structural information since several spatial relationships can be expressed in terms of morphological operations (mainly dilations). We consider here topological relationships (which include part-whole relationships such as inclusion, exclusion, adjacency, etc.) and metric relationships (distances and directional relative position), the interest of these relations being highlighted in very different types of works (vision, GIS, cognitive psychology, artificial intelligence, etc.). The aim of this paper is to show that the framework of mathematical morphology allows to represent in a unified way spatial relationships in various settings: a purely quantitative one if objects are precisely defined, a semi-quantitative one if objects are imprecise and represented as spatial fuzzy sets, and a qualitative one, for T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 153–164, 2003. c Springer-Verlag Berlin Heidelberg 2003

154

Isabelle Bloch

reasoning in a logical framework about space. The proposed framework, briefly presented in Section 2, allows us to address three questions. We first consider the problem of defining and computing spatial relationships between two objects, in both the crisp and fuzzy cases (Section 3). Then in Section 4 we propose a way to represent spatial knowledge in the spatial domain. Finally in Section 5 we show that spatial relationships can be expressed in the framework of normal modal logics, using morphological operations applied on logical formulas. This can be useful for symbolic (purely qualitative) spatial reasoning. These threse types of problems are further developed in [5].

2 2.1

Basic Morphological Operations, Fuzzy and Logical Extensions Classical Morphology

Let us first recall the definitions of dilation and erosion of a set X by a structuring element B in a space S (e.g. IRn , or ZZn for discrete spaces like images), denoted respectively by DB (X) and EB (X) [23]: = ∅}, DB (X) = {x ∈ S | Bx ∩ X

(1)

EB (X) = {x ∈ S | Bx ⊆ X},

(2)

where Bx denotes the translation of B at point x. In these equations, B defines a neighborhood that is considered at each point. It can also be seen as a relationship between points. ¿From these two fundamental operations, a lot of others can be built [23]. 2.2

Fuzzy Mathematical Morphology

Several definitions of mathematical morphology on fuzzy sets with fuzzy structuring elements have been proposed in the literature (see e.g. [8, 24, 11]). Here we use the approach using t-norms and t-conorms as fuzzy intersection and fuzzy union. However, what follows applies as well if other definitions are used. Erosion and dilation of a fuzzy set µ by a fuzzy structuring element ν, both defined in a space S, are respectively defined as: Eν (µ)(x) = inf T [c(ν(y − x)), µ(y)],

(3)

Dν (µ)(x) = sup t[ν(y − x), µ(y)],

(4)

y∈S

y∈S

where t is a t-norm, c a fuzzy complementation, and T is the t-conorm associated to t with respect to c. These definitions guarantee that most properties of morphological operators are preserved [8, 21].

Spatial Relation Knowledge Representations

2.3

155

Morpho-logics

Now, we express morphological operations in a symbolic framework, using logical formulas. Let us consider a language generated by a finite set of propositional symbols and the usual connectives. Kripke’s semantics is used. The set of all worlds is denoted by Ω. The set of worlds where a formula ϕ is satisfied is M od(ϕ) = {ω ∈ Ω | ω |= ϕ}. The underlying idea for constructing morphological operations on logical formulas is to consider set interpretations of formulas and worlds. Since in classical propositional logics, the set of formulas is isomorphic to 2Ω , up to the logical equivalence, we can identify ϕ with M od(ϕ), and then apply set-theoretic morphological operations. We recall that M od(ϕ ∨ ψ) = M od(ϕ) ∪ M od(ψ), M od(ϕ ∧ ψ) = M od(ϕ) ∩ M od(ψ), and M od(ϕ) ⊆ M od(ψ) iff ϕ |= ψ. Using these equivalences, dilation and erosion of a formula ϕ are defined as [7]: = ∅}, (5) M od(DB (ϕ)) = {ω ∈ Ω | B(ω) ∩ M od(ϕ) M od(EB (ϕ)) = {ω ∈ Ω | B(ω) |= ϕ},

(6)

where B(ω) |= ϕ means ∀ω ∈ B(ω), ω |= ϕ. The structuring element B represents a relationship between worlds and defines a “neighborhood” of worlds. It can be for instance defined as a ball of a distance between worlds [18]. The condition for dilation expresses that the set of worlds in relation to ω should be consistent with ϕ, i.e.: ∃ω ∈ B(ω), ω |= ϕ. The condition for erosion is stronger and expresses that ϕ should be satisfied in all worlds in relation to ω. Now we consider the framework of normal modal logics [10] and use an accessibility relation as relation between worlds. We define an accessibility relation from any structuring element B (or the converse) as: R(ω, ω ) iff ω ∈ B(ω). Let us now consider the two modal operators ✷ and ✸ defined from the accessibility relation as [10]: M, ω |= ✷ϕ iff ∀ω ∈ Ω, R(ω, ω ) ⇒ M, ω |= ϕ,

(7)

M, ω |= ✸ϕ iff ∃ω ∈ Ω, R(ω, ω ) and M, ω |= ϕ,

(8)

where M denotes a standard model related to R. Equation 7 can be rewritten as: ω |= ✷ϕ ⇔ B(ω) |= ϕ, (9) which exactly corresponds to the definition of erosion of a formula, and Equation 8 can be rewritten as: ω |= ✸ϕ ⇔ B(ω) ∩ M od(ϕ) = ∅,

(10)

which exactly corresponds to a dilation. This shows that we can define modal operators derived from an accessibility relation as erosion and dilation with a structuring element: (11) ✷ϕ ≡ EB (ϕ),

156

Isabelle Bloch

✸ϕ ≡ DB (ϕ).

(12)

The modal logic constructed from erosion and dilation has a number of theorems and rules of inference, detailed in [4, 6], which increase its reasoning power. All these definitions and properties extend to the fuzzy case, if we consider fuzzy formulas, for which M od(ϕ) is a fuzzy set of Ω. A fuzzy structuring element can be interpreted as a fuzzy relation between worlds. Its usefulness will appear for expressing intrinsically vague spatial relationships such as directional relative position.

3

Computing Spatial Relationships from Mathematical Morphology: Quantitative and Semi-quantitative Setting

In this Section we consider the problem of defining and computing spatial relationships between two objects. We consider the general case of a 3D space S, where objects can have any shape and any topology, and consider both topological and metric relationships [17, 13]. We distinguish also between relationships that are mathematically well defined (such as set relationships, adjacency, distances) and relationships that are intrinsically vague, like relative directional position, for which fuzzy definitions are appropriate. If the objects are imprecise, as is often the case if they are extracted from images, then the semi-quantitative framework of fuzzy sets proved to be useful for their representation, as spatial fuzzy sets (i.e. fuzzy sets defined in the space S), and both types of relations have then to be extended to the fuzzy case. Results can also be semi-quantitative, and provided in the form of intervals or fuzzy numbers. 3.1

Set Relationships

Computing set relationships, like inclusion, intersection, etc. if the objects are precisely defined does not call for specific developments. If the objects are imprecise, stating if they intersect or not, or if one is included in the other, becomes a matter of degree. A degree of inclusion can be defined as an infimum of a t-conorm (as for erosion). A degree of intersection µint can be defined using a supremum of a t-norm (as for fuzzy dilation) or using the fuzzy volume of the t-norm in order to take more spatial information into account. The degree of non-intersection is then simply defined by µ¬int = 1 − µint . The interpretations in terms of erosion and dilation allow to include set relationships in the same mathematical morphology framework as the other relations. 3.2

Adjacency

Adjacency has a large interest in image processing and pattern recognition, since it denotes an important relationship between image objects or regions. For any


157

two subsets X and Y in the digital space ZZn , the adjacency of X and Y can be expressed in terms of morphological dilation, as: = ∅, DB (Y ) ∩ X = ∅, X ∩ Y = ∅ and DB (X) ∩ Y

(13)

where B denotes the elementary structuring element associated to the chosen digital connectivity. This structuring element is usually symmetrical, which means that the two conditions DB (X) ∩ Y = ∅ and DB (Y ) ∩ X = ∅ are equivalent, so only one needs to be checked. Adjacency between fuzzy sets can be defined by translating this expression into fuzzy terms, by using fuzzy dilation. The binary concept becomes then a degree of adjacency between fuzzy sets µ and ν: µadj (µ, ν) = t[µ¬int (µ, ν), µint [DB (µ), ν], µint [DB (ν), µ]].

(14)

This definition represents a conjunctive combination of a degree of nonintersection µ¬int between µ and ν and a degree of intersection µint between one fuzzy set and the dilation of the other. This definition is symmetrical, reduces to the binary definition if µ, ν and B are binary, and is invariant with respect to geometrical transformations. 3.3

Distances

The importance of distances in image processing is well established. Their extensions to fuzzy sets (e.g. [25]) can be useful for several aspects of image processing under imprecision. Mathematical morphology allows to define distances between fuzzy sets that combine spatial information and membership comparison. In the binary case, there exist strong links between mathematical morphology (in particular dilation) and distances (from a point to a set, and several distances between two sets), and this can also be exploited in the fuzzy case. The advantage is that distances are then expressed in set theoretical terms, and are therefore easier to extend with nice properties than usual analytical expressions. Here we present the case of Hausdorff distance. The binary equation defining the Hausdorff distance: dH (X, Y ) = max[sup d(x, Y ), sup d(y, X)] x∈X

(15)

y∈Y

can be expressed in morphological terms as: dH (X, Y ) = inf{n, X ⊆ Dn (Y ) and Y ⊆ Dn (X)}.

(16)

A distance distribution, expressing the degree to which the distance between µ and µ is less than n is obtained by translating this equation into fuzzy terms: ∆H (µ, µ )(n) = t[ inf T [Dνn (µ)(x), c(µ (x))], inf T [Dνn (µ )(x), c(µ(x))]], x∈S

x∈S

where c is a complementation, t a t-norm and T a t-conorm.

(17)

158

Isabelle Bloch

A distance density, expressing the degree to which the distance is equal to n, can be derived implicitly from this distance distribution. A direct definition of a distance density can be obtained from: dH (X, Y ) = 0 ⇔ X = Y,

(18)

and for n > 0 dH (X, Y ) = n ⇔ X ⊆ Dn (Y ) and Y ⊆ Dn (X) and X ⊂ Dn−1 (Y ) or Y ⊂ Dn−1 (X) . (19) Translating these equations leads to a definition of the Hausdorff distance between two fuzzy sets µ and µ as a fuzzy number: δH (µ, µ )(0) = t[ inf T [µ(x), c(µ (x))], inf T [µ (x), c(µ(x))]], x∈S

x∈S

(20)

δH (µ, µ )(n) = t[ inf T [Dνn (µ)(x), c(µ (x))], inf T [Dνn (µ )(x), c(µ(x))], x∈S

x∈S

T (sup t[µ(x), c(Dνn−1 (µ )(x))], sup t[µ (x), c(Dνn−1 (µ)(x))])]. x∈S x∈S

(21)

The obtained distance is positive (the support of this fuzzy number is included in IR+ ). It is symmetrical with respect to µ and µ . The separability property (i.e. d(µ, ν) = 0 ⇔ µ = ν) is not always satisfied. However, we have δH (µ, µ )(0) = 1 implies µ = µ for T being the bounded sum (T (a, b) = min(1, a + b)), while it implies µ and µ crisp and equal for T = max. The triangular inequality is not satisfied in general. 3.4

Directional Relative Position from Conditional Fuzzy Dilation

Relationships between objects can be partly described in terms of relative position, like “to the left of”. Because of the inherent vagueness of such expressions, they may find a better understanding in the framework of fuzzy sets, as fuzzy relationships, even for crisp objects. A few works propose fuzzy approaches for assessing the directional relative position between objects, which is an intrinsically vague relation [2, 15, 16, 19, 20]. The approach used here relies on a fuzzy dilation that provides a map (or fuzzy landscape) where the membership value of each point represents the degree of the satisfaction of the relation to the reference object. This approach has interesting features: it works directly in the image space, without reducing the objects to points or histograms, and it takes the object shape into account. We consider a (possibly fuzzy) object R in the space S, and denote by µα (R) the fuzzy subset of S such that points of areas which satisfy to a high degree the relation “to be in the direction uα with respect to object R” have high membership values, where uα is a vector making an angle α with respect to a reference axis. We express µα (R) as the fuzzy dilation of µR by ν, where ν


159

is a fuzzy structuring element depending on α: µα (R) = Dν (µR ) where µR is the membership function of the reference object R. This definition applies both to crisp and fuzzy objects and behaves well even in case of objects with highly concave shape. In polar coordinates (but this extends to 3D as well), ν is defined by1 : ν(ρ, θ) = f (θ − α) and ν(0, θ) = 1, where θ − α is defined modulo π and f is a decreasing function, e.g. f (β) = max[0, cos β]2 for β ∈ [0, π]. Once we have defined µα (R), we can use it to define the degree to which a given object A is in direction uα with respect to R. Let us denote by µA the membership function of the object A. The evaluation of relative position of A with respect to R is given by a function of µα (R)(x) and µA (x) for all x in S. The histogram of µα (R) conditionally to µA is such a function. A summary of the contained information could be more useful in practice, and an appropriate tool for this is the fuzzy pattern matching approach [12]: the matching between two possibility distributions is summarized by two numbers, a necessity degree N (a pessimistic evaluation) and a possibility degree Π (an optimistic evaluation), as often used in the fuzzy set community. The possibility corresponds to a degree of intersection between the fuzzy sets A and µα (R), while the necessity corresponds to a degree of inclusion of A in µα (R). These operations can also be interpreted in terms of fuzzy mathematical morphology, since Π corresponds to a dilation, while N corresponds to an erosion.

4

Spatial Representations of Spatial Relationships

Now we address a second type of problem, and given a reference object, we define a spatial fuzzy set that represents the area of the space where some relationship to this reference object is satisfied (to some degree). The advantage of these representations is that they map all types of spatial knowledge in the same space, which allows for their fusion and for spatial reasoning (this occurs typically in model-based pattern recognition, where heterogeneous knowledge has to be gathered to guide the recognition). This constitutes a new way to represent spatial knowledge in the spatial domain [3]. For each piece of knowledge, we consider its “natural expression”, i.e. the usual form in which it is given or available, and translate it into a spatial fuzzy set in S having different semantics depending on the type of information (on objects, spatial imprecision, relationships to other objects, etc.). The numerical representation of membership values assumes that we can assign numbers that represent degrees of satisfaction of a relationship for instance. These numbers can be derived from prior knowledge or learned from examples, but usually there remain some quite arbitrary choices. However, we have to keep in mind that mostly the ranking is important, not the individual numerical values. 1

This definition of ν is discontinuous at the origin. A continuous function could be obtained by modeling the fact that the direction of a point or of an object closed to the origin is imprecise.

160

4.1

Isabelle Bloch

Set Relationships

Set relationships specify if areas where other objects can be localized are forbidden or possible. The corresponding region of interest has a binary membership function (1 in authorized portions of the space, 0 elsewhere). This extends to the fuzzy case as: µset (x) = t[µOin (x), 1 − µOout (x)], where t is a t-norm, which expresses a conjunction between inclusion constraint in the objects Oin and exclusion constraint from the objects Oout . The properties of t-norms guarantee that good properties are satisfied. 4.2

Other Topological Relations

Other topological relations (adjacency, etc.) can be treated in a similar way and involve morphological operators. For instance, an object that is a non tangential proper part of µ has to be searched in Eν (µ). 4.3

Distances

Again, morphological expressions of distances, as detailed in Section 3, directly lead to spatial representation of knowledge about distances. Let us assume that we want to determine B, subject to satisfy some distance relationship with an object A. According to the algebraic expressions of distances, dilation of A is an adequate tool for this. For example, if knowledge expresses that d(A, B) ≥ n, then B should be looked for in Dn−1 (A)C . Or, if knowledge expresses that B should lay between a distance n1 and a distance n2 of A, i.e. the minimum distance should be greater than n1 and the maximum distance should be less than n2 , then the possible domain for B is reduced to Dn2 (A) \ Dn1 −1 (A). In cases where imprecision has to be taken into account, fuzzy dilations are used, with the corresponding equivalences with fuzzy distances. The extension to approximate distances calls for fuzzy structuring elements. We define them through their membership function ν on S, with a spherical symmetry, where ν only depends on the distance to the center of the structuring element and corresponds to the knowledge expression, as a fuzzy interval for instance [14]. The increasingness of fuzzy dilation with respect to both the set to be dilated and the structuring element guarantees that the obtained expressions have the required properties. 4.4

Relative Directional Position

The definition of directional position between two sets described in Section 3 relies directly on a spatial representation of the degree of satisfaction of the relation to the reference object. Therefore the first step of the proposed approach directly provides the desired representation as the fuzzy set µα (A) in S.


5

161

Symbolic Representations of Spatial Relationships

In this Section, we use the logical framework presented in Section 2. For spatial reasoning, interpretations can represent spatial entities, like regions of the space. Formulas then represent combinations of such entities, and define regions, objects, etc., which may be not connected. For instance, if a formula ϕ is a symbolic representation of a region X of the space, it can be interpreted for instance as “the object we are looking at is in X”. In an epistemic interpretation, it could represent the belief of an agent that the object is in X. The interest of such representations is also to deal with any kind of spatial entities, without referring to points. If ϕ represents some knowledge or belief about a region X of the space, then ✷ϕ represents a restriction of X. If we are looking at an object in X, then ✷ϕ is a necessary region for this object. Similarly, ✸ϕ represents an extension of X, and a possible region for the object. 5.1

Topological Relationships

Let us first consider topological relationships, and two formulas ϕ and ψ representing two regions X and Y of the space. Note that all what follows holds in both crisp and fuzzy cases. Simple topological relations such as inclusion, exclusion, intersection do not call for more operators than the standard ones of propositional logic. But other relations such that X is a tangential part of Y can benefit from the morphological modal operators. Such a relationship can be expressed as: ϕ → ψ and ✸ϕ ∧ ¬ψ consistent. (22) Indeed, if X is a tangential part of Y , it is included in Y but its dilation is not. If we also want X to be a proper part, we have to add the condition: ¬ϕ ∧ ψ consistent.

(23)

Let us now consider adjacency (or external connection). Saying that X is adjacent to Y means that they do not intersect and as soon as one region is dilated, it intersects the other. In symbolic terms, this relation can be expressed as: ϕ ∧ φ inconsistent and ✸ϕ ∧ ψ consistent and ϕ ∧ ✸ψ consistent.

(24)

It could be interesting to link these types of representations with the ones developed in the community of mereotopology, where such relations are defined respectively from parthood and connection predicates [1, 22]. Interestingly enough, erosion is defined from inclusion (i.e. a parthood relationship) and dilation from intersection (i.e. a connection relationship). Some axioms of these domains could be expressed in terms of dilation. For instance from a parthood postulate P (X, Y ) between two spatial entities X and Y and from dilation, tangential proper part could be defined as: T P P (X, Y ) = P (X, Y ) ∧ ¬P (Y, X) ∧ ¬P (D(X), Y ).

(25)

162

5.2

Isabelle Bloch

Distances

Again we use expressions of minimum and Hausdorff distances in terms of morphological dilations. The translation into a logical formalism is straightforward. Expressions like dmin (X, Y ) ≤ n translate into: ✸n ϕ ∧ ψ consistent and ✸n ψ ∧ ϕ consistent.

(26)

Similarly for Hausdorff distance, we translate dH (X, Y ) = n by: (∀m < n, ψ ∧ ¬✸m ϕ consistent or ϕ ∧ ¬✸m ψ consistent) and (ψ → ✸n ϕ and ϕ → ✸n ψ).

(27)

The first condition corresponds to dH (X, Y ) ≥ n and the second one to dH (X, Y ) ≤ n. Let us consider an example of possible use of these representations for spatial reasoning. If we are looking at an object represented by ψ in an area which is at a distance in [n1 , n2 ] of a region represented by ϕ, this corresponds to a minimum distance greater than n1 and to a Hausdorff distance less than n2 . Then we have to check the following relation: ψ → ¬✸n1 ϕ ∧ ✸n2 ϕ.

(28)

This expresses in a symbolic way an imprecise knowledge about distances represented as an interval. If we consider a fuzzy interval, this extends directly using fuzzy dilation. These expressions show how we can convert distance information, which is usually defined in an analytical way, into algebraic expressions through mathematical morphology, and then into logical ones through morphological expressions of modal operators. 5.3

Directional Relative Position

Here we rely again on the approach where the reference object is dilated with a particular structuring element defined according to the direction of interest. Let us denote by Dd the dilation corresponding to a directional information in the direction d, and by ✸d the associated modal operator. Expressing that an object represented by ψ has to be in direction d with respect to a region represented by ϕ amounts to check the following relation: ψ → ✸d ϕ. In the fuzzy case, this relation can hold to some degree. This formulation directly inherits the properties of directional relative position defined from dilation, such as invariance with respect to geometrical transformations.

6

Conclusion

The spatial arrangement of objects in images provides important information for recognition and interpretation tasks, in particular when the objects are embedded in a complex environment like in medical or remote sensing images. Such


163

information can be expressed in different ways varying from purely quantitative and precise ones to purely qualitative and symbolic ones. We have shown in this paper that mathematical morphology provides a unified and consistent framework to express different types of spatial relationships and to answer different questions about them, with good properties. Due to the strong algebraic structure of this framework, it applies to objects represented as sets, as fuzzy sets, and as logical formulas as well. This establishes links between theories that were so far disconnected. Applications of this work concern model-based pattern recognition, spatial knowledge representation issues, and spatial reasoning. First results have already been obtained using this framework in brain imaging [14] and mobile robotics [9]. Illustrations are also whown in [3, 5].

References [1] N. Asher and L. Vieu. Toward a Geometry of Common Sense: A Semantics and a Complete Axiomatization of Mereotopology. In IJCAI’95, pages 846–852, San Mateo, CA, 1995. 161 [2] I. Bloch. Fuzzy Relative Position between Objects in Image Processing: a Morphological Approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(7):657–664, 1999. 158 [3] I. Bloch. Spatial Representation of Spatial Relationships Knowledge. In A. G. Cohn, F. Giunchiglia, and B. Selman, editors, 7th International Conference on Principles of Knowledge Representation and Reasoning KR 2000, pages 247–258, Breckenridge, CO, 2000. Morgan Kaufmann, San Francisco, CA. 159, 163 [4] I. Bloch. Using Mathematical Morphology Operators as Modal Operators for Spatial Reasoning. In ECAI 2000, Workshop on Spatio-Temporal Reasoning, pages 73–79, Berlin, Germany, 2000. 156 [5] I. Bloch. Mathematical Morphology and Spatial Relationships: Quantitative, Semi-Quantitative and Symbolic Settings. In L. Sztandera and P. Matsakis, editors, Applying Soft Computing in Defining Spatial Relationships, pages 63– 98. Physica Verlag, Springer, 2002. 154, 163 [6] I. Bloch. Modal Logics based on Mathematical Morphology for Spatial Reasoning. Journal of Applied Non Classical Logics, 2002. 156 [7] I. Bloch and J. Lang. Towards Mathematical Morpho-Logics. In 8th International Conference on Information Processing and Management of Uncertainty in Knowledge based Systems IPMU 2000, volume III, pages 1405–1412, Madrid, Spain, 2000. 155 [8] I. Bloch and H. Maˆıtre. Fuzzy Mathematical Morphologies: A Comparative Study. Pattern Recognition, 28(9):1341–1387, 1995. 154 [9] I. Bloch and A. Saffiotti. On the Representation of Fuzzy Spatial Relations in Robot Maps. In IPMU 2002, volume III, pages 1587–1594, Annecy, France, 2002. 163 [10] B. Chellas. Modal Logic, an Introduction. Cambridge University Press, Cambridge, 1980. 155 [11] B. de Baets. Fuzzy Morphology: a Logical Approach. In B. Ayyub and M. Gupta, editors, Uncertainty in Engineering and Sciences: Fuzzy Logic, Statistics and Neural Network Approach, pages 53–67. Kluwer Academic, 1997. 154 [12] D. Dubois, H. Prade, and C. Testemale. Weighted Fuzzy Pattern Matching. Fuzzy Sets and Systems, 28:313–331, 1988. 159

164

Isabelle Bloch

[13] S. Dutta. Approximate Spatial Reasoning: Integrating Qualitative and Quantitative Constraints. International Journal of Approximate Reasoning, 5:307–331, 1991. 156 [14] T. Géraud, I. Bloch, and H. Maˆıtre. Atlas-guided Recognition of Cerebral Structures in MRI using Fusion of Fuzzy Structural Information. In CIMAF’99 Symposium on Artificial Intelligence, pages 99–106, La Havana, Cuba, 1999. 160, 163 [15] J. M. Keller and X. Wang. Comparison of Spatial Relation Definitions in Computer Vision. In ISUMA-NAFIPS’95, pages 679–684, College Park, MD, September 1995. 158 [16] L. T. Koczy. On the Description of Relative Position of Fuzzy Patterns. Pattern Recognition Letters, 8:21–28, 1988. 158 [17] B. J. Kuipers and T. S. Levitt. Navigation and Mapping in Large-Scale Space. AI Magazine, 9(2):25–43, 1988. 156 [18] C. Lafage and J. Lang. Logical Representation of Preferences for Group Decision Making. In A. G. Cohn, F. Giunchiglia, and B. Selman, editors, 7th International Conference on Principles of Knowledge Representation and Reasoning KR 2000, pages 457–468, Breckenridge, CO, 2000. Morgan Kaufmann, San Francisco, CA. 155 [19] P. Matsakis and L. Wendling. A New Way to Represent the Relative Position between Areal Objects. IEEE Trans. on Pattern Analysis and Machine Intelligence, 21(7):634–642, 1999. 158 [20] K. Miyajima and A. Ralescu. Spatial Organization in 2D Segmented Images: Representation and Recognition of Primitive Spatial Relations. Fuzzy Sets and Systems, 65:225–236, 1994. 158 [21] M. Nachtegael and E. E. Kerre. Classical and Fuzzy Approaches towards Mathematical Morphology. In E. E. Kerre and M. Nachtegael, editors, Fuzzy Techniques in Image Processing, Studies in Fuzziness and Soft Computing, chapter 1, pages 3–57. Physica-Verlag, Springer, 2000. 154 [22] D. Randell, Z. Cui, and A. Cohn. A Spatial Logic based on Regions and Connection. In B. Nebel, C. Rich, and W. Swartout, editors, Principles of Knowledge Representation and Reasoning KR’92, pages 165–176, San Mateo, CA, 1992. Kaufmann. 161 [23] J. Serra. Image Analysis and Mathematical Morphology. Academic Press, London, 1982. 154 [24] D. Sinha and E. Dougherty. Fuzzy Mathematical Morphology. Journal of Visual Communication and Image Representation, 3(3):286–302, 1992. 154 [25] R. Zwick, E. Carlstein, and D. V. Budescu. Measures of Similarity Among Fuzzy Concepts: A Comparative Analysis. International Journal of Approximate Reasoning, 1:221–242, 1987. 157

A New Class of Morphological Pyramids for Multiresolution Image Analysis Jos B.T.M. Roerdink Institute for Mathematics and Computing Science University of Groningen, P.O. Box 800, 9700 AV Groningen, The Netherlands [email protected]

Abstract. We study nonlinear multiresolution signal decomposition based on morphological pyramids. Motivated by a problem arising in multiresolution volume visualization, we introduce a new class of morphological pyramids. In this class the pyramidal synthesis operator always has the same form, i.e. a dilation by a structuring element A, preceded by upsampling, while the pyramidal analysis operator is a certain (n) operator RA indexed by an integer n, followed by downsampling. For (n) n = 0, RA equals the erosion εA with structuring element A, whereas (n) for n > 0, RA equals the erosion εA followed by n conditional dilations, which for n → ∞ is the opening by reconstruction. The resulting pair of analysis and synthesis operators is shown to satisfy the pyramid condition for all n. The corresponding pyramids for n = 0 and n = 1 are known as the adjunction pyramid and Sun-Maragos Pyramid, respectively. Experiments are performed to study the approximation quality of the pyramids as a function of the number of iterations n of the conditional dilation operator.

1

Introduction

Multiresolution signal decomposition schemes have enjoyed a long standing interest. Analyzing signals at multiple scales may be used to suppress noise and can lead to more robust detection of signal features, such as transitions in sound data, or edges in images. Multiresolution algorithms also may offer computational advantages, when the analysis of the signal is performed in a coarse-to-fine fashion. Examples of linear multiresolution schemes are the Laplacian pyramid [1] and decomposition methods based on wavelets [5]. This paper is concerned with nonlinear multiresolution signal decomposition based on morphological pyramids. A detailed study of such pyramids was recently made by Goutsias and Heijmans [2, 3]. Morphological pyramids systematically split the input signal into approximation and detail signals by repeatedly applying a pyramidal analysis operator which involves morphological filtering followed by downsampling. As the level of the pyramid is increased, spatial features of increasing size are extracted. The original signal can be recovered from the pyramid decomposition by repeated application of a pyramid synthesis operator. If the analysis and synthesis operators satisfy the so-called pyramid condition, T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 165–175, 2003. c Springer-Verlag Berlin Heidelberg 2003

166

Jos B.T.M. Roerdink

then perfect reconstruction holds, i.e. the original signal can be exactly recovered from the pyramidal decomposition data. The goal of this paper is to derive a class of morphological pyramids in ↓ = δA σ ↓ is fixed to be a dilation which the pyramidal synthesis operator ψA δA by a structuring element A (preceded by an upsampling operator σ ↓ ), but ↑ where the pyramid analysis operator has the form ψA = σ ↑ ηA where σ ↑ denotes downsampling and ηA may be chosen in different ways. Two particular cases of this type of pyramid were mentioned in [2, 3]: (i) the adjunction pyramid, where ηA equals an erosion εA by a structuring element A; (ii) the Sun-Maragos Pyramid, where ηA is an opening αA = δA εA . As we will show below, choosing the operator ηA to be an erosion εA , followed by an arbitrary number of conditional dilations with structuring element A also leads to a valid synthesis ↑ ↓ operator, that is, the pair (ψA , ψA ) satisfies the pyramid condition. Note that this class also contains the opening by reconstruction, which is the connected filter obtained by iterating the conditional dilations until idempotence [9]. The motivation to study this class of pyramids stems from our work on multiresolution volume visualization. Volume visualization or volume rendering is a technique to produce two-dimensional images of three-dimensional data from different viewpoints, using advanced computer graphics techniques such as illumination, shading and colour. Interactive rendering of volume data is a demanding problem due to the large sizes of the signals. For this purpose multiresolution models are developed, which can be used to visualize data incrementally (‘progressive refinement’). In preview mode, when a user is exploring the data from different viewpoints, a coarse representation is used whose data size is smaller than that of the original data, so that rendering is accelerated and thus user interaction is improved. For the case of X-ray volume rendering, which is a linear transform based upon integrating the 3-D data along the line of sight, wavelets have been studied extensively for multiresolution visualization [4, 10]. Another volume rendering method widely used in medical imaging is maximum intensity projection (MIP) where one computes the maximum, instead of the integral, along the line of sight. Since this transform is nonlinear, morphological pyramids are a suitable tool for multiresolution analysis. More in particular, pyramids where the synthesis operator is a dilation are particularly appropriate, because in this case the maxima along the line of sight can be computed on a coarse level (where the size of the data is reduced), before applying a two-dimensional synthesis operator to perform reconstruction of the projection image to full grid resolution. Two cases we have recently investigated for MIP volume rendering are the adjunction pyramid [7, 6] and the Sun-Maragos Pyramid [8]. One of the problems with the adjunction pyramid is that too few small features are retained in higher levels of the pyramid. The basic reason is that the initial erosion of the analysis operator removes fine details. The subsequent downsampling step only aggravates the situation. In the Sun-Maragos Pyramid this situation improved, essentially because erosions are replaced by openings, which keep image features to a larger extent, so that the chance that (parts of) these features survive the downsampling step is larger. From here, it

A New Class of Morphological Pyramids for Multiresolution Image Analysis

167

is only a small step to conjecture that perhaps a number of conditional dilations after the erosion might do even better, because such operators reconstruct more of a certain feature provided some part of it survives the initial erosion. In future work, we intend to apply the new class of pyramids derived here to the MIP volume rendering problem to see whether further improvements can be obtained. The remainder of this paper is organized as follows. Section 2 recalls a few preliminaries on morphological pyramids. In section 3 we derive the new class of morphological pyramids. Some examples are discussed in section 4. Section 5 contains a summary and discussion of future work.

2

Preliminaries

Consider signals in a d-dimensional signal space V0 , which is assumed to be the set of functions on (a subset of) the discrete grid Zd , where d = 2 or d = 3 (image and volume data), that take values in a finite set of nonnegative integers. The general structure of linear as well as nonlinear pyramids is as follows. From an initial signal f0 , approximations {fj } of increasingly reduced size are computed by a decomposition or analysis operator ψ ↑ : fj = ψ ↑ (fj−1 ),

j = 1, 2, . . . L.

Here j is called the level of the decomposition. In the case of a Gaussian pyramid, the analysis operator consists of Gaussian low-pass filtering, followed by downsampling [1]. An approximation error associated to fj+1 may be defined by taking the difference between fj and an expanded version of fj+1 : ˙ ψ ↓ (fj+1 ). dj = fj −

(1)

˙ is a generalized subtraction operator. The set d0 , d1 , . . . , dL−1 , fL is reHere − ferred to as a detail pyramid. Assuming there exists an associated generalized addition operator such that, for all j, ˙ fˆj ) = fj , fˆj (fj −

where fˆj = ψ ↓( ψ ↑ (fj )),

we have perfect reconstruction, that is, f0 can be exactly reconstructed by the recursion (2) fj = ψ ↓ (fj+1 ) dj , j = L − 1, . . . , 0. For the linear case, the detail pyramid is called a Laplacian pyramid, and the synthesis operation consists of upsampling, followed by Gaussian low-pass filtering [1]. In the case of morphological pyramids, the analysis and synthesis operators involve morphological filtering instead of Gaussian filtering [2, 3]. It should be noted that, in principle, the analysis and synthesis operators may depend on level, but we assume them to be the same for all levels j throughout this paper.

168

Jos B.T.M. Roerdink

To guarantee that information lost during analysis can be recovered in the synthesis phase in a non-redundant way, one needs the so-called pyramid condition: (3) ψ ↑ (ψ ↓ (f )) = f for all f. By approximations of f we will mean signals in V0 of the same size as the initial signal f which are reconstructed from higher levels of the pyramid by (0) omitting some of the detail signals. More precisely, a level-j approximation fˆj of f is defined as (0) fˆj = ψ ↓j (fj ), (4) where ψ ↓ j means repeating the ψ ↓ operator j times. ˙ appearing in The generalized addition and subtraction operators and − the definition (1) of the detail signals and the reconstruction equation (2) may be taken as ordinary addition and subtraction, although other choices are sometimes possible as well [2, 7, 6]. 2.1

Adjunction Pyramid

Morphological adjunction pyramids [2] involve the morphological operators of dilation δA (f ) and erosion εA (f ) with structuring element A. In this case the ↑ ↓ and ψA , respectively, and analysis and synthesis operators are denoted by ψA have the form ↑ ψA (f ) = σ ↑ (εA (f )),

(5)

↓ ψA (f )

(6)

↓

= δA (σ (f )),

where the arrows indicate transformations to higher (coarser) or lower (finer) levels of the pyramid. Here σ ↑ and σ ↓ denote downsampling and upsampling by a factor of 2 in each spatial dimension: σ ↑ (f )(n) = f (2n) f (n), ↓ σ (f )(m) = 0,

if m = 2n otherwise

The pyramid condition (3) is satisfied, if there exists an a ∈ A such that the translates of a over an even number of grid steps are never contained in the structuring element A, cf. [2]. Introducing the notation Zd [n] = {k ∈ Zd |k − n ∈ 2Zd } A[n] = A ∩ Zd [n] the pyramid condition can be expressed as A[a] = {a} for some a ∈ A.

(7)


2.2

169

Sun-Maragos Pyramid

This pyramid is defined by the following choice of analysis and synthesis operators: ↑ ψA (f ) = σ ↑ (αA (f )),

(8)

↓ (f ) ψA

(9)

↓

= δA (σ (f )),

where αA = δA εA is the opening by structuring element A. Note that the synthesis operator is identical to that of the adjunction pyramid, cf. (6). Under the condition that A[0] = {0}, (10) where 0 is the origin of Zd , the pyramid condition (3) is satisfied (see [2, Proposition 5.9]), that is, ↑ ↓ ψA ψA = σ ↑ δA εA δA σ ↓ = id, where id denotes the identity operator. Since (εA , δA ) is an adjunction, we have that δA εA δA = δA . Therefore, when A satisfies (10), the previous formula implies that (11) σ ↑ δA σ ↓ = id.

3

A New Class of Morphological Pyramids

In this section we present the main contribution of this paper, which is the derivation of a new class of morphological pyramids containing the adjunction pyramid and Sun-Maragos Pyramid as special cases. We start by recalling the definition of the opening by reconstruction. Let f be a d-dimensional signal (e.g. image or volume data). We define a sequence of (n) operators RA for n = 0, 1, 2, . . . by the following recursion: (0)

RA (f ) = εA (f ) (n)

(12) (n−1)

RA (f ) = f ∧ δA (RA

(f )),

n = 1, 2, . . . .

(13)

The operator in (13) is a conditional dilation, that is, after each dilation step (∞) the infimum with the original signal f is taken. Then RA (f ) is the opening by reconstruction of f from its erosion εA (f ). In practice, f is defined on a finite subset D ⊆ Zd and the recursion terminates after a finite number of steps. We now consider the class of pyramids whose analysis/synthesis operator pairs have the form ↑ (f ) = σ ↑ (RA (f )), ψA

(14)

↓ ψA (f )

(15)

(n)

↓

= δA (σ (f )),

where σ ↑ and σ ↓ denote dyadic downsampling and upsampling as introduced in section 2.1, and the structuring element A satisfies condition (10). By observing

170

Jos B.T.M. Roerdink (0)

(1)

that RA (f ) is the erosion εA of f and RA (f ) is the opening αA = δA εA of f , we see that the cases n = 0 and n = 1 correspond to the adjunction pyramid and Sun-Maragos Pyramid, respectively. Our task is to prove that the pair of operators (14), (15) satisfies the pyramid condition. First, the following lemma is proved. Lemma 1. Consider a morphological pyramid with analysis operator ψ ↑ = σ ↑ η and synthesis operator ψ ↓ = δA σ ↓ , satisfying the following assumptions: 1. η is an anti-extensive operator 2. η δA ≥ id 3. The structuring element A satisfies condition (10). Then the pyramid condition holds. Proof. By assumption 2 we have that ψ ↑ ψ ↓ = σ ↑ η δA σ ↓ ≥ σ ↑ σ ↓ = id. On the other hand, from assumption 1, ψ ↑ ψ ↓ ≤ σ ↑ δA σ ↓ . By assumption 3, formula (11) holds, that is, σ ↑ δA σ ↓ = id. Hence we found that ψ ↑ ψ ↓ ≤ id and that ψ ↑ ψ ↓ ≥ id, so ψ ↑ ψ ↓ = id. Now we return to the problem of showing that the pair (14), (15) satisfies the (n) pyramid condition for each n. It is sufficient to show that the operator η = RA satisfies assumptions 1 and 2 of the lemma, since assumption 3 was assumed to hold anyhow. (0)

1. The operator RA = εA is anti-extensive, because (10) implies that 0 ∈ A, and hence the erosion εA is anti-extensive. For n > 0, equation (13) trivially (n) (n) implies that RA (f ) ≤ f . Hence RA is anti-extensive for all n ≥ 0. (0) 2. We prove by induction that assumption 2 holds. First, RA (δA (f )) = εA (δA (f )) ≥ f since εA δA is a closing. Second, for n > 0 (n) (n−1) (δA (f )) . RA (δA (f )) = δA (f ) ∧ δA RA (n−1)

Applying the induction hypothesis, i.e. RA

δA ≥ id, we find

(n)

RA (δA (f )) ≥ δA (f ) ∧ δA (f ) = δA (f ). Finally, (10) implies that 0 ∈ A, and hence the dilation δA is extensive. (n) Therefore RA (δA (f )) ≥ f , and we are done.

4

Example

In this section, we apply the pyramids discussed in the previous section for image analysis. We computed image decompositions according to a number of analysis/synthesis operator pairs (14), (15) corresponding to various values of n.


171

(j)

Two aspects were considered in the experiments. First, the error Ek of a level-j (0) approximation fˆj as defined in (4): (j) (0) Ek = f − fˆj k / f k ,

(16)

for k = 1, 2, ∞ corresponding to the L1 , L2 and L∞ norms, respectively. Here (0) (0) ↓j fˆj is computed by fˆj = ψA (fj ), cf. (4), where the partial reconstruction fj (0) is computed according to the recursion (2). For a pyramid with L levels, fˆj only takes the highest approximation signal fL , as well as the detail signals dm with m = L − 1, L − 2, . . . , j into account. In all cases, we computed the detail signals and reconstructions by using ordinary addition and subtraction in (1) and (2). Second, we looked at the entropy of the detail signals, which is a measure for the amount of data compression which is achievable. Both measures are essential quality indicators for the case of volume rendering of three-dimensional data which is the motivation for this work (see the introduction). As an example image, we used a two-dimensional view of an angiographic volume data set of a human head, which shows arteries and veins of various sizes. (j) (0) Table 1 shows the error Ek between a level-j approximation image fˆj and the full image f for j = 1, 2 (because of the perfect reconstruction property, the error is exactly zero for j = 0, i.e. complete reconstruction). These errors have been computed for various values of the pyramid parameter n ranging from 0 to 4. Recall that n = 0 and n = 1 correspond to the adjunction pyramid and Sun-Maragos Pyramid, respectively. A structuring element A of size 2 × 2 was used. Figure 1 shows the corresponding approximation images for n = 0, 1, 2. In Figure 2 we show contrast stretched images of the detail signals d0 corresponding to Figure 1. We observe a significant error decrease from n = 0 to n = 1. For n = 2, only the L1 error for the level-2 approximation decreases a little more. For larger n, the L1 and L2 errors start to increase, while the L∞ error stabilizes to a fixed value. This behaviour is reflected in the detail signals dj , cf. Table 2: the L1 norm is largest for n = 0, decreases for n = 1, and stabilizes for larger n. On the other hand, the range of values of d0 , and also the entropy, is smallest for n = 0 (in particular, all elements of the detail signals are non-negative, a general property of adjunction pyramids [2]). We performed a substantial number of experiments for several other images and observed similar behaviour w.r.t. the improvement for n = 1. The approximation quality of the n = 2 pyramid was sometimes slightly better, and sometimes slightly worse, than the n = 1 pyramid, but always better than the n = 0 pyramid.

5

Discussion

We have considered nonlinear multiresolution signal decomposition based on morphological pyramids. Motivated by a problem arising in multiresolution volume visualization, we have introduced a new class of morphological pyramids. In

172

Jos B.T.M. Roerdink

(0)

n = 0: fˆ1

(0)

n = 1: fˆ1

(0)

n = 2: fˆ1

n = 0: fˆ2

n = 1: fˆ2

n = 2: fˆ2

(0)

(0)

(0)

Fig. 1. Input image (top) and approximations from a 2-level pyramid, for n = 0, 1, 2


n=0

n=1

173

n=2

Fig. 2. Detail signals d0 (contrast stretched, white: positive values, light grey: zero values, black: negative values) for the pyramid decomposition of Figure 1

this class the pyramidal synthesis operator always has the same form, i.e. a dilation by a structuring element A, preceded by upsampling, while the pyramidal (n) analysis operator is a certain operator RA indexed by an integer n, followed by (n) downsampling. For n = 0, RA equals the erosion εA with structuring element A, (n) whereas for n > 0, RA equals the erosion εA followed by n conditional dilations, which for n → ∞ is the opening by reconstruction. The corresponding pyramids for n = 0 and n = 1 are known as the adjunction pyramid and Sun-Maragos ↑ ↓ , ψA ) Pyramid, respectively, as discussed in [2, 3]. It was shown that the pair (ψA satisfies the pyramid condition for all n, which implies that the original signal can be exactly recovered from the pyramid.

Table 1. Approximation error as a function of level j for various values of the pyramid parameter n. Cases n = 0 and n = 1 correspond to the adjunction pyramid and Sun-Maragos Pyramid, respectively n 0 1 2 3 4

level j=2 j=1 j=2 j=1 j=2 j=1 j=2 j=1 j=2 j=1

L1 error 0.04406 0.01727 0.02867 0.01420 0.02855 0.01423 0.02856 0.01420 0.02859 0.01419

L2 error 0.09497 0.04385 0.07120 0.03753 0.07150 0.03776 0.07161 0.03777 0.07169 0.03778

L∞ error 0.93701 0.68504 0.84646 0.61024 0.84646 0.61024 0.84646 0.61024 0.84646 0.61024

174

Jos B.T.M. Roerdink

Table 2. Results for the detail signals d0 and d1 for various values of the pyramid parameter n. Shown are the minimum and maximum values over all pixels of the detail images, the L1 norm and the entropy n 0 1 2 3 4

level d0 d1 d0 d1 d0 d1 d0 d1 d0 d1

minimum 0 0 -140 -208 -140 -208 -140 -208 -140 -208

maximum 174 203 155 197 155 197 155 197 155 197

L1 norm 0.01727 0.00670 0.01420 0.00522 0.01423 0.00524 0.01420 0.00525 0.01419 0.00525

entropy 2.164 3.054 2.330 3.017 2.324 2.987 2.316 2.967 2.313 2.964

One of the problems with the adjunction pyramid is that too few small features present in the data are retained in higher levels of the pyramid. Experiments on two-dimensional images were performed indicating that approximation quality improves for n = 1 (the Sun-Maragos Pyramid). The n = 2 pyramid performed sometimes slightly better, and sometimes slightly worse, than the n = 1 pyramid, but always better than the n = 0 pyramid. For n > 2 no further improvement was observed. In future work, we intend to apply the new class of pyramids derived here to the volume rendering problem to see whether improvements can be obtained with respect to the adjunction pyramid and Sun-Maragos Pyramid, as studied in [7, 6, 8].

References [1] Burt, P. J., and Adelson, E. H. The Laplacian pyramid as a compact image code. IEEE Trans. Communications 31 (1983), 532–540. 165, 167 [2] Goutsias, J., and Heijmans, H. J. A. M. Multiresolution signal decomposition schemes. Part 1: Linear and morphological pyramids. IEEE Trans. Image Processing 9, 11 (2000), 1862–1876. 165, 166, 167, 168, 169, 171, 173 [3] Heijmans, H. J. A. M., and Goutsias, J. Multiresolution signal decomposition schemes. Part 2: morphological wavelets. IEEE Trans. Image Processing 9, 11 (2000), 1897–1913. 165, 166, 167, 173 [4] Lippert, L., and Gross, M. H. Fast wavelet based volume rendering by accumulation of transparent texture maps. Computer Graphics Forum 14, 3 (1995), 431–443. 166 [5] Mallat, S. A Wavelet Tour of Signal Processing. Academic Press, New York, 1998. 165


175

[6] Roerdink, J. B. T. M. Multiresolution maximum intensity volume rendering by morphological adjunction pyramids. Tech. Rep. 2001-9-03, Institute for Mathematics and Computing Science, University of Groningen, the Netherlands, July 2001. Under review. 166, 168, 174 [7] Roerdink, J. B. T. M. Multiresolution maximum intensity volume rendering by morphological pyramids. In Data Visualization 2001. Proc. Joint Eurographics – IEEE TCVG Symposium on Visualization, May 28-30, 2001, Ascona, Switzerland, D. Ebert, J. M. Favre, and R. Peikert, Eds. Springer, Wien, New York, 2001, pp. 45–54. 166, 168, 174 [8] Roerdink, J. B. T. M. Comparison of morphological pyramids for multiresolution MIP volume rendering. In Data Visualization 2002. Proc. Eurographics – IEEE TCVG Symposium, May 27-29, 2002, Barcelona, Spain, D. Ebert, P. Brunet, and I. Navazo, Eds. Association for Computing Machinery, New York, 2002, pp. 61–70. 166, 174 [9] Serra, J. Image Analysis and Mathematical Morphology. Academic Press, New York, 1982. 166 [10] Westenberg, M. A., and Roerdink, J. B. T. M. Frequency domain volume rendering by the wavelet X-ray transform. IEEE Trans. Image Processing 9, 7 (2000), 1249–1261. 166

Root Images of Median Filters – Semi-topological Approach Ulrich Eckhardt Fachbereich Mathematik – Optimierung und Approximation – Universit¨ at Hamburg, Bundesstraße 55, D-20146 Hamburg [email protected]

Abstract. Median filters are frequently used in signal analysis because they are robust edge–preserving smoothing filters. Since median filters are nonlinear filters, the tools of linear theory are not applicable to them. One approach to deal with nonlinear filters consists in investigating their root images (fixed elements or signals transparent to the filter). Whereas for one–dimensional median filters the set of all root signals can be completely characterized, this is not true for higher dimensional filters. Tyan (1981) and D¨ ohler (1989) proposed a method for construction of small root images for two–dimensional median filters. Although the Tyan–D¨ ohler construction is valid for a wide class of median filters, their arguments were not correct and their assertions do not hold universally. In this paper we give a rigorous treatment for the construction of Tyan and D¨ ohler. Moreover, the approach is generalized to the d–dimensional case. Keywords: semi–topology, median filter, rank order filter, multi–dimensional filter, root image, fixed point

1

Introduction

A d–dimensional discrete signal (see [15]) is a mapping f : Zd −→ Y where Zd is the d–dimensional discrete space which is the set of all points in Rd having integer coordinates. Usually Y is the set of real numbers or more generally an ordered set. We denote by S the set of all discrete signals. Let B be a finite subset of Zd . The neighborhood of a point x ∈ Zd is the set of all points x + u, u ∈ B. Denote by #B the number of elements of B. For an integer k with 1 ≤ k ≤ #B the k–th rank order filter ρk : S −→ S is defined in the following way: For x ∈ Zd sort (e.g. in ascending order) the values f (x + u), u ∈ B. Then (ρk f )(x) is the kth element in the sorted sequence. For k = 1 we get the minimum filter and for k = #B the maximum filter which #B + 1 yields are relevant in mathematical morphology. If #B is odd, then k = 2 the median filter µ : S −→ S which is the main subject of this paper. We first collect some of the properties of the median filter (see e.g. [20, Chapter 4] for details and precise formulations): T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 176–195, 2003. c Springer-Verlag Berlin Heidelberg 2003

Root Images of Median Filters – Semi-topological Approach

177

– – – –

µ is a nonlinear smoothing filter. The median filter preserves edge sharpness better than the mean filter. The median value is the best estimator with respect to the 1–norm. The most striking property of the median filter is its “robustness”. Specifically, it is insensitive to “outliers” in the data. – The output of a rank order filter has the same data type as the input data. For example, if the input values f (x) are of data type Byte, then the output values (ρk f )(x) have type Byte, too. – Rank order filters can be implemented very efficiently on signal processing hardware since their evaluation needs only very simple arithmetic. – The investigation of convergence properties of iterated median filters becomes interesting in the context of smoothing algorithms for curve evolution. Merriman, Bence and Osher [18] proposed a method for calculating mean curvature flow which was based on iterated median filtering. This approach was mathematically justified by Evans [8] and extended by Guichard and Morel [13]. There exist numerous applications of rank order filters. Specifically in digital image processing they have certain advantages over linear filters. Numerous classes of related filters were developed for different applications in this area [19]. One very important application is compression of digital images. For a more recent publication on this application see e.g. [7]. For such applications, properties of two and higher dimensional filters become important. Whereas for linear filters one has a very powerful theory (Fourier transform, Z–transform, zero– and pole plans etc.), these theoretical tools are not applicable to nonlinear filters. Therefore one tries to model the properties of nonlinear filters according to linear theory. One tool for understanding nonlinear filters are fixed points or “root images” of them. These are signals which are not changed by the filter under investigation. The set of all such signals is the nonlinear analog of the passband of linear filters. In mathematical morphology we can decompose a function into a root image with respect to opening (or closing) and a signal which vanishes under opening (or closing) [4]. Such an analog of “orthogonal decomposition” of a signal is not obviously possible in the rank order filter context. However, we can try to decompose a signal into a suitable root signal and a remainder signal. There is the hope that the root signal carries the “most important” information about the given signal and the remainder signal provides detail information. 1.1

Historical Sketch

The median filter was invented in 1977 by J. W. Tuckey [25]. In 1954 E. N. Gilbert [11] had shown — in a different context — that for rank order filters it is sufficient to know their effect on binary signals. The so called “stacking principle” allows (under certain conditions) some sort of basis decomposition of a signal into binary signals. The importance of Gilbert’s result rests in the fact that binary functions can be interpreted as characteristic

178

Ulrich Eckhardt

functions of sets. Thus rank order filters can be treated in a geometric language as mappings between sets in Zd . In 1981 Neal C. Gallagher, Jr. and Gary L. Wise [9] proved a characterization theorem for root signals of one–dimensional median filters. In 1991 Zi–Jun Gan and Ming Mao provided a complete convergence theory for one–dimensional iterated median filters. This theory states that by iterated application of a median filter one has either convergence in a “weak” sense to a root signal or else the iterates oscillate — under very special conditions — between two limit signals. E. Goles and J. Olivos [12] proved in 1981 that independently of the dimension by iterated application of a median filter to a finite set after a finite number of iterations either a root signal is obtained or else the sequence of iterates becomes periodic with period 2. In 1981 S. G. Tyan [26] published an article on deterministic properties of such filters. In this article he proposed a method for constructing root signals of two–dimensional median filters for binary signals. For a neighborhood (or “window”) which is “centrally symmetric” and contains the zero element, Tyan’s construction of root images or “objects” is as follows: For each window, consider those directed line segments (spokes) which emanate from the center (base) and end at a point (tip) on the boundary. Each of these line segments can also be identified by its angle θ, 0 ≤ θ ≤ 360o with respect to a segment joining the center, say (0, 0) and the point (1, 0). Then one can easily recognize that each object has a boundary which is piecewise linear and is made up of those spokes in the descending or ascending order of θ, each with its base connected to the tip of its predecessor. Then the object is the convex set in Z2 with the above constructed boundary. ... However, it seems that it is true provided that the window A satisfies the following; lim An = Z2. n→∞

Where An+1 = An + A . . . We can consider a window A as degenerate if it fails to satisfy the above condition, . . .

Tyan did not attempt to prove anything, he only gave five examples. Hans– Ulrich Döhler [2] provided in 1989 a first attempt to prove validity of Tyans construction. In his proof he invested two very clever ideas, however, the proof was neither professional nor correct. The main results in D¨ ohler’s paper are: . . . our approach is restricted on a special root signal, which we call the smallest surviving object (SSO) of a median filter. This SSO is a convex digital polygon without holes. ... Proposition 3. The contour of the SSO results from the linkage of all possible straight line segments that connect the center of the window with some other lattice point within the window. This linkage must be done in such a manner that all straight line segments are sorted with respect to their slopes. If there are segments with the same slope, only the longest of them is to be considered.


179

It can be shown by means of counterexamples that the construction of Tyan and D¨ ohler does not always result in a root image. In Figure 1 such a counterexample is given. The neighborhood used in this example is nondegenerate in the sense of Tyan. It is well-known that each two points on a (infinite) checkerboard can be joined by a sequence of knight’s moves. The aim of this paper is to prove some results concerning Tyan–D¨ ohler root images. Moreover, the construction process of these root images is generalized to the d–dimensional case. In [3] it was shown that for “normal” neighborhoods ohler root image is always convex and indeed a root image. It in Rd the Tyan–D¨ is also smallest at least among all convex root images.

a) · · · · · · · ·

c) · · · · · · · ·

Fig. 1.

· · · · · · · · ·

· · · · · · · · · · · ·r · ·r · ❆❑ ✁✕ ·r ·r❍ · ❆·✁ ·✟ ✯ ❍ · ·✟ ·r✟ ❍·❍ · ✙· ✁·❆ ·❥ ·r ·r✟ ✁☛ ❆ · ·r · ·r · · · · · · · · · · ·

· · · · · · · · ·

· · · · · · · · ·

b)

· · · · ·r · · · ·

· · ·r r ·❞ ·r r ·❞ ·r · ·

· · · · ·r · · · ·

· · · · · · · · ·

d)

· · ·r❞ ·r ·r ·r ·r❞ · ·

· ·r ·r ·r ·r ·r ·r ·r ·

· · ·r❞ ·r ·r ·r ·r❞ · ·

· · ·r r ·❞ ·r r ·❞ ·r · ·

· · · · · · · ·

· · · · · · · ·

· · · · · · · · · ·✟ ·r❍ · · · ✟· · ·❍ ·r · · ·r✙ ❑ · · ✁· · · · ·❆ r·☛✁ · · · · · ❆·r ✕· ·❆ · · · · · ✁ ❆ r ✁ r · · ❍· · ·✟ ✯· · ❥ ·r✟· · · · · ·❍ · · · · · · ·

· · · · · · · · ·

· · · · ·r · · · ·

· · · · · · · · ·

· · ·r · · · ·r · ·

· · · ·r ·r ·r · · ·

· ·r · ·r · ·r · ·r ·

· · · ·r ·r ·r · · ·

· · ·r · · · ·r · ·

· · · · ·r · · · ·

The Tyan–Döhler construction for the knight’s neighborhood

a) Knight’s neighborhood. b) Tyan–D¨ ohler construction of the boundary of an “object”. c) The convex hull of the constructed object is not a root image. The encircled points have five neighbors not in the convex hull and thus they do not belong to the median filtered image. d) Root image contained in the Tyan–D¨ ohler set.

180

1.2

Ulrich Eckhardt

Notation

For a subset S ⊆ Zd denote by CS = Zd \ S the set theoretic complement (with respect to Zd ) of S. PZd is the set of all subsets of Zd . For two subsets S1 and S2 the set operation S1 ⊕ S2 := {x1 + x2 | x1 ∈ S1 , x2 ∈ S2 } is called the Minkowski–addition of S1 and S2 . Θd is the zero element of Rd or Zd , respectively. Let B = {u1 , u2 , · · · , un } with ui = uj for i = j be a nonempty finite zero neighborhood in Zd . For x ∈ Zd is B(x) = x + B = {x} ⊕ B the neighborhood of x. B is called a centrally symmetric neighborhood, if u ∈ B implies −u ∈ B. We need a convergence concept for sets: d Definition 1. A sequence {Sr }∞ r=0 of sets in Z converges weakly to a set S if d for any bounded set C ⊆ Z there exists a number r0 such that

Sr ∩ C = S ∩ C

for all

r ≥ r0 .

Notation: Sr −→ S. In a certain special case weak convergence can be guaranteed: Lemma 1. Let {Sr }∞ r=0 be a monotonically decreasing sequence of sets (i. e. Sr+1 ⊆ Sr for all r). Then Sr −→

∞

Sr .

r=0

Let {Sr }∞ r=0 be a monotonically increasing sequence of sets i. e. Sr ⊆ Sr+1 for all r). Then Sr −→

∞

Sr .

r=0

Finally we introduce the concept of semi–topology (see [16, 17]): Definition 2. Given a set X . The system T of subsets of X is a semi–topology for X if 1. ∅ ∈ T and X ∈ T , 2. The union of any system of elements of T belongs to T . The elements of T are termed semi–open sets or open sets for short. A set whose complement is open is called a closed set. Remark 1. There are mainly three approaches for dealing with rank order filters. These three approaches are essentially equivalent: – The theory of complete lattices (Heijmans [14], Ronse [22]), – Mathematical morphology (Serra [23]), – The semi–topological approach (Latecki [16, 17], Eckhardt [4]).


1.3

181

Definitions and Simple Properties

Definition 3. Given a set S ⊆ Zd and a number k ∈ {1, 2, · · · #B}, the k–th d d rank order filter ρk = ρB k is a mapping ρk : PZ −→ PZ which is defined by ρk (S) = x ∈ Zd | #(B(x) ∩ S) ≥ k . The median filter µ = µB , which is the main topic of this paper is a special rank order filter. It is usually defined for neighborhoods with an odd number of –th rank order filter, µ(S) = ρ #B+1 (S). We therefore have elements as the #B+1 2 2

x ∈ µ(S)

⇐⇒

#(B(x) ∩ S) ≥

#B + 1 . 2

Remark 2. If we introduce the characteristic function χS : Zd −→ {0, 1} for a set S by 1 if x ∈ S χS (x) = 0 if x ∈ /S then the definition given here coincides with the definition given above in the introduction. Lemma 2. If Sr −→ S then ρk (Sr ) −→ ρk (S). That means that the mapping ρk is continuous with respect to weak convergence. Examples There are some neighborhoods which are often used for multi– dimensional rank order filtering. Here we restrict ourselves on two–dimensional filters. An often used neighborhood in Z2 is the 4–neighborhood (cross–neighborhood , rook’s neighborhood ) (the origin in Z2 is emphasized): B (4) =

· ·r ·

·r · ·❞r ·r ·r ·

The 8–neighborhood (king’s neighborhood ) is B (8) =

·r ·r ·r

·r ·❞r ·r

·r ·r ·r

Sometimes the 6–neighborhood is used. It is the natural neighborhood of the space         1     1   2 2     Z6 = n1 ·   + n2 ·  √  n1 , n2 ∈ Z .   3     0 2

182

Ulrich Eckhardt

In Z2 this neighborhood is not invariant with respect to the natural motions of Z2 · ·r ·r

B (6) =

·r ·❞r ·r

·r ·r ·

In picture processing also the 12–neighborhood is used [5, 6]: · · ·r · ·

B (12) =

· ·r ·r ·r ·

·r ·r ·❞r ·r ·r

· ·r ·r ·r ·

· · ·r · · (m)

Rank order filters with neighborhood B (m) are denoted ρk , k = 1, 2, · · · , m+ 1, m = 4, 6, 8, 12.

2

Semi-topological Formulation

The proofs of most of the following Lemmas are not difficult and are therefore omitted. Definition 4. For an index k ∈ {1, 2, 3, · · · , #B} the complementary index is k ∗ = #B + 1 − k. The filter ρk∗ is the dual filter to ρk . (see [14, p. 98]). Lemma 3. 1. ρk∗ (S) = Cρk (CS) for all S ⊆ Zd . 2. µ is self–dual, i.e. µ(S) = C(µ(CS)). 3. ρk (k = 1, 2, · · · , #B) is isotone, i. e. for S1 , S2 ∈ Zd , S1 ⊆ S2 is ρk (S1 ) ⊆ ρk (S2 ). 4. ρk+1 (S) ⊆ ρk (S) for each S ⊆ Zd and any k = 1, 2, · · · , #B − 1. 5. For any system {Sα } of sets one has ρk Sα ⊆ ρk (Sα ) α

and α

α

ρk (Sα ) ⊆ ρk

α

Sα

.


183

Lemma 4. Given a system {Sα } of sets with Sα ⊆ ρk (Sα ) for all α. Then also S α ⊆ ρk Sα . α

α

Given a system {Sα } of sets with ρk (Sα ) ⊆ Sα for all α. Then also Sα ⊆ Sα . ρk α

α

The assertions of the Lemma justify the following definition: Definition 5. The semi–topology Tk on Zd is the collection of all sets S with S ⊆ ρk (S). A set S ∈ Tk is termed k–open. A set S ∈ Tk whose complement is k–open is termed a k–closed set. We denote the set of all k–open subsets of Zd by Ok , the set of all k–closed subsets of Zd by Fk . The set of all root images of ρk is the set Ok ∩ Fk . Remark 3. In lattice theory the set Ok is called the domain of extensivity of ρk (Ext(ρk )), and Fk is the domain of antiextensivity of ρk (Antext(ρk )) and Ok ∩ Fk is the invariance domain of ρk (Inv(ρk )) (see [22, p. 23] or [14, Chapter 6.1]). We state two rather trivial properties of these topologies in the next Lemma: Lemma 5. 1. The closed sets in the topology Tk are exactly the sets S with ρk∗ (S) ⊆ S. 2. Each (k + 1)–open set is k–open. That means, the topology Tk is finer than the topology Tk+1 or the topology Tk+1 is coarser than the topology Tk . Definition 6. Let S ⊆ Zd be a set. The (k–) closure clk S of S is the smallest k– closed set containing S. The (k–) interior intk S is the largest k–open set contained in S. It is a direct consequence of the definition that these two concepts are well– defined. Example 1. Let B be a zero neighborhood in Zd , Θd ∈ B. We investigate some special rank order filters ρk . For k = 1 is ρ1 (S) = {x | #(B(x) ∩ S) ≥ 1}, specifically S ⊆ ρ1 (S), that means that each set is 1–open. Hence each set is also 1–closed. In the terminology of mathematical morphology (for centrally symmetric B) the set ρ1 (S) is termed the dilation of S.

184

Ulrich Eckhardt

For k = 2 is ρ2 (S) = {x | #(B(x) ∩ S) ≥ 2}. A point x ∈ S is called a B– isolated point if B(x) ∩ S = {x}. A set is 2–open if and only if it does not contain any B–isolated points. A point is termed a 1–hole if it is an isolated point of the complement of S. 1–holes play a special role in thinning of binary images [5, 6]. A set is 2–closed if and only if it contains no 1–holes. int2 S is the set obtained from S by removing all isolated points from it. cl2 S is the set obtained from S by filling all its 1–holes. One has cl2 int2 S = int2 cl2 S for any set S. Lemma 6. If S is open then the same is true for ρk (S). If S is closed then the same is true for ρk∗ (S). In other words: ρk maps the set Ok into itself and ρk∗ maps the set Fk into itself. Lemma 7. It is always intk S ⊆ S ∩ ρk (S) ⊆ S ∪ ρk∗ (S) ⊆ clk (S). We now define recursively a sequence of sets: For a given set S we put S (0) = S and S (r+1) = S (r) ∩ ρk S (r) for r = 0, 1, 2, · · ·. In Heijmans’ book the limit of the sequence {S (r) } in the case of the median filter is called the median opening [14, Example 13.36]. Lemma 8. The sequence {S (r) } has the properties 1. S (r+1) ⊆ S for all r. 2. S (r+1) ⊆ S (r) and S (r+1) ⊆ ρk (S (r) ). r ρk S (i) . 3. S (r+1) ⊆ i=0

4. intk S ⊆ S (r) for r = 0, 1, 2, · · ·. Lemma 9. For a set S ⊆ Zd let S (r) be defined iteratively as above. Then S (r) −→ intk S in the sense of weak convergence (Lemma 1).


185

Proof. From monotonicity of the sequence S (r) we get by Lemma 1 weak ∞ convergence to S := r=0 S (r) . This means that for any bounded set C ⊆ Zd there is a number r0 such that S (r) ∩ C = S ∩ C for all r ≥ r0 . In particular, S (r+1) ∩ C = S (r) ∩ C for all r ≥ r0 . Consequently, for all such r S (r+1) ∩ C = S (r) ∩ ρk (S (r) ) ∩ C = S (r) ∩ C This implies S (r) ∩ C ⊆ ρk (S (r) ) ∩ C for all r ≥ r0 . ∞ Now, S = ν=0 S (ν) ⊆ S (r) for all r, hence S ∩ C ⊆ S (r) ∩ C ⊆ ρk S (r) ∩ C. Therefore, for r ≥ r0

S ∩ C = S (r) ∩ C ⊆ ρk S (r) ∩ C.

By Lemma 2 for all sufficiently large r ρk S (r) ∩ C = ρk S ∩ C. This implies S ∩ C ⊆ ρk S ∩ C. Thus it is proved that the set S is k–open. Furthermore, (Lemma 8, properties Since intk S is the largest open set ⊆ S, we have 1 and 4) S ⊆ S and intk S ⊆ S. intk S = S. Dualization yields the following assertions:

For a given set S we put S S

(r+1)

(0)

= S and (r) (r) = S ∪ ρk S

for r = 0, 1, 2, · · ·.

Lemma 10. The sequence {S

(r)

} has the properties

(r)

1. S ⊆ S for all r. (r) (r+1) (r) (r+1) and ρk (S ) ⊆ (S ). 2. S ⊆ S r (i) (r+1) ⊆S 3. ρk S . i=0 (r)

4. S

⊆ clk S for r = 0, 1, 2, · · ·.

186

Ulrich Eckhardt

Lemma 11. For a set S ⊆ Zd let S (r) S −→ clk S.

(r)

be defined iteratively as above. Then

Remark 4. Both iteration processes provide by Lemma 9 (or Lemma 11, respectively) a constructive method for determining intk S (or clk S, respectively). Remark 5. In 4–topology x ∈ µ(S) = ρ3 (S) and x ∈ S means that x has at least two direct neighbors in S. x ∈ / int3 S means for x ∈ S, that x has at most one direct neighbor in S. Consequently, in 4–topology always int3 S = S ∩ µ(S) and S ∪ µ(S) = cl3 S. Given any set S ⊆ Zd , we can easily assign to this set two root images of the median filter which are in some sense “close” to this set. Lemma 12. Assume that Zd is equipped with the semi–topology associated to the median filter (i.e. k = #B+1 ). 2 1. If S is (k–) open then cl S is a root image. 2. If S is (k–) closed then int S is a root image. 3. For any set S ⊆ Zd the sets clk intk S and intk clk S are root images. (0)

Proof. Assume that S is k–open. Then by Lemma 10 the iteration process S = (r+1) (r) (r) S, S = S ∪ µ(S ) yields a sequence converging to clk S. Since S is open, S

(r)

⊆ µ(S

(r)

), hence S

(r+1)

= µ(S

(r)

).

Let C be a bounded subset of Zd , C = C ⊕ B. Since the sequence {S converges monotonically to cl S, there exists a number r0 such that S

(r)

∩ C = cl S ∩ C

(r)

}

for all r ≥ r0 .

If x ∈ C then B(x) ⊆ C by construction of C . Since in C the sets S cl S coincide, we have (r) ∩ C = µ (cl S) ∩ C for all r ≥ r0 . µ S

(r)

and

(r)

Since this holds for all bounded C, limr→∞ S = cl S. The second assertion holds by duality, the third one is a consquence of the first two assertions. In general topology so–called regular sets play a special role. These are sets having the property that S = int cl S or S = cl int S. This concept is due to Stone [24] and it has many applications (see [21] and [14, Example 7.9]). For an application in our context, see [1]. The following Corollary is an immediate consequence of the last Lemma: Corollary 1. For the topology associated to the median filter (i.e. k = the following equivalence holds S = intk clk S or ⇐⇒ S is a root image of µ. S = clk intk S

#B+1 ) 2


187

In the following Lemmas some properties of the interior and the closure operators are collected. Lemma 13. Always is int int S = int S. Lemma 14. For each set S ⊆ Zd holds int (S \ int S) = ∅ and int (cl S \ S) = ∅. Proof. Let S0 = S \ int S and Sr+1 = Sr ∩ µ(Sr ) for r = 0, 1, 2, · · · furthermore S0 = S and Sr+1 = Sr ∩ µ(Sr ) for r = 0, 1, 2, · · ·. Then S1 = S0 ∩ µ(S0 ) = S ∩ C int S ∩ µ (S ∩ C int S) ⊆ ⊆ S ∩ C int S ∩ µ(S) ∩ µ (C int S) = S ∩ C int S ∩ µ(S) ∩ C Cµ(C int S) = =µ

= S ∩ C int S ∩ µ(S) ∩ Cµ(int S) ⊆ S ∩ µ(S) ∩ C int S = S ∩ µ(S) \ int S, ⊆CintS

hence S1 ⊆ S1 \ int S. If Sr ⊆ Sr \ int S then also Sr+1 = Sr ∩ µ(Sr ) ⊆ Sr \ int S ∩ µ Sr \ int S ⊆ ⊆ Sr ∩ C int S ∩ µ Sr ∩ µ (C int S) ⊆ Sr+1 ∩ C int S. From Sr −→ int S we conclude Sr −→ ∅. Analogously the proof of the second assertion is carried out. Corollary 2. For any set S the following identities are true 1. 2. 3. 4.

int (S \ cl int S) = ∅, int (S \ int cl S) = ∅, int (cl int S \ S) = ∅, int (int cl S \ S) = ∅.

Remark 6. From Lemma 12 we know that clk intk S and intk clk S (k = #B+1 ) 2 are root images. Corollary 1 even states that the regular sets are exactly all root images. These root images are in some sense “close” to the set S which means that the difference set has empty interior. So we can state that Lemma 2 provides a decomposition of a set into a root image and a “thin” set which contains “less important information”. This corresponds to the orthogonal decomposition of a signal into a main signal carrying “essential” information and a detail signal in linear theory. Lemma 15. We have cl int cl S ⊆ cl S

and

int S ⊆ int cl int S

and cl int S = cl int cl int S

and

int cl S = int cl int cl S.

188

3

Ulrich Eckhardt

Existence of Finite k–Open Sets

The empty set and the whole space Zd are k–open for all k, 1 ≤ k ≤ #B. For k = 1 all sets are k–open (if Θd ∈ B). It is also possible to find “large” k–open sets since the set Ok is stable with respect to intersections. However, “small” finite nontrivial k–open sets are of importance. In favourable cases a semi–topology can be described by means of its smallest open sets and in very lucky cases there exist finite point bases of a semi–topology. In the case of the topologies Tk the situation is certainly more complicated as is illustrated by the following example. Example 2. Consider the cross median filter which is based on the neighborhood B (4) . In the following picture the left set is without any doubt a “smallest” open set and even a root image in the median–filter topology T3 . It is indeed the Tyan–D¨ ohler set for the cross median filter. If any point of this set is removed from it then the remaining set is no longer a 4–open set. The right set is also a “smallest” set in the same sense, however, the left set is not contained in the right one. Moreover, an infinite number of sets can be constructed in a similar manner as the right set and none of them is contained in the other. That means, there is no finite base for the topology T3 . ·

·

·

·

·

·

r ·r ·r ·r

·

·

·

·

·

·

·

·

·

·

·

·

·

·

r ·r ·r ·r ·r · · ·r ·r · ·r ·r ·r ·r ·r ·

·

·

·

·

·

·

·

·

· · · · · ·

We start with a negative assertion. Lemma 16. 1. If Θd ∈ B then there are no finite nonempty k–open sets for k > #B+1 . 2 / B then there are no finite nonempty k–open sets for k > #B 2. If Θd ∈ 2 . Proof. Given a bounded digital set S. Since B is finite, there exists a vector x∗ ∈ Rd such that x∗ , uj = 0 for all uj ∈ B, uj = Θd (·, · is the scalar product in Rd ). Let B + = {u ∈ B | x∗ , u > 0} and B − = {u ∈ B | x∗ , u < 0} and (without loss of generality) #B − ≤ #B + . Choose x0 ∈ S so that x∗ , x0 ≥ x∗ , x for all x ∈ S. Such an x0 exists by finiteness of S. Then (by B + (x0 ) ∩ S = ∅)    #(B − (x0 ) ∩ S) + 1 ≤ #B − + 1 ≤ #B + 1 if Θd ∈ B, 2 #(B(x0 ) ∩ S) = #B  − −  if Θd ∈ / B, #(B (x0 ) ∩ S) ≤ #B ≤ 2 hence in both cases x0 ∈ / ρ(k) (S), therefore S is not open. Corollary 3. 1. If Θd ∈ B then there are no finite nonempty (k–) root images for k = #B+1 . 2 2. If Θd ∈ / B then there are no finite nonempty (k–) root images at all.


189

Proof. Let S be a k–root image, i. e. ρ(k) (S) = S. Furthermore, ρ(k) (CS) = CS ∗ implies ρ(k ) (S) = S. Therefore is S k ∗ –open. 1. If Θd ∈ B then S k–open implies k ≤ #B+1 and S k ∗ –open implies k ∗ = 2 #B+1 #B+1 #B + 1 − k ≤ 2 + 1 or k ≥ 2 . Hence, root images are only possible for k = #B+1 , which implies that #B is odd. 2 ∗ ∗ / B then S k–open implies k ≤ #B 2. If Θd ∈ 2 . S k –open implies k = #B #B #B + 1 − k ≤ 2 or k ≥ 2 + 1. Hence, there is no such k. Lemma 17. If B is not centrally symmetric then there are no (nontrivial) root images at all. / B there are no nontrivial root images Proof. The Corollary implies that for Θd ∈ anyway. Therefore we can assume Θd ∈ B. B is centrally symmetric if and only if for each x∗ = Θd #{u ∈ B | x∗ , u > 0} = #{u ∈ B | x∗ , u < 0}. Therefore, if B is not centrally symmetric, there exists an x∗ = Θd such that x∗ , u = 0 for all u ∈ B \ {Θd } and, if B + = {u ∈ B | x∗ , u > 0} and B − = {u ∈ B | x∗ , u < 0} then #B − > #B + (without loss of generality). We choose x0 ∈ S such that x∗ , x0 ≥ x∗ , x for all x ∈ S. Then x∗ , x0 + u > x∗ , x for all u ∈ B + , hence x0 +u ∈ / S for all u ∈ B + . We obtain #(B(x0 )∩S) = − − #(B ∩ S) ≤ #B + 1. Since #B = #B − + #B + + 1 > 2 · #B − + 1, we get #B+1 1 #B − < #B , hence S is not a root 2 − 2 , consequently #(B(x0 ) ∩ S) < 2 image. Definition 7. A cycle is a sequence {ui1 , ui2 , · · · , ui } of elements in B such that there are positive integers αj , j = 1, 2, · · · , with

!

αj uij = Θd .

(1)

j=1

Lemma 18. The sequence {ui1 , ui2 , · · · , ui } is a cycle if and only if there are real numbers λ1 , λ2 , · · · , λ such that

! j=1

λj uij = Θd ,

λj > 0

for j = 1, 2, · · · , ,

!

λj = 1.

j=1

Proof. 1. If there are no such numbers λj then there is no nontrivial solution of (1) in nonnegative real numbers and thus also no nontrivial solution in nonnegative integers. 2. Assume that there are real numbers λ1 , λ2 , · · · , λ with λj > 0 for j = " " 1, 2, · · · , and j=1 λj = 1 and j=1 λj uij = Θd . It is possible (for example by Gaussian elimination) to eliminate dependent variables. Approximating the independent variables suitably by rational numbers, the dependent variables remain positive and are rational. Multiplying by a suitable integer one gets an integer solution.

190

Ulrich Eckhardt

The cycles which can be formed by the elements of B are of course not uniquely determined. Lemma 19. If there are k − 1 cycles having no elements uj ∈ B in common, then there exists a bounded nontrivial k–open set. Proof. Assume that the equation (1) has a nontrivial solution. Let x0 ∈ Zd . We construct a chain v0 , v1 , v2 , · · ·, of vectors with v0 = x0 and vi+1 = vi + uij for all j. We require that each uij occurs exactly ναj times in the construction. By (1) the chain of the vi is closed. By choosing S = {vi } we get a 2–open set. Assume that there is a second cycle. We generate from the set S a second set by shifting the set S along the second cycle (in morphological terminology: the Minkowski addition of S with the second cycle is performed). The resulting set has the property that both cycles pass through each of its points, hence it is 3–open. Continuing the process with all cycles, we get a k–open set. For 2–open sets we can prove the converse of the Lemma: Lemma 20. There exists a 2–open set if and only if there exists a cycle. Proof. Assume that the Diophantine equation (1) has only the trivial nonnegative integer solution. Then Θd ∈ / conv{ui1 , ui2 , · · · , ui }. By the Separation The= Θd , orem for Convex Sets [27, Chapters II, V] there exists a vector x∗ ∈ Rd , x∗ $ # ∗ such that uij , x# > 0 for all$ j. Choose x0 ∈ S such that x0 , x∗ ≥ x, x∗ for /S all x ∈ S. Then x0 + uij , x∗ > x, x∗ for all x ∈ S, consequently x0 + uij ∈ for all j. Remark 7. For a centrally symmetric neighborhood B (containing Θd ) there are cycles {ui , −ui }. Therefore for such a neighborhood (and only for always #B−1 2 –open finite sets. such a neighborhood) there exist always nontrivial #B+1 2 Example 3. We consider in Z2 the neighborhood B = {Θ2 , u1 , u2 , u3 } with u1 = (1, 0), u2 = (−1, 1), u2 = (−1, −1). By construction of Lemma 19 we get the 2–open set in the right image.

B=

· · · · ·

· ·r · ·r ·

· · ·❞r · ·

· · ·r · ·

· · · · ·

−→

· · · ·

· · ·r ·

· ·r ·r ·

· · ·r ·

· · · ·


4

191

Construction of k–Open Sets, Centrally Symmetric Case

We now assume that B is centrally symmetric and Θd ∈ B. Definition 8. A digital set S is termed a B–path if for each element x ∈ S 2 ≤ #(B(x) ∩ S) ≤ 3 and if for at most two elements (the end points) #(B(x)∩ S) = 2. A bounded B– path with #(B(x) ∩ S) = 3 for all x is a closed B–path. Remark 8. For centrally symmetric B (having more than three elements) each B–path is 2–open. Each B–path without end points (i.e. a closed B–path or an infinite B–path) is 3–open. Lemma 21. Given any set S. Then the set δu (S) := S ⊕ {Θd , u} = S ∪ (S + u) is 2–open for each u ∈ B \ {Θd } Proof. x ∈ δu (S) means either x ∈ S and then x + u ∈ δu (S), or else x ∈ / S, then there exists an x ∈ S such that x + u ∈ S. Since B is centrally symmetric, x − u ∈ δu (S) and −u ∈ B. Hence δu (S) is 2–open. Let δ(S) = S ⊕ B Lemma 22. Given a k–open set S and any neighborhood B. be the (usual) dilation of S with the structuring element B. Then δ(S) is also a k–open set. Proof. Let x ∈ δ(S). If x in S, then clearly #(B(X)∩S) ≥ k. If x ∈ / S, then there exist an x ∈ S and a u ∈ B such that x = x + u. Since S is k–open, there are k \ {Θd } such that x + uij ∈ S pairwise different elements ui1 , ui2 , · · · , uik of B for all j. Consequently, x + uij ∈ δ(S) for all j, hence δ(S) is also k–open. From these observations we get a very simple method for construction of a k–open set (in case of a centrally symmetric neighborhood containing Θd ) for k ≤ #B+1 . Assume that a digital set S is given and elements ui1 , ui2 , · · · , uik of 2 B \ {Θd }, such that uij = ui and uij = −ui for j, = 1, 2, · · · k.

Algorithm Ak (S; ui1 , ui2 , · · · , uik ) Start Choose any subset ui1 , ui2 , · · · , uik of B \ {Θd }, such that uij = ui and uij = −ui for j, = 1, 2, · · · k. Let S0 := S. Iteration Let Sj = δuij (Sj−1 ) for j = 1, 2, · · · , k. Result Ak (S; ui1 , ui2 , · · · , uik ) := Sk

192

Ulrich Eckhardt

Lemma 23. Given ui1 , ui2 , · · · , uik in B \ {Θd }, such that uij = ui and uij = −ui for j, = 1, 2, · · · k. Then Ak (S; ui1 , ui2 , · · · , uik ) is a k + 1–open set. 4.1

Construction of Tyan–D¨ ohler Sets

Let B = {Θd , u1 , u2 , · · · , un } be a centrally symmetric neighborhood (ui = Θd for all i = 1, 2, · · · , n and ui = uj for i = j). Let furthermore {v1 , v2 , · · · , vm } be a complete system of generators of B, which means that m = n2 , and for each i = 1, 2, · · · , n there is a j ∈ {1, 2, · · · , m}, such that ui = vj or ui = −vj . Conversely, for each j ∈ {1, 2, · · · , m} there is an i ∈ {1, 2, · · · , n}, such that vj = ui . Definition 9. The Tyan–D¨ ohler set belonging to the complete system of generators {v1 , v2 , · · ·, vm } is the set B0 = Am ({x0 }; v1 , v2 , · · · , vm ) for a given point x0 ∈ Zd . Lemma 24. Let {i1 , i2 , · · · , im } be a permutation of the indices {1, 2, · · · , m}. The set of all paths x0 , x0 + vi1 , x0 + vi1 + vi2 , · · · x0 + vi1 + vi2 + · · · + vim

(2)

covers B0 completely. Proof. 1. Without loss of generality let x0 = Θd . Choose x1 ∈ B0 . By construction there are directions vi1 , vi2 , · · · , vi in B ( ≤ m, vij pairwise different), such that vi1 + vi2 + · · · + vi = x1 . We can complete the set {vij } to get a complete system of generators which shows that each x1 ∈ B0 is on a path as given above. This implies that B0 is contained in the union of all these paths. 2. Assume that x1 is contained in a path of the form (2), say x1 = vi1 + vi2 + · · · + vi . Since the composition of dilations is commutative, we can permute the directions vij such that the indices are sorted in ascending order. This implies that x1 wil be met by one of the dilations generating B0 , hence x1 ∈ B0 . Lemma 25. Let {v1 , v2 , · · · , vm } be a complete system of generators of B and let B0 = Am ({x0 }; v1 , v2 , · · · , vm ). Given x1 ∈ B0 , there exists a complete system {w1 , w2 , · · · , wm } of generators of B such that B0 = Am ({x1 }; w1 , w2 , · · · , wm ). Proof. Assume that x1 = x0 +vi1 . Then we choose the complete system of generators −vi1 , vi2 , · · ·, vim . The path x0 +vi1 , x0 +vi1 +vi2 , · · · , x0 +vi1 +vi2 · · · , +vim is a subpath of x0 , x0 + vi1 , x0 + vi1 + vi2 , · · · , x0 + vi1 + vi2 · · · , +vim and thus it is completely contained in B0 . We add the point x0 + vi1 + vi2 · · · , +vim − vi1 . Also this point is in B0 (Sorting yields the point x0 + vi2 + vi3 · · · , +vim , and this point is by Lemma 24 in B0 ). Assume that x2 is a point of the permuted sequence. If for the construction of x2 the direction −vi1 was not used, everything is clear.


193

Otherwise, if the direction vi1 was used for construction of x2 , then the remaining directions lead from x0 to x2 ∈ B0 . In the same way one argues for other starting points x1 ∈ B0 . ohler set whenever the Remark 9. In Z2 , the construction yields the Tyan–D¨ latter is convex. This can be easily seen by choosing generators which are ordered by their slopes.

5

B–Convexity

Throughout this section we assume that B is centrally symmetric and Θd ∈ B. Definition 10. A digital set S is B–convex if the following inclusion is valid for each u ∈ B x + tu ∈ S for any t > 0

and

x − su ∈ S for any s < 0

=⇒

x ∈ S.

Remark 10. Instead of B–convexity it is sufficient to require local B–convexity: For each u ∈ B x±u∈S

=⇒

x ∈ S.

Obviously, each digitally convex set (i.e. S = Zd ∩ conv S, conv denoting the usual convex hull in Rd ) is B–convex for any centrally symmetric B containing Θd . Lemma 26. Any B–convex set is

#B+1 –closed. 2

Proof. If x ∈ / S then, by B–convexity, x ± u for u ∈ B cannot both belong to S. Hence at least one half of the elements x + u, u ∈ B, belong not to S. Therefore, x∈ / µ(S). Remark 11. If the Tyan–D¨ ohler set B0 is not convex, then we know that between B0 and conv B0 there is a root image. This can be found by iteratively applying µ to both sets. Since B0 is open and conv B0 is closed, both iterates converge towards a root image. Remark 12. D¨ ohler [2] was the first one who emphasized the role of convexity in the construction of Tyan–D¨ ohler sets.

6

Conclusions

It was shown that the Tyan–D¨ ohler construction for “small” root images of median filters can be rigorously defined and generalized to the nonconvex and higher–dimensionbal cases. This was achieved by reformulating the problem in a semi–topological context. As a by–product insights were obtained about the decomposition of a set into a related root image and a “thin” reimaining set (Lemma 14). There remain, however, open problems. We list some of them:

194

Ulrich Eckhardt

– Under what conditions will the process in Section 4.1 indeed yield a root image? A partial answer was given in [3]. – Are the root images obtained by the Tyan–D¨ ohler construction indeed “smallest” as claimed by Döhler [2]? In the convex case this can be proved. – How does a basis of the semi–topology T #B+1 look like? For example, in the 2 topology corresponding to the cross median filter, do the closed 4–curves (as sketched in Example 2) constitute a basis of T3 ? – Sets which can be represented as unions of Tyan–Döhler sets are open. The question is, whether they have other distinctive properties among open sets. The semi–topology generated by all unions of translates of B0 has character 1. This implies that it has many attractive properties [17, 4] which make such sets attractive for investigation.

References [1] Alpers, A.: Digital topology: Regular sets and root images of the cross-median filter. J. Math. Imaging and Vision, to appear 186 [2] D¨ ohler, H.-U.: Generation of root signals of two dimensional median filters. Signal Processing 18 (1989) 269–276 178, 193, 194 [3] Eckhardt, U.: Root Images of Median Filters. J. Math. Imaging and Vision, to appear 179, 194 [4] Eckhardt, U., Hundt, E.: Topological approach to mathematical morphology. In: Solina, F., Kropatsch, W. G., Klette, R., Bajcsy, R. (eds.): Advances in Computer Vision. Springer-Verlag, Wien New York (1997) 11–20 177, 180, 194 [5] Eckhardt, U., Latecki, L., Maderlechner, G.: Irreducible and thin binary sets. In: Arcelli, C., Cordella, L. P., Sanniti di Baja, G. (eds.): Aspects of Visual Form Processing. 2nd International Workshop on Visual Form, Capri, Italy, May 30 – June 2, 1994. World Scientific Publishing Co. Pte. Ltd., Singapore New Jersey London Hong Kong (1994) 199–208 182, 184 [6] Eckhardt, U., Maderlechner, G.: Invariant thinning. Int. J. Pattern Recognition and Artificial Intelligence 7 (1993) 1115–1144 182, 184 [7] Estrakh, D. D., Mitchell, H. B., Schaefer, P. A., Mann, Y., Peretz, Y.: “Soft” median adaptive predictor for lossless picture compression. Signal Processing 81 (2001) 1985–1989 177 [8] Evans, L. C.: Convergence of an algorithm for mean curvature motion. Indiana Univ. Math. J. 42 (1993) 533–557 177 [9] Gallagher, N. C., Wise, G. L.: A theoretical analysis of the properties of median filters IEEE Trans. ASSP-29 (1981) 1136–1141 178 [10] Gan, Z. J., Mao, M.: Two convergence theorems on deterministic properties of median filters. IEEE Trans. Signal Processing 39 (1991) 1689–1691 [11] Gilbert, E. N.: Lattice theoretic properties of frontal switching functions. J. Math. Phys. 33 (1954) 57–67 177 [12] Goles, E., Olivos, J.: Comportement periodique des fonctions ` a seuil binaires et applications. Discr. Appl. Math. 3 (1981) 93–105 178 [13] Guichard, F., Morel, J.-M.: Partial differential equations and image iterative filtering. In: Duff, I. S., Watson, G. A. (eds.): The State of the Art in Numerical Analysis, Based on the proceedings of a conference organized by the Institute of Mathematics and its Applications (IMA), University of York, York, GB, April 1–4, 1996. Inst. Math. Appl. Conf. Ser. New Ser. v. 63. Clarendon Press, Oxford (1997) 525–562 177


195

[14] Heijmans, H. J. A. M.: Morphological Image Operators. Advances in Electronics and Electron Physics. Academic Press, Inc., Harcourt, Brace & Company, Publishers, Boston San Diego New York London Sydney Tokyo Toronto (1994) 180, 182, 183, 184, 186 [15] Krabs, W.: Mathematical Foundations of Signal Theory. Sigma Series in Applied Mathematics, Vol. 6. Heldermann Verlag, Berlin (1995) 176 [16] Latecki, L. J.: Discrete Representation of Spatial Objects in Computer Vision. Computational Imaging and Vision. Kluwer Academic Publishers, Dordrecht Boston London (1998) 180 [17] Latecki, L.: Digitale und Allgemeine Topologie in der bildhaften Wissensrepr¨ asentation. DISKI Dissertationen zur K¨ unstlichen Intelligenz 9. infix, St. Augustin (1992) 180, 194 [18] Merriman, B., Bence, J., Osher, S.: Diffusion generated motions by mean curvature. UCLA Computational and Applied Mathematics Reports (1992) 92–18 177 [19] Pitas, I., Venetsanopoulos, A. N.: Order statistics in digital image processing. Proc. IEEE 80 (1992) 1893–1921 177 [20] Pitas, I., Venetsanopoulos, A. N.: Nonlinear Digital Filters: Principles and Applications. The Kluwer International Series in Engineering and Computer Science, 84. VLSI, Computer Archtecture and Digital Signal Processing. Kluwer Academic Publishers Group, Boston Dordrecht London (1990) 176 [21] Rinow, W.: Lehrbuch der Topologie. Hochschulb¨ ucher f¨ ur Mathematik, Bd. 79. Deutscher Verlag der Wissenschaften, Berlin (1975) 186 [22] Ronse, C.: Lattice-theoretical fixpoint theorems in morphological image filtering. J. Math. Imaging and Vision 4 (1994) 19–41 180, 183 [23] Serra J., (ed.): Image Analysis and Mathematical Morphology, Volume 2: Theoretical Advances. Academic Press, Harcourt Brace Jovanovich Publishers, London San Diego New York Boston Sydney Tokyo Toronto 1988) 180 [24] Stone, M. H.: Applications of the theory of Boolean rings to general topology. Trans. Amer. Math. Soc. 41 (1937) 375–481 186 [25] Tuckey, J. W.: Exploratory Data Analysis. Addison–Wesley, Reading. Mass. (1977) 177 [26] Tyan, S. G.: Median filtering: Deterministic properties. In: Huang,T. S. (ed.): Two-Dimensional Digital Signal Processing II. Transforms and Median Filters. Topics in Applied Physics, Vol. 43. Springer–Verlag, Berlin Heidelberg New York (1981) 197–217 178 [27] Valentine, F. A.: Convex Sets. McGraw–Hill Series in Higher Mathematics. McGraw–Hill Book Company, New York San Francisco Toronto London (1964) 190

Medial Set, Boundary, and Topology of Random Point Sets A. Imiya1,2 , H. Ootani3 , and K. Tatara3 2

1 National Institute of Informatics, Japan Institute of Media and Information Technology, Chiba University, Japan 3 School of Science and Technology, Chiba University, Japan

Abstract. In this paper, we aim to develop an algorithm for the extraction of a medial set of a random point set in two- and three-dimensional spaces. Using the medial set of a random point, we define the topology of a random point set. The algorithm for the extraction of a median set is based on the principal surface analysis.

1

Introduction

In this paper, we introduce a skeletonization procedure for random point sets, employing the principal curve and surface of random point sets [1, 2], which are extensions of principal components of random variables in a vector space [3, 4, 5]. Using the skeleton of a random point set, we define the combinatorial topology of a random point set. The skeletons of a binary object in a plane and in a space are a tree form curve [6, 7, 8, 9, 10] and the collection of part of the curved surface [11, 12, 13], respectively. These skeletons are called the medial axis and surface, or more generally, the medial set of an object [11, 12, 13]. The skeleton of an object is a fundamental geometric feature for image and shape analysis. Therefore, skeletonization has been studied in the field of pattern recognition and computer vision for a long time. Since the distance-transform-based skeletonization [14, 15] or the grass-fire model of the skeleton [6, 7] derives points inside a shape as the local symmetry axis of an object [8], this local symmetry property of the skeleton leads to the conclusion that the distance from a point on the distance-transform-based skeleton to points on the boundary takes the minimum for at least two points on the boundary. The interpolation procedure [16, 17] reconstructs a curve and surface from a sequence and array of sample points, respectively. There have been many efforts for the reconstruction of smooth surface from unorganized sample points which are, for example, measured by range sensors [18, 19, 20]. Because of the resolution and sampling procedure, the three-dimensional positions of measured sample points usually contain noise. Therefore, for the surface reconstruction from noisy data, a smoothing process is required [16, 21]. A typical method is mesh simplification base on signal processing on a polyhedral surface [22, 23, 24, 25]. T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 196–217, 2003. c Springer-Verlag Berlin Heidelberg 2003

Medial Set, Boundary, and Topology of Random Point Sets

197

Computational geometry provides combinatorial methods for the recovery of boundary curves and surfaces as polygonal curves and polyhedral surfaces, respectively. These algorithms are based on Voronoi tessellation, Delaunay triangulation, Gabriel graphs, crust, α-shape, and β-skeleton [9, 10, 25]. These methods include a triangulation process of sample points and smoothing operations based on the configurations of point on the triangulated polyhedron. In the second step, these algorithms add, eliminate, and merge a number of sample points based on the criterions of the algorithms. Our algorithm constructs a piecewise linear manifold from a random point set. However, the manifolds do not always contain original sample points, that is, starting from a simple initial shape, for example, a line segment and triangle in a plane and a tetrahedron in a space, the algorithm constructs a medial set, which is piecewise linear, for the random sample points. This is the fundamental difference between our method and conventional methods for the reconstruction of curves and surfaces. In this sense, our method is a model fitting and selection process. Since our algorithm preserves the combinatorial topology [26] of the initial shape, the selection of initial shape affects the topology of the reconstructed surface. This mathematical property is the basis the definition of the combinatorial topology of random point sets by using the combinatorial topology of the initial shape which minimizes a criterion.

2 2.1

Mathematical Background Spline Interpolation

Regression model fitting for planar sample points {(xi , yi ) }ni=1 for x1 < x2 < · · · < xn is achieved, for example [16], by minimizing the criterion J(f ) =

n

ρ(|yi − f (xi )|) + λ

i=1

n−1 xi+1 i=1

xi

2 d f (τ ) dτ 2

dx,

(1)

τ =x

where ρ(τ ) is a positive symmetry function. Furthermore, a two-dimensional analogue of this criterion is, for example [16], defined as J(f ) =

n

ρ(|zi −f (xi , yi )|)+λ

i=1

n−1 xi+1 i=1

xi

yi+1

yi

2

|fτ τ + 2fτ µ + fµµ |τ =x,µ=y dxdy,

(2) for sample points {(xi , yi < zi ) }ni=1 for x1 < x2 < · · · < xn and y1 < y2 < · · · < yn . These criterions yield solutions in the forms f (x) =

n−2 i=2

wi si (x), f (x, y) =

n−2

wi si (x, y),

i=2

where si (x) and si (x, y) are base functions for the interpolation.

(3)

198

2.2

A. Imiya et al.

Principal Curves and Surfaces

Let X be a mean-zero point distribution in Rn . The major principal component w maximizes the criterion J(w) = Ex∈X |x w|2

(4)

with respect to |w| = 1, where Ex∈X expresses the expectation over set X. Line x = tw is a one-dimensional linear subspace which approximates X. A maximization criterion (5) J(P ) = Ex∈X |P x|2 with respect to rankP = k, for 2 ≤ k < n, determines a k-dimensional linear subspace which approximates X. If X is not a mean-zero point distribution in Rn and the centroid of X is not predetermined, the maximization criterion J(P , g) = Ex∈X |P (x − g)|2

(6)

with respect to rankP = k, for 2 < k < n, determines a k-dimensional linear manifold which approximates point distribution X. If g = 0, P is computed using PCA [3]. In the previous papers [4, 5], we extend the idea of PCA for g = 0. This mechanism automatically estimates g and P , even if many clusters exist in a space [4, 5]. N For the partition of X into {Xi }N i=1 such that X = ∪i=1 Xi , vectors g i and wi which maximize the criterion J(w 1 , · · · , wN , g 1 , · · · , g N ) =

N

Ex∈Xi |(x − g i ) w i |2

(7)

i=1

determine a polygonal curve [1], l = g i + twi . Furthermore, for an appropriate N partition of X into {X}N i=1 , such that X = ∪i=1 Xi , vector g i and orthogonal projector P i , which maximize the criterion J(P 1 , · · · , P N , g 1 , · · · , g N ) =

N

Ex∈Xi |P i (x − g i )|2

(8)

i=1

with respect to rankP i = k, for 2 < k < n, determine a piecewise linear manifold, Mi = {x + g i |P i x = x}, if M ⊂ Xi which approximates X. These piecewise linear curve and surface are called the principal curve and principal surface, respectively [1]. In Figure 1, (a), (b), and (c) show the geometries of data compression by the principal component, the principal curve, and the principal surface, respectively. In Figures 1 (b) and 1 (c), wp,i is the i-the principal component in a part of a random point set. We adopt the principal curve and principal surface of random point sets in two- and three-dimensional spaces, respectively, as the medial sets of random point sets. For the construction of continuous curves and surfaces, we are required to construct a minimization criterion which connects the portions of principal components and linear manifolds.


199

wp,3 w2

wp,2

wp,2

w1

wp,1

wp,1

Xu

Xu

Xu

(a)

(b)

(c)

Fig. 1. (a) Principal components, (b) Principal curve, and (c) Principal surface

3

Curve Detection

Set D and S to be a random point set and the vertices of a polyhedral surface, respectively, and the distance between point x ∈ S and y ∈ D is defined as d(x, D) = miny ∈D d(x, y) for the Euclidean distance in an n-space. The initial shapes S and C are a line segment whose direction is equivalent to the major component w1 of a random point set and a regular triangle whose vertices are determined from the principal components w 1 and w2 . For a sequence of vertices v 1 , v 2 , · · · v n of a polygonal curve, we define the tesselation as Vα = {x|d(x, v α ) < d(x, v i ), d(x, v α ) < d(x, eij ), α = i}, Eαα+1 = {x|d(x, eαα+1 ) < d(x, v i ), d(x, eαα+1 ) < d(x, eii+1 ), α = i}, where eii+1 is the edge which connects v i and v i+1 . The minimization criterion of reference [2] is expressed as I= for F (v k , D) =

v k ∈C

F (v k , D) + λ

x∈Ek−1k

d(x, v k ) +

1

v k ∈P

x∈Vk

v i−1i v ii+1 |v ||v ii+1 | i−1i i=−1

d(x, v k ) +

x∈Ekk+1

(9)

d(x, v k ).

Using this criterion, we obtain an algorithm for the detection of the principal curve [2] where IK is the value of I with K vertices. 1. Set the vertices of the initial curve as S. 2. Move all vertices v i i = 1, 2, · · · , K, to minimize IK . 3. Generate the new vertex v K+1 on the curve S.

200

A. Imiya et al.

4. If |IK −IK−1 | ≤ ε for a positive constant ε, then stop, else set S := S∪{v K+1 } and go to 2. This incremental algorithm preserves the topology of the initial curve, since the algorithm generates new vertices on the curve. This geometrical property leads to the conclusion that this algorithm reconstructs closed or open curves, if the initial curve is closed or open, respectively. Therefore, if we evaluate the number of data points in the tessellations yielded by the algorithm, we can also detect the topology of point clouds on a plane. If the distribution of sample points in a region Eii+1 is space, we can conclude that the principal curve of a random point set is open. Then, we obtain an extension of the algorithm derived in reference [2]. Algorithm C 1. First phase: (a) Set the vertices of the initial curve as a closed curve such that C := {v 1 , v 2 , v 3 }. (b) Move all vertices v i i = 1, 2, · · · , K to minimize IK . (c) Generate the new vertex v K+1 on the curve C. (d) If |IK − IK−1 | ≤ ε for a positive constant ε, then stop, else set C := C ∪ {vK+1 } and go to 1(b). 2. Condition evaluation: After an appropriate number of iterations, evaluate the topological condition of the curve. If the topology of the initial shape derives a contradiction of the tessellations, then go to 3, else go to 1 (b). 3. Second phase: (a) Select the new initial point set as an open curve such that S := {v 1 , v 2 }. (b) Move all vertices v i i = 1, 2, · · · , K to minimize IK . (c) Compute v K+1 . (d) If |IK − IK−1 | ≤ ε for a positive constant ε, then stop, else set S := S ∪ {vk+1 } and go to 3(b). The criterion defined by eq. (9) does not depend on the dimensions of a space. Therefore, the method extracts the principal curve of a random point set in a 3space. Algorithm C detects the medial set of a random point set as the principal curve. This geometrical property yields the definition of the curve skeleton of a random point set, following the definition of the curve skeleton in the discrete 2-space. Definition 1 A polygonal curve derived by Algorithm C is the curve skeleton of a random point set on a plane and in a space.

4

Surface Detection

4.1

Optimization Criterion

Setting α = i, β = j, and γ = k, we define the partitions of space by a random point set as Vα = {x|d(x, v α ) < d(x, v i ), d(x, v α ) < d(x, eij ), d(x, v α ) < d(x, f ijk )},


201

Eαβ = {x|d(x, eαβ ) < d(x, v i ), d(x, eαβ ) < d(x, eij ), d(x, eαβ ) < d(x, f ijk )}, Fαβγ = {x|d(x, f αβγ ) < d(x, v i ), d(x, f αβγ ) < d(x, eij ), d(x, f αβγ ) < d(x, f ijk )}, where eij is the edge which connects a pair of vertices v i and v j , and f ijk is the triangular face whose vertices are v i , v j , and v k . This decomposition of space is the Voronoi tessellation whose generators are polygonal surfaces with the vertex K(n) set {v i }i=1 , where K(n) is the total number of vertices of the polyhedral surface generated in the K-th step. Setting {v k(i) }ni=1 , for v k(n+i) = v k(i) , to be the vertices adjunct to vertex v k and v m(i) to be the vertex whose minimum path length to vertex v i is two and which lies on face f k(i)k(i+1)m(i) , we define the following minimization criterion J= F (d(v k , D)) + λ1 S1 (v k ) + λ2 S2 (v k ) (10) v k ∈S v k ∈S v k ∈S for F (v k , D) =

n

d(v k , Fkk(i)k(i+1) ) +

i=1

n

d(v k , Ekk(i) ),

i=1

k(n)

S1 (v k ) =

|v k − v k(j) |2 ,

j=1

k(n)

S2 (v k ) = −

j=1

v k(j)k × v k(j+1)k |v k(j)k × v k(j+1)k |

v k(j+1)m(j) × v k(j)m(j) |v k(j+1)m(j) × v k(j)m(j) |

,

where v ij = v i − v j . The first term controls the fitness of a reconstructed surface since this term minimizes the distance between the reconstructed surface and points in a random point set. The second and third terms are stabilizers. The second term becomes minimum if point v k moves to the centroid of the connectk(n) ing vertices {v k(i) }i=1 . Therefore, the second term controls the local smoothness of the reconstructed surface. If triangles ∆v k v k(i) v k(i+1) and ∆v k v m(i) v m(i+1) lie on the same plane for each i, the third term becomes minimum. Therefore, the third term controls the local roughness of the surface. Setting JK to be the value of J with K vertices, we derive the following algorithm. 1. Set the vertices of the initial surface as S. 2. Move all vertices v i , i = 1, 2, · · · , K, to minimize JK . 3. Generate the new vertex v k+1 on the surface S. This new vertex also generates new edges and faces. 4. If |JK − JK−1 | ≤ ε for a positive small constant ε, then stop, else set S := S ∪ {v K+1 } and go to 2. We have no proof for the convergence property of this algorithm. However, the criterion |JK −JK−1 | usually decreses starting from an appropriate initial shape. In the final subsection of this section, we introduce a method for the definition of initial shapes, using the geometric property of the point-generation procedure.

202

A. Imiya et al.

va

va

vp

vb

vc vb

(a)

(b)

va

va

vp

vb

vc vb

vd

Fig. 2.

vp

vc

vc

vd

(c)

4.2

vp

(d)

Generation of new vertices

Point Generation

If the point density in Fαβγ is higher than that of Fα β γ for (α, β, γ) = (α , β , γ ), we bend Fαβγ by adding a vertex and edges, and separating this face into faces. Starting from the initial set, we generate a new vertex at the center of the longest edge in S as illustrated in Figure 2. Vertices v a , v b , v c , and v d are vertices of a polyhedral surface and v p is a newly generated point. Edges v p v a , v p v b , v p v c , and v p v d are new edges generated by adding a new vetex. There are two types of configurations of the longest edge on surface S. If the longest edge eij lies on a unique face f ijk we generate a vertex v m and an edge ekm . Then, the face f ijk is separated into a pair of new faces f kim and f kmj . Furthermore, if the longest edge eij is shared by a pair of faces f ijk and f ijn , we generate a vertex v m and a pair of edges ekm and emn . Then, a pair of faces f ijk and f nij is separated into a quadruplet faces f ikm , f jmk , f jnm , and f imn . Systematically, these rules are expressed as vi , vi, vk → vi, vi , vk , vm , eij , ejk , eki → eim , emj , ejk , eki , ekm , f ijk → f ijm , f jmk ,


203

and vi , vi , vk , vn → vi , vi , vk , vm , vn eij , ejk , eki , ejn , eni → eij , ejk , eki , ejn , f ijk

eni , eim , ekm , ejm , enm , → f ikm , f kjm , f jnm , f nim .

Setting K(v), K(e), and K(f ) to be the numbers of vertices, edges and faces, respectively, for the K-th iteration, the generation rules for vertices imply the relations, (K + 1)(v) = K(v) + 1, (K + 1)(e) = K(e) + 2, (K + 1)(f ) = K(e) + 1, (11) and (K + 1)(v) = K(v) + 1, (K + 1)(e) = K(e) + 3, (K + 1)(f ) = K(e) + 2, (12) if the longest edge lies on a face and is shared by a pair of faces, respectively. Since these relations hold the equality (K + 1)(v) − (K + 1)(e) + (K + 1)(f ) = K(v) − K(e) + K(f ),

(13)

we have the following theorem. Theorem 1 The vertex and edge generation rule defined in the previous subsection preserves the topology of the initial surface S. 4.3

Initial Shapes

Since the algorithm preserves the combinatorial topology of the initial shape, we are required to prepare appropriate initial shapes. In this paper, and a closed surface topologically equivalent to the torus. we deal with the shapes whose Euler characteristics are zero and two. As the initial surfaces for we adopt a tetrahedron, which is topologically equivalent the sphere, a tetrahedron without a face, which is topologically equivalent an open surface, a triangle-pipe without two triangle faces, which is topologically equivalent a surface with two holes, and polyhedral torus with nine vertices and eighteen edges, which is topologically equivalent the torus. These polyhedral surfaces are are shown in figure 3 (a), 3 (b), 3 (c), and 3 (d), respectively. Since for the principal curve and surface analysis, the reference point and initial curve are determined using the centroid and the principal component of a random point set, we define the reference points and configurations of vertices of initial shapes by using the principal components and values of a random point set according to the following rules: – The centroid of the tetrahedron is placed at the centroid of a random point set. The vector from the centroid to a vertex is parallel to the principal major component. The distances between vertices and the centroid depend on the principal major value.

204

A. Imiya et al.

(a)

(b)

(c)

(d)

Fig. 3. Initial polyhedrons in the three-dimensional space. (a) An initial polyhedron for closed surfaces. (b) An initial polyhedron for open surfaces. (c) and (d) are initial polyhedrons whose Euler number is zero

– The centroid of a random point set is adjusted to the center of the open face. The open face is defined in subsection 4.4. The distances between vertices and the centroid depend on the principal major value. – The centroid of the triangle-pipe is placed at the centroid of a random point set. The open faces are defined in subsection 4.4. The distances of side edges depend on the principal major value. – The centroid of the torus is placed at the centroid of a random point set. The edges of the rim of the polyhedral torus are parallel to the plane which is defined in section 4.4. The lengths of edges of the inner triangle of the hole depend on the principal major value. 4.4

Topology of Random Point Set

We assume that points of a random point set distribute uniformly in the neighborhood of a shape V in Rn for n = 2, 3. A shape is the boundary of a closed finite region in Rn . We call this shape the generator shape. We introduce a method to detect the topology of the generator shape from point distribution. Let s(Fijk ) be the number of points in the tessellation defined by face f ijk and s be the average of s(Fijk ). If s(Fijk ) < β s, we say that the point distribution of tessellation Fijk is sparse. Furthermore, if the point distribution of Fijk is sparse, face f ijk might be yielded in a region where the generator shape of a random point set does not exist. We call these faces the sparse face. An appropriately selected initial shapes does not yield any sparse faces, since we assume points distribute uniformly in the neighborhood of a generator shape. For example, let the generator shape of a point set is a torus. However a tetrahedron, which is topologically equivalent to a sphere, is selected as the initial shape. Then, after an appropriate number of iterations, the point distributions in the tessellations defined by some faces are sparse. This geometric property is yielded by the miss-selection of the initial shape.


205

As we have proven in the previous subsection, the algorithm preserves the topology of the initial shape. Therefore, we can estimate the topology of the generator shape from a random point set, if we select an appropriate initial shape which is topologically equivalent to the generator shape. For the resetimation of the appropriate initial shape, we evaluate the density of points in each tessellation defined by the principal manifold and determin the sparse faces. To remove the sparse faces, we reset the initial shape to an appropriate one and restart the evolution of the initial shape using the algorithm. For the selection of the new initial shape, we use the number of connected components of faces 1 which determine space tessellations. After an appropriate number of iterations for the evolution of the principal manifold, we define a collection of faces hαβγ , α, β, γ ∈ {k(i)}ni=1 , G = {x|d(x, hαβγ ) < d(x, v i ), d(x, hαβγ ) < d(x, eij ), d(x, hαβγ ) < d(x, f ijk )}. (14) Since G is the collection of the sparse faces, we define the combinatorial topology of the generator shape using the number of connected components of G. Proposition 1 Let c(G) be the number of the connected components of G. If c(G) = 0, S is a closed surface topologically equivalent to a sphere. If c(G) = 1, S is an open surface. If c(G) = 2, S is a closed surface topologically equivalent to a torus or an open surface with a hole. Using these topological properties of S, we define the topology of a random point set. Definition 2 The combinatorial topology of the generator shape of a random point set is that of the reconstructed principal surface. Then, we adopt the combinatorial topology of the generator shape as the topology of a random point set. As proven in subsection 4.2, our algorithm preserves the combinatorial topology of the initial polyhedron. We propose the following algorithm, considering the topology of the initial polyhedron. The algorithm starts to compute the principal surface by adopting the initial shape as a closed polyhedron, evaluates the distribution of sample points in tessellation Fijk , then selects an appropriate initial polyhedron to restart the algorithm. Algorithm S 1. First phase: (a) Set the vertices of the initial polyhedron as a closed tetrahedron such that S := {v 1 , v 2 , v 3 , v 4 } (b) Move all vertices v i , i = 1, 2, · · · , K, to minimize JK (c) Compute v K+1 (d) If |JK − JK−1 | ≤ ε for a positive small constant ε, then stop, else set S := S ∪ {v K+1 } and go to 1(b). 1

If a pair of faces shares an edge, we call these faces are connected. The connected component is the collection of all connected faces.

206

A. Imiya et al.

2. Condition evaluation: After an appropriate number of iterations, evaluate the topological conditions of the surface. If the topology of the initial shape derives a contradiction then go to 3, else go to 1 (b). 3. Second phase: (a) Select the new initial point set S, reset the value Jn . (b) Move all vertices v i i = 1, 2, · · · , K, to minimize JK (c) Compute v K+1 (d) If |JK − JK−1 | ≤ ε for a positive small constant ε, then stop, else set S := S ∪ {v K+1 } and go to 3(b). Definition 3 A polyhedral surface derived by Algorithm S is the surface skeleton of a random point set. Setting H to be the collection of linear manifolds which approximate the collection of connected faces corresponding to sparse tessellations, we use H for the generation of the new initial shape in step 3. In step 3, we adopt an initial shape which does not contain any faces on the collection of linear manifolds which are parallel to the elements of H.

5

Topology of Random Point Set

Setting {vk(i) }ni=1 for k(n + i) = k(i) to be the boundary of S for k-th iteration, where {v 1(i) }4i=1 = {v 1 , v 2 , v 3 , v 4 }, we define a collection of faces hαβγ , α, β, γ ∈ {k(i)}ni=1 . We define the set G as G = {x|d(x, hαβγ ) < d(x, v i ), d(x, hαβγ ) < d(x, eij ), d(x, hαβγ ) < d(x, f ijk )} (15) for α, β, γ ∈ {k(i)}ni=1 Setting s(Fijk ) to be the number of sample points in region Fijk , if s(Fijk ) ≤ β s, where s is the average of s(Fijk ) for all faces sharing edges and vertices with f ijk , the distribution of sample points in Fijk is sparse. G is a collection of faces such that the distribution of sample points in Fijk is sparse. If each pair of vertices of G has paths, S has a region in which the distribution of sample points is sparse, otherwise S has regions in which the distribution of sample points is sparse. Therefore, the number of connected components of G, characterises the topology of reconstructed surface S. Using the following definition, we define the initial shapes. Definition 4 Setting H to be the linear manifold or manifolds approximating the point cloud in the union of Fijk which share edges and vertices on the boundary of G, we call H the corresponding manifold to G. The number of H is equivalent to the number of connected components of G. As the initial polyhedron of which Euler number is one, we adopt an open tetrahedron the face corresponding to H is removed from the closed tetrahedron.


207

We deal with the case that the number of connected components of G, say c(G), is zero, one and two. Now, we define the topology of point clouds employing the number of connecting components of Gp . For the number of connected components of G, we have the following proposition. Proposition 2 1. If c(G) = 0, S is a closed surface topologically equivalent to sphere. 2. If c(G) = 1, S is an open surface. 3. If c(G) = 2, S is a closed surface topologically equivalent to torus or an open surface with a hole Using these topological properties of S, we define the topology of a random point set which derives the principal surface S using our algorithm. Definition 5 The combinatorial topology of a random point set is that of the reconstructed principal surface. As proven in section 3, our algorithm preserves the combinatorial topology of the initial polyhedron. we propose the following algorithm, considering the topology of the initial polyhedron. The algorithm starts to compute the principal surface adopting the initial shape as a closed polyhedron, evaluates the distribution of sample points in Fijk , then selects an appropriate initial polyhedron again to restart the algorithm. Algorithm S 1. First phase: (a) set S := {v1 , v 2 , v 3 , v 4 } (b) move all v k to minimize Jn (c) compute v k+1 (d) If |Jn − Jn−1 | ≤ ε for a positive small constant ε, then stop, else set S := S ∪ {v k+1 } and go to (a). 2. Condition evaluation: After an appropriate number of iterations, evaluate the topological conditions of the surface. if the topology of the initial shape derives contradiction then got to step 3, else go to 1 (a). 3. Second phase: (a) select the new initial point set S, reset the value Jn . (b) move all v k to minimize Jn (c) compute v k+1 (d) If |Jn − Jn−1 | ≤ ε for a positive small constant ε, then stop, else set S := S ∪ {v k+1 } and go to 3(a).

6

Numerical Examples

We present some examples. For the computation of these examples, we select √ λ1 = 0.15ρ and λ = 0.06ρl, where ρ = ra n2/3 in which a, r, l and n are the average of distances between a surface and points, the maximal distance between

208

A. Imiya et al.

Table 1. Statistics of the surfaces reconstructed from synthetic data. k(v)∗ is the number of vertices of each reconstructed surface after 185 iterations Volume sphere computed 3.8188 expected 4.1879 semisphere computed 1.9931 expected 2.0944 cube computed 7.7085 expected 8.0000

Area k(v)∗ 12.0783 396 12.5664 6.3285 369 6.2832 22.0074 396 24.0000

points and their centroid, the averages of all edges on the surface, and the number sample points. These parameters, a, r, and l are automatically computed from data, since they depend on data. In all our examples of synthetic data, we set n = 1000. We generated a random point set, adding Gaussian distribution with variance 0.005 in the perpendicular direction of the geometric surfaces. The geometric surfaces are a semi-sphere, a sphere, and a cube. We evaluated the volumes and surface areas of reconstructed polyhedral surfaces. The statistics are listed in Table 1. Figures 4 (a) and (b), respectively, show the volume and surface area of the reconstructed polyhedrons with respect to the number of vertices. These figures show that the volume and surface area of a reconstructed object are smaller than the original values, because the algorithm is fundamentally based the curvature flow [27, 28, 29, 30] for vertices of a polyhedron. The reconstructed objects by curvature flow are usually smaller than the original values since the curvature flow usually shrinks the surface. In Figure 5, (a), (b), and (c) are random point sets and (d), (e), and (f) are reconstructed surfaces. In Figure 6, (a) and (d) are random point sets, and (b) and (e) are reconstructed figures assuming that the initial polyhedrons are closed, that is, the original polyhedrons are closed tetrahedrons. Then, there exist triangular patches with Voronoi regions in which the distribution of points is sparse. Figures 6 (c) and (f) are polyhedrons reconstructed from appropriate initial polyhedrons which are an open polyhedron and torus, respectively. Figure 7 shows the process for the detection of an open surface with a hole. (a) and (b) show a random point set and its side view. (c), (d), and (e) are reconstructed polyhedrons whose initial polyhedrons are a closed tetrahedron, torus, and triangle-pipe without two triangles. Figure 7 (e) shows the reconstructed polyhedron without triangles with Voronoi regions in which the distribution of points is sparse. Figure 8 shows the two-dimensional version of topology detection of random point sets. (a), (b), and (c) show the original random point sets and the initial curves superimposed on them. As shown in (a), (b), and (c) and (d), starting from a closed curve, if a segment of a polygonal curve appears in the region in


209

4

12 3.5

11.5 11

3

10.5 10

2.5

9.5 9 2

8.5 8

1.5

7.5 1

50

100

150

(a)

200

250

300

7

50

100

150

200

250

300

(b)

Fig. 4. Volume (a) and surface area (b) with respect to the number of vertices of a reconstructed polyhedron for the sphere which the point distribution is sparse, we conclude the principal curve of random point set is open. Figure 9 shows the detection of the principal curves in a space. These results show that our method estimates the topology of a random point set detecting the triangle patches with Voronoi regions in which the point distribution is sparse. In Figure 10, (a) and (b) show the principal curve in a three-dimensional space whose initial curve is closed, and (b) the principal curve detected as an open curve using the two-phase method. Furthermore, (c) shows a closed principal curve for a random point set which distributes along a closed curve in a space. In Figure 11, we show (a) a synthetic random point set and (b) its γ-boundary extracted by the algorithm developed in section 5. If we apply Algorithm C to the result in Figure 11 (b), it is possible to extract the γ-skeleton of the point set in Figure 11 (a) as the boundary of the generator shape of a random point set using Algorithm C.

7

Conclusions

For tele-surgery for robots, it is necessary to detect the geometric data of deformable organs in the human body during the surgery. If we measure such deformable objects using a multi-directional camera system, we obtain a noisy cloud of sample points which distribute around the surface of the deformable object. For the tracking and computation of geometric parameters of deformable objects from the cloud of sample points, it is necessary to estimate the boundary of the object which lies in the cloud of sample points.

210

A. Imiya et al.

1

1

0.5

0.5

0

0

1.5

−0.5

z

z

z

1 0.5

−0.5

0 −0.5

−1

−1 −1

1

1

1 0.5

1 0.5

0

1

−1

0.5

0

0

−0.5 y

0.5 0

−0.5

−0.5

−1

−1 x

y

−0.5

−1.5 −1.5

0.5 −1

0 −0.5

0

−0.5 0.5

1

−1 x

−1 1.5

−1.5

y

x

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 5. (a), (b), and (c) are random point sets and (d), (e), and (f) are reconstructed surfaces


211

Fig. 6. (a) A random point set distributes along an open semisphere, (b) the reconstructed surface whose initial shape is a closed tetrahedron, and (c) the surface reconstructed by the two-phase method wich selected an open tetrahedron as the correct initial shape, (d) a random point set distributes along a torus, (e) the reconstructed surface whose initial shape is a closed tetrahedron, and ( f ) the surface reconstructed by the two-phase method which selected a torus as the correct initial shape

212

A. Imiya et al.

1 0.8

1

0.6 0.4

0.5

0.2 0

z

z

0

−0.2

−0.5

−0.4

−1 1

−0.6

1

0.5 0.5

0 −0.5 y

−0.8

0

−1

−0.5 −1

−1

1

x

(a)

(c)

0.5

0 y

−0.5

−1

−0.5

0 x

0.5

1

(b)

(d)

(e)

Fig. 7. (a) A random point set distributes along a truncated spherical shell, and (b) its side view which has two corresponding linear manifolds. (c) The surface reconstructed from a closed tetrahedron, (d) the surface reconstructed from a torus, and (f) the surface reconstructed using two-phase method


1.5

1.5

1.5

1

1

1

0.5

0.5

0.5

0

0

0

−0.5

−0.5

−0.5

−1

−1

−1

−1.5 −1.5

−1

−0.5

0

0.5

1

1.5

−1.5 −1.5

−1

−0.5

(a)

0

0.5

1

1.5

−1.5 −1.5

1.5

1

1

1

0.5

0.5

0.5

0

0

0

−0.5

−0.5

−0.5

−1

−1

−1

−0.5

0

(d)

0.5

1

1.5

−1.5 −1.5

−1

−0.5

0

(e)

0

0.5

1

1.5

0.5

1

1.5

(c)

1.5

−1

−0.5

(b)

1.5

−1.5 −1.5

−1

213

0.5

1

1.5

−1.5 −1.5

−1

−0.5

0

(f)

Fig. 8. Initial curves and final curves. (a) A closed initial curve detects the closed princpal curve (d). (b) An open initial curve detects the open principal curve (e). (c) A closed initial curve detects the closed principal curve (f). The initial point distributions of (a) and (b) are the same

214

Fig. 9. from [: from [:

A. Imiya et al.

'

Reconstructed surfaces of Stanford-Bunny and Mannequin. (a) data (b) noisy data based on (a), and (c) reconstructed from (b). (d) data (e) noisy data based on (d), and (f) reconstructed from (e)


1.5

1.5

1

1

0.5

0.5

0

0

−0.5

−0.5

215

1.5 1

z

0.5 0 −0.5 −1

1

1.5

0.5

1

0

0.5 0

−0.5

−0.5

−1 −1.5

−1 −1.5

(a)

−1.5 −1.5

1

1.5

0.5

1

0

0.5 0

−0.5

1 0.5

−1

0

−0.5 0

−1.5

1.5

−1 −1.5

(b)

−1

1

−0.5

−1

−0.5

0.5 −1.5

y

x

(c)

Fig. 10. Curves in a space. (a) The principal curve in a three-dimensional space whose initial curve is closed, and (b) the principal curve detected as an open curve using the two-phase method. (c) A closed principal curve for a point cloud which distributes along a closed curve in a space

We have introduced a discrete variational method for the surface fitting of a random point set. There are some methods for the learning of the geometric properties of a random point set [2, 20, 26]. Our method automatically learns the combinatorial topology of a random point set by evaluating the number of data points in tessellations which are derived as the solution our discrete variational method in each step. If we start the computation of a shape estimation process from an inappropriate initial shape, our algorithm yields triangle meshes for the regions in which data points sparsely distribute. Many variational criterions for curve fittings and boundary extractions are first defined as a continuous model, and are second discretized for numerical computation. On the other hand, our method is defined as a discrete model. Therefore, there is no approximation caused by discretization for the numerical computation. In section 6, we have demonstrated the convergence of our method with respect to the volume and the surface area of geometrically generated data. We have also tested our algorithm for two numerical examples of practical data obtained from the world-wide-web [31, 32]. These results demonstrated the performance of our method is sufficient. However, the theoretical analysis for the convergence of our discrete variational method remains an open problem. Using the estimated principal manifolds as the initial forms, it is possible to detect the time trajectory of the manifolds of a time varying random point set. For the application to trajectory detection, a fast computation method for the variational method is required. Furthermore, the detection of transition between the median surface and the medial curves of random point sets is required. One possible solution to this problem is to compute both the principal curve and the principal surface concurrently, to evaluate the values of minimization criterions

216

A. Imiya et al.

(a)

(b)

Fig. 11. (a) A synthetic random point set and (b) its γ-boundary. It is possible to extract the γ-skeleton from the γ-boundary as the boundary of the generator shape of a random point set

for the curve and the surface, and to select the description which derives the minimum.

References [1] Hasite, T., Stuetzle, T., Principal curves, J. Am. Statistical Assoc., 84, 502-516, 1989. 196, 198 [2] Kégl, B., Krzyzak, A., Linder, T., Zeger, K., Learning and design of principal curves, IEEE PAMI, 22, 281-297, 2000. 196, 199, 200, 215 [3] Oja, E., Principal components, minor components, and linear neural networks, Neural Networks, 5, 927-935, 1992. 196, 198 [4] Imiya, A., Ootani, H., PCA-based model selection and fitting for linear manifolds, LNAI, 2123, 278-292, 2001. 196, 198 [5] Imiya, A., Kawamoto, K., Learning dimensionality and orientations of 3D objects, Pattern Recognition Letters, 22, 75-83, 2001. 196, 198 [6] Blum, H., Biological shape and visual science, J. Theoretical Biology, 38, 205285, 1963. 196 [7] Rosenfeld, A., Axial representations of shapes, CVGIP, 33, 156-173, 1986. 196 [8] Bookstein, F. L., The line-skeleton, CVGIP, 11, 1233-137, 1979 196 [9] Amenta, N., Bern, M., Eppstein, D., The crust and the β-skeleton: Combinatorial curve reconstruction, Graphical Models and Image Processing, 60, 125-135, 1998. 196, 197 [10] Attali, D. and Montanvert, A., Computing and simplifying 2D and 3D continuous skeletons, CVIU, 67, 261-273, 1997. 196, 197 [11] Nystrom, I., Sanniti di Baja, G., Svensson, S., Curve skeletonization by junction detection Lecture Notes in Computer Science 2059, 229-238, 2001. 196


217

[12] Svensson, S., Nystrom, I., Sanniti di Baja, G., Curve skeletonization of surfacelike objects in 3D images guided by voxel classification, Pattern Recognition Letters, 23, 1419-1426, 2002. 196 [13] Sanniti di Baja, G., Svensson, S., Surface skeletons detected on the D6 distance transform. Lecture Notes in Computer Science 1876, 387-396, 2000. 196 [14] Svensson, S., Borgefors, G., Nystrom, I., On reversible skeletonization using anchor-points from distance transforms Journal on Visual Communication and Image Representation 10, 379-397, 1999. 196 [15] Svensson, S., Sanniti di Baja, G., Using distance transforms to decompose 3D discrete objects, Image and Vision Computing, 20, 529-540, 2002. 196 [16] Silverman, B. W., Some aspects of the spline smoothing approach to nonparametric regression curve fitting, J. R. Statist. Soc, B. 47, 1-52, 1985. 196, 197 [17] Wahba, G., Surface fitting with scattered noisy data on Euclidean D-space and on the sphere, Rocky Mountain Journal of Mathematics, 14, 281-299, 1984. 196 [18] Reed, M. K., Allen, A. K., 3-D modeling from range imagery, Image and Vision Computing, 17, 99-111, 1999. 196 [19] Terzopoulos, D., The computation of visible-surface representations, IEEE PAMI, 10, 417-438, 1988. 196 [20] Freedman, D., Efficient simplicial reconstruction of manifolds from their samples, IEEE PAMI, 24, 1349-1357, 2002. 196, 215 [21] Li, W. and Swetits, J. J., The linear l2 estimation and the Huber M -estimator, SIAM J. Optimization 8, 457-475, 1998. 196 [22] Hoppe, H., DeRose, T., Duchamp, T., McDonald, J., Stuetzle, W., Surface reconstruction from unorganized points, Computer Graphics (SIGGRAPH ’92 Proceedings)”, 26, 71-78, 1992. 196 [23] Taubin, G. Detecting and reconstructing subdivision connectivity, The Visual Computer, Vol18, 357-367, 2002. 196 [24] Taubin, G., Ronfard, R., Implicit simplical models I: Adaptive curve reconstruction, PAMI, 18, 321-325, 1996. 196 [25] Edelsbrunner, H., Shape reconstruction with Delaunay complex, Lecture Notes in Computer Science, 1380, 119-132, 1998. 196, 197 [26] Martinetz, T., and Schulten, K., Topology representing networks, Neural Networks, 7, 507-522, 1994. 197, 215 [27] Huisken, G., Flow by mean curvature of convex surface into sphere, Journal of Differential Geometry, 20, 237-266, 1984. 208 [28] Bruckstein, A. M., Shapiro, G., and Shaked, D., Evolution of planar polygons, Journal of Pattern Recognition and Artificial Intelligence, 9, 991-1014, 1995. 208 [29] Zhao, H.-K., Osher, S., Merriman, B., and Kang, M., Implicit and nonparametric shape reconstruction from unorganized points using variational level set method, CVIU, 80, 285-319, 2000. 208 [30] Sethian, J. A., Level Set Methods: Evolving Interfaces in Geometry Fluid Mechanics, Computer Vision, and Material Science. Cambridge University Press, Cambridge, 1996. 208 [31] http://www-graphics.stanford.edu/. 214, 215 214, [32] ftp://ftp.research.microsoft.com/users/hhoppe/data/thesis/input pts/. 215

Scale-Based Corner Extraction of a Contour Figure Using a Crystalline Flow Hidekata Hontani1 and Koichiro Deguchi2 1

2

Department of Informatics, Yamagata University 4-3-16, Johnan, Yonezawa, Yamagata 992-8510, Japan [email protected] Graduate School of Information Sciences, Tohoku University Aramaki-aza Aoba01, Aoba-ku, Sendai 980-8579, Japan [email protected]

Abstract. We propose a scale-based method for extracting corners from a given polygonal contour figure. A crystalline flow is introduced to represent geometric features in a scale-space. It is an extension of a usual curvature flow. A special class of polygonal contours is evolved based on the nonlocal curvature. The nonlocal curvature is determined for each facet by its length. In the crystalline flow, a given polygon remains polygonal through the evolving process. Different from a usual curvature flow, it is easy to track a facet in a given polygon through the evolution. This aspect helps us to extract a set of dominant corners. Experimental results show that our method extracts a set of dominant corner facets successfully from a given contour figure.

1

Introduction

Evolution based scale-space methods play an important role to characterize a contour figure[1][2]. Especially, a curvature flow is widely used to obtain the scale-space. In a curvature flow, a contour evolves in the normal direction with the speed V that is determined by the curvature κ. As a contour evolves, small geometric features in the contour are smoothed out, and large features come to appear. There are classical and thoroughly elaborated theories on the contour evolution. For example, it is proved that, letting V = κ, any contour converges to a circle at finite time, and that no new curvature inflection point is generated as the contour evolves. These aspects permit us to define the scale using the time in the evolving process[3]. The scale-space methods have been mainly treated a smooth curve. In a case of V = κ, for example, it is known that any evolving contour is analytic at t > 0. In a digital image, however, contours that we encounter cannot be considered smooth: they are discrete. Most of all methods for obtaining a curvature flow, e.g. a level set method[4] or other smoothing methods[5], represents an smooth contour with a series of sampled points in it. Because the arc length of the evolving contour keeps changing through the evolving process, it is not easy T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 218–230, 2003. c Springer-Verlag Berlin Heidelberg 2003

Scale-Based Corner Extraction of a Contour Figure Using a Crystalline Flow

219

to track a point through the process so as to represent characteristics in the scale-space. In this article, in order to characterize a contour shape, we introduce a crystalline flow, which is an essentially discrete version of a classical curvature flow. A crystalline flow is a family of evolving polygons. An initial contour must be a polygon that is called an admissible crystal, and the evolving contour remains an admissible crystal through the evolution. In the crystalline flow, each facet evolves in the normal direction with the speed V that is determined by the nonlocal curvature Λ. The quantity of the nonlocal curvature is calculated using the length of each facet. It is proved that letting V = Λ, for example, the number of facets in the polygon decreases as t increases, and the polygon becomes convex at finite time. We can track each facet straightforwardly through the evolving process, because the contour remain polygonal. We propose a method for extracting dominant corners. In the classical framework, a corner is defined as a point that has a maximal curvature[6]. In the presented framework of the crystalline flow, on the other hand, a corner is defined as a facet that has non-zero nonlocal curvature. As the time t increases, the number of corner facets decreases, in the evolving process. Tracking each facet, we determine the lifetime of it in the evolving process, and extract dominant corners. Some experimental results show that the presented method extracts dominant facets well.

2 2.1

A Crystalline Flow A Curvature Flow and an Wulff Shape

A curvature flow is a family of evolving curves, and is widely used for characterizing of shapes. In the flow, the normal velocity V of every point in the evolving curve depends on the curvature κ. In a physical problem, a curvature flow corresponds to the first variation of the interfacial energy S γ(n)ds with respect to the change of the area enclosed by the curve S, where γ is the interfacial energy density, n is the outward unit normal, and s is the arc length along the curve. For example, when γ(x) = |x| and no other constraints exist, the initial contour evolves with the speed V = κ. The flow that corresponds to this evolution is called a curve-shortening flow: the total length of the curve becomes shorter through the evolution at the speed S κ2 ds, and the curve converges to a circle before it disappears at finite time. The characteristics of the energy density function γ influences the evolution of a curve. A special family of evolving curves that corresponds to the case when the interfacial energy γ is not necessarily C 1 has been studied[8][9][12]. This family is called a crystalline flow. When γ is not C 1 , the evolution of a curve has nonlocal feature. A typical example of such a γ is a piecewise linear function. This γ is called a crystalline energy as proposed by [8] and [9]. For a crystal, the energy of a surface depends on its normal orientation with respect to the crystal lattice. If the energy density of a surface is plot as a radius

220

Hidekata Hontani and Koichiro Deguchi

Fig. 1. An example of an Wulff shape (left) and an admissible crystal (right)

along its normal direction, we obtain a locus that is called an Wulff shape. The Wulff shape represents the characteristics of the energy density function. In 2D problem, the Wulff shape is a simple closed contour in a plane. In the case γ(x) = |x|, the Wulff shape is a unit circle. On the other hand, the Wulff shape is a polygon, when γ is piecewise linear. 2.2

The Nonlocal Curvature of an Admissible Crystal

A crystalline flow is a family of evolving polygons that are called admissible crystal. An admissible crystal is defined using an Wulff shape, which plays a role of an unit circle for a classical curvature flow. Assume that a convex polygon W is given as an Wulff shape. An admissible crystal is defined as follows. Definition 1 Let N be a set of all outward unit normals of the Wulff shape W . An admissible crystal is a simple polygonal curve P of which all outward unit normals belong to N and any pair of two adjacent facet normals of P can be found in a set of pairs of adjacent facet normals of the Wulff shape W . Figure 1 shows an example of an Wulff shape and a corresponding admissible crystal. In a classical case, a local curvature of a smooth curve can be defined as an inverse of the radius of the inscribing circle. On the other hand, a nonlocal curvature Λ of an admissible crystal is defined for each facet using an inverse of the length of the facet (see Fig.2) as follows. Here, we number facets in clockwise. Definition 2 A nonlocal curvature Λi that corresponds to the ith facet of an admissible crystal P is defined as Λ i = χi

∆(ni ) , Li

(1)

where Li is the length of the ith facet, ∆(ni ) is the facet length of W that is parallel to the ith facet. The quantity χi is called a transition number, and takes +1 (-1) if the ith facet is concave (convex). If the admissible polygon is neither convex nor concave at the ith facet, then the χi = 0. Figure 3 shows the variety of the transition numbers. By this convention, each facet of the Wulff shape W itself has the nonlocal curvatures of Λi = −1.


221

Li

r

Fig. 2. A local curvature κ = 1/r and a nonlocal curvature Λi = χi ∆(ni )/Li

χ = −1

χ=0

χ = +1

Fig. 3. A transition number χj

2.3

A Crystalline Flow

It has been widely studied on a family of evolving polygons. Moving each facet at the outward normal speed V = g(ni , Λi ) for some time interval J, we obtain a set of polygons {Pt }t∈J from a given polygon P0 . In this article, given an Wulff shape W , we set (2) V = Λi , which corresponds to V = κ. If a given initial contour is an admissible crystal and the equation (2) is satisfied, then the evolving polygon has the following properties[10]. Property 1 There exists a unique flow {Pt }t∈ J for a given admissible crystal, where J = [0, T∗ ), any Pt is an admissible polygon, and the evolving polygon disappears at t = T∗ . Property 2 Before the evolving polygon Pt disappears at t = T∗ , the polygon becomes convex. Property 3 Every facet disappearing before t = T∗ always has zero transition number and at most two consecutive facet disappear together. Property 4 No new facet is generated through the evolution. Let the ith facet length at time t be denoted with Li (t). Then, Li (t) always satisfies next equation. dLi (t) = (cot ψi+1 + cot ψi )Vi − (sin ψi )−1 Vi−1 − (sin ψi+1 )−1 Vi+1 , dt

(3)

222


where, ψi = θi − θi−1 for the outward normal vector ni = (cos θi , sin θi ). We obtain the crystalline flow based on the equation (3). A crystalline flow is an extended version of a curvature flow. In [12], a convergence property of a crystalline flow is proved. The proof indicates that a crystalline flow obtained from an admissible crystal P0 that approximates another contour C0 approximates the curvature flow obtained from the original contour C0 . This property is seriously important for this article, because the property allows us to approximate a given contour by means of an admissible crystal.

3

Multi-scale Analysis Using a Crystalline Flow

A contour in an image is often represented with a series of points in a lattice that are linked together. In the presented article, we consider a contour that is represented with points in a lattice each of which links to two points of eight neighbors. For example, a contour extracted from an image consists of pixels each of which links to two neighbor pixels, and can be described with a Freeman chain-code. It should be noted that such a contour is a polygon and the normals of the curve can have only eight orientations. We introduce a crystalline for multi-scale analysis of such a polygonal contour, because such a contour is easily approximated with an admissible crystal, the computational procedure for obtaining the flow is simple, and the resulted flow is stable. Assume that the Wulff shape is a regular unit octagon (you can use an unit rectangle as the Wulff shape, if the given contour is described with fourconnections). There exist eight normals nj (j = 1, 2, · · · , 8) of the Wulff shape. Let the set of all the normals be denoted as N , and assume that each facet length of the Wulff shape ∆j = 1. Any normal of a given polygonal contour P0 considered here is included in N . We introduce a crystalline flow Pt that satisfies V = χj /Lj (t).

(4)

In order to obtain a crystalline flow, a given contour must be an admissible crystal. We approximate a given contour with an admissible crystal. If there exists a non-admissible part in a given contour as is shown in Fig.4, then, we insert short facets so as to be admissible. We must note that we do not know clearly about the relation between the given polygonal contour and the crystalline flow obtained from an admissible crystal of the approximation. It should be noted, though, that a crystalline flow is stable to the perturbation of the shape of an initial contour[11][12]. An example of a crystalline flow is shown in Fig.5. Any admissible crystal becomes an octagon at finite time. In order to extract dominant corners from a given contour, we make a scalespace representation using the crystalline flow. Here, the x-axis of the scale-space shows the facet number i in an initial admissible crystal, and the y-axis shows the time t. For making the representation, we must track each facet in the initial contour through the evolving process to fix the facet number i. As is described in 2.3, it is known that no new facet is generated as t increases, only a facet of χj = 0 can disappear, and the evolving polygon becomes convex before it


223

Fig. 4. Approximations with an admissible crystal

disappears at t = T∗ . These knowledge helps us to track each facet in an initial polygon through the evolving process. Every facet in the initial contour whose transition number is 0 necessarily disappears at some time in the evolving process. When a facet is disappearing, it is eroded by a neighboring facet, and finally its length becomes 0, so that the neighboring two facets are merged into one facet. Hence, a facet in an evolving contour at t = t0 can be tracked back to several facets in the initial contour. Tracking each facet in the initial contour, we make a scale-space representation of the transition number. As is shown in Fig.6, the transition number is plot at the corresponding position in the scale-space. Let the representation be denoted as χ(j, t), where j is the facet number in the initial contour and t is the time. This representation is analogous to a usual curvature scale-space. As t increases, the number of (non-trivial) facets in an evolving contour decreases. Following (4), the evolving contour becomes an octagon at finite time, and finally disappears. When an evolving contour becomes an octagon, the transition number of each facet becomes -1. Figure 7 shows the scale-space representation corresponds to a crystalline shown in Fig.5.

4

Dominant Corner Facet Extraction

In the notion of the crystalline flow, the curvature is not defined for each point in the contour, but is defined for each facet. In this article, we propose a method for extracting a dominant corner facet from a given contour. Referring to the scalespace representation of the transition number, we extract a dominant corner facet whose transition number is not 0 and the value of the transition number is

Fig. 5. An example of a crystalline flow

224


χ=0

Time χ=0 χ=0 χ=0 χ=+1 χ=0 χ=−1 χ=+1 χ=0 χ=−1 χ=+1 χ=0 χ=−1

χ=+1

χ=0 χ=−1

Facet Number in Initial Contour

Fig. 6. A scale-space representation of the transition number. The left figures show an evolving curve, and the right one shows the corresponding scale-space representation. The x-axis represents the facet number in the initial contour, and the y axis represents the time t

inherited for a long time interval in the evolving process. The proposed algorithm is as followings. 1. Approximate a given contour with an admissible crystal by inserting short facets at the non admissible part, and obtain a crystalline flow that follows (4). 2. Make the scale-space representation of the transition number. Let the representation be denoted as χ(j, t), where j is the facet number in the initial contour and t is the time. 3. Divide the scale-space into areas, so that each area has the uniform value of χ(j, t) inside, and has different value from the neighboring areas. Let denote

t

o

j

Fig. 7. A scale-space representation that corresponds to Fig.5. The white area represents χj = −1, black one χj = +1, and the gray one χj = 0


225

Time χ=−1

A1

Base Scale t0

χ=0

A2

A4

A5

0

χ=+1 A3

A2

Facet Number

Fig. 8. Dominant corner extraction using the scale-space representation. The facets in an initial contour χ = 0 are extracted, if they can be tracked to the base scale t0

such the area as Ak , where k = 1, 2, · · · , n is the serial number. It should be noted that every area Ak touches to the line t = 0(see Fig.8). 4. Set the base scale t0 , and draw a line t = t0 in the scale space. Then, find a set of numbers Mt0 , so that the area Ak contains the line t = t0 and has the value of χ(j, t) = 0, if k ∈ Mt0 . 5. Extract a set of points from the line t = 0 that are included in the area Ak for all k ∈ Mt0 . Each of these points corresponds to a facet in the initial contour whose transition number is inherited to an evolving contour at t = t0 . This set of extracted facets is the resulted set of dominant corner facets for the base scale t0 . In Fig.8, the position of the extracted points are indicated by up-arrows. 6. Increase the base scale t0 by small amount ∆t, and repeat 3.4.5. if t0 is smaller than the scale at which the evolving contour becomes convex. We obtain different set of dominant corner facets for different base scale t0 . If t0 < t1 , then, Mt0 ⊃ Mt1 . As the result, the number of the extracted facets decreases as t0 increases, and only more dominant facets remain. By changing t0 continuously, we have only finite sets of dominant facets. We will show some experimental results in the next section.

5

Experimental Results

Figure 9 shows the extracted facets by the proposed method. The initial contour has 310 facets. By changing the base scale t0 , we obtain several sets of dominant corner facets. The approximations of a given contour are shown that are obtained

226


by linking the facets. As shown in Fig.9, fewer facets are extracted for higher value of the base scale. Figure 10 shows the graph of the number of extracted facets with respect to the change of the base scale. The graph has a staircase pattern. The stable sets of the dominant facets give an intuitively good results.

(A)

(B)

(C)

(D)

(E)

Scale (E) (D) (C) (B) (A)

Facet Number

Fig. 9. An experimental result of dominant facet extraction. The corresponding base scales are shown with lines in the scale-space

Number of Extracted Facet

40

O Base Scale 250

Fig. 10. The change of the extracted dominant facets with respect to the base scale t0


227

Time

Facet Number in Initial Contour

Extracted Dominant Corner Facet

Fig. 11. Left: An example of a crystalline flow. Middle: An experimental result of dominant facet extraction. Right: Corresponding base scales shown with lines in the scale-space

Figure 11, 12, and 13 show another experimental results. As shown in Fig.11 and Fig.12, small notches in an initial contour disappear as t increases, and finally, the contour becomes an octagon. The dominant corner facets extracted by the method are shown in Fig.11 and 13. As the base scale t0 increases, more global shape is represented by the set of dominant facets. The graph of the number of extracted facets is shown in Fig.13 with some results. Stable sets of dominant facets give intuitively good results.

6

Summary

For a multi-scale analysis of a contour figure, we propose to introduce a crystalline flow. In this article, a method for extracting a dominant corner is proposed. A crystalline flow is an extension of a usual curvature flow: some class of a polygonal curve, an admissible crystal, is evolved based on the nonlocal curvature. The nonlocal curvature is determined for each facet on an evolving curve by the length of a facet, which is easily calculated correctly. A crystalline flow is not a family of smooth evolving curves, but of polygonal evolving curves. The conditions of the facet disappearance are, therefore, clearly known, and it is not difficult to track each facet through the evolving process.

228


Time

Facet Number

Fig. 12. Another example of a crystalline flow and its scale-space representation of the transition number

These aspects are the main advantages of introducing a crystalline flow for shape analysis. In many cases, a given contour is represented as a polygon, of which normals can have only finite directions. For example, a contour figure extracted from an image is a polygon, and the normal of the polygon can have only eight directions. We propose to approximate such a given contour with an admissible crystal, and obtain the crystalline flow for multi-scale analysis. Because the crystalline flow is defined with a polygonal curve, implementation is very straightforward. It is also not difficult to track each facet in the initial contour through the evolving process. Making a scale-space representation using the crystalline flow, our method extracts a dominant corner facet from a given contour. A facet in an initial contour is extracted, if the value of its transition number can be tracked to the base scale t0 through the evolving process. By changing this base scale, we obtain several sets of dominant facets. The set of the facets comes to represent more global shape, when the base scale is increased. Some experimental results show that our method extracts a set of dominant corner facets well from a given contour figure. For using a crystalline flow, we firstly approximate a given contour with an admissible one. We do not know the mathematical relationship correctly between the approximation and the given original contour. We need more mathematical study on the crystalline flow, especially on a crystalline flow for non-admissible initial polygon.


229

Number of Extracted Facets

100

0 100

Base Scale

Fig. 13. The change of the extracted dominant facets with respect to the base scale t0

References 1. J. J. Koenderink, The structure of images, Biological Cybernetics, 50(1984), pp.363–370 218 2. L. Alvarez, F. Guichard, P. Lions, and J. Morel, Axioms and fundamental equations of image processing, Arch. Ration, Mech. Anal., 123, 3(1993), pp.199–257 218 3. T. Lindeberg, Scale-space theory in computer vision, Kulwer Academic(1994), in chap.8 218 4. J. A. Sethian, Level Set Methods : Evolving interfaces in geometry, Fluid Mechanics, Computer Vision, and Material Science, Cambridge(1996) 218 5. F. Mokhtarian and A. K. Mackworth, A theory of multi-scale, curvature-based shape representation for planar curves, IEEE Trans. Pattern Analysis and Machine Intelligence, 14, 8(1992), pp.789–805 218 6. A. Rattarangsi and R. T. Chin, Scale-based detection of corners of planar curves, IEEE Trans. Pattern Analysis and Machine Intelligence, 14, 4(1992), pp.430–449 219 7. B. M. ter Haar Romeny edt. Geometry Driven Diffusion in Computer Vision, Kluwer Academic Publishers(1994) 8. J. Taylor, Constructions and conjectures in crystalline nondifferential geometry, In Differential Geometry (eds. B. Lawson and K. Tanenblat), Proceedings of the

230

9.

10.

11.

12.

Hidekata Hontani and Koichiro Deguchi Conference on Differential Geometry, Rio de Janeiro, Pitman Monographs in Pure and Applied Math. 52(1991), pp.321–336 219 S. B. Angenent and M. E. Gurtin, Multiphase thermomechanics with interfacial structure 2. Evolution of an isothermal interface, Arch. Rational Mech. Anal. 108(1989), pp.323-391 219 M.-H. Giga and Y. Giga, Motion driven by nonlocal curvature in the plane, In Asymptotics in Nonlinear Diffusive Systems, (eds. Y. Nishiura et al.), Tohoku Math. Publ. 8(1998), pp. 75-83 221 M.-H. Giga and Y. Giga, Crystalline and level set flow -convergence of a crystalline algorithm for a general anisotropic curvature flow in the plane, GAKUTO International Series Mathematical Sciences and Applications, 13(2000), pp.64–79 222 M.-H. Giga and Y. Giga, Generalized motion by nonlocal curvature in the plane, Arch. Rational Mech. Anal. 159(2001), pp.295–333 219, 222

Hyperfigures and Their Interpretations Kokichi Sugihara Department of Mathematical Informatics, University of Tokyo 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan [email protected]

Abstract. This paper proposes a new formulation of the Minkowski algebra for figures. In the conventional Minkowski algebra, the sum operation was always defined, but its inverse was not necessarily defined. On the other hand, the proposed algebra forms a group, and hence every element has its inverse, and the sum and the inverse operation can be taken freely. In this new algebraic system, some of the elements does not correspond to the figures in an ordinary sense; we call these new elements “hyperfigures”. Physical interpretations and practical usage of the hyperfigures are also discussed. Keywords: Minkowski sum, inverse, slope-monotone curve, hyperfigures, algebraic extension.

1

Introduction

The Minkowski sum is one of the basic concepts in morphology, and has a wide range of applications. The typical applications include data compression for character images [5, 8], detection and deletion of pattern noises [12, 26], manipulation of solid models [23] and generations of collision-avoidance paths [6, 17, 18, 19]. The Minkowski sum is also useful for constructing efficient geometric algorithms for computing the minimum distance of two convex polygons [10, 25], for judging the polygon containment [6] and for many other problems [6, 10, 2]. The Minkowski sum has been studied also from a computational point of views, and actually many algorithms are proposed for general polygons [11, 13, 14, 21], for restricted classes of polygons [1, 15, 22], for figures with curved boundaries [16], and for higher dimensional figures [8]. However, the algebraic structure of the Minkowski sum is complicated because the inverse operations cannot be defined in general. There is another operation called the Minkowski decomposition [5]. This operation behaves like the inverse of the Minkowski sum for some restricted class of figures such as convex figures. Ghosh [7, 8] tried to extend the inverse operation to a larger class of figures, but did not succeed in it completely. Sugihara et al. [27] observed that an operation similar to the Minkowski sum can be defined on a class of differentiable closed curves in such a way that the inverse operation is always defined. Thus the invertibility problem was partially solved. T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 231–246, 2003. c Springer-Verlag Berlin Heidelberg 2003

232

Kokichi Sugihara

This paper is an extension of [27]. That is, we extend the class of closed curves to a larger one, including a kind of nondifferentiable curves, and construct a new Minkowski algebra in which the sum operation is always invertible. This class of curves can represent polygons in an arbitrarily close approximation. In this algebra, new elements are generated through inverse operations; we call these new elements “hyperfigures”, because they are generated just in the same manner as the hyperfunctions are generated from functions [20]. We will review the conventional Minkowski sum in Section 2, and define our new Minkowski sum in Section 3. In Section 4, we extend the object would so that the sum operation is always invertible, and thus define the “hyperfigures”. In Section 5 we farther extend the algebra so that it includes arbitrarily close approximations of polygons. In Section 6 we give some physical interpretation of the hyperfigures.

2

Conventional Minkowski Sum

For two figures A, B in the plane, the Minkowski sum, denoted by A ⊕ B, is defined by A ⊕ B ≡ {a + b | a ∈ A, b ∈ B}, (1) where A and B are considered the sets of position vectors and a + b is considered the vector sum. Let A and B be the figures shown in Fig. 1(a) and (b), respectively, where the horizontal and vertical lines represent the x and y axes. Then, A ⊕ B is as shown in Fig. 1(c). The Minkowski sum can be intuitively understood in the following way. Consider the figures A and B in Fig. 1(a) and (b) again. As shown in Fig. 2, A ⊕ B is the union of the figure A and the region swept by B when the origin of B moves on the boundary of A. Hence, in particular, the boundary point a + b of A ⊕ B is obtained by a pair of boundary points a and b with the same tangent direction. As seen by this interpretation, the Minkowski sum depends only on the boundary of the figure. Based on the observation, several representation such as slope diagram [9], and support functions [9, 24] have been used to reformulate the Minkowski sum. Another interpretation of the Minkowski sum is the following. For a figure X, let r(X) denote the figure we obtain by rotating X by π around the origin of the coordinate system. Suppose that, as shown in Fig. 3, the figure r(B) moves around the figure A while keeping in contact. At the point of contact, the two figures have a common tangent line, and consequently the vector sum a + b that generates the boundary of A ⊕ B represents the location of the origin of r(B). Hence, A ⊕ B represents the region in which the origin of r(B) cannot enter unless A and B intersect. Because of this property, the Minkowski sum can be used for finding collision-free path of a convex object amidst obstacles [6, 17, 18, 19].

Hyperfigures and Their Interpretations

233

A B (a)

(b)

A

B

(c)

Fig. 1. Minkowski sum of two convex figures: (a) figure A; (b) figure B; (c) sum A⊕B

b a

B

A A

B

Fig. 2. An interpretation of the Minkowski sum

234

Kokichi Sugihara

b A

a

A

B

Fig. 3. Another interpretation of the Minkowski sum Suppose that we are given two figures A and C, and we consider equation A ⊕X =C

(2)

for unknown figure X. This equation can be solved partially in the following way. For two figures A and B, the region defined by A B ≡ {a − b | a ∈ A} (3) b∈B

is called the Minkowski difference or the Minkowski decomposition of A and B. It is known that if the figures A and B are convex and C = A⊕B, then B = C A; that is, the solution of eq. (2) is given by C A [7]. However, for general figures A and C, eq. (2) behaves in a complicated way; sometimes the solution does not exist, while in other cases it exists but is not necessarily unique. Therefore, we have to be careful when we apply the Minkowski operation because otherwise the result of the computation may become nonsense. So our goal is to construct a simpler algebra in which the Minkowski sum and its inverse operation can be taken freely.

3

New Algebra

As we have seen in Fig. 2 and Fig. 3, the Minkowski sum is substantially determined by the boundaries of the two figures. In the definition in eq. (1), a and b


235

moves in A and B, respectively, but the essential information is conveyed in the shapes of the boundaries of A and B. Hence, let us concentrate on boundary curves. We consider curve c(t) that satisfies the following five properties. Property 1. c(t) is a continuous mapping from [0, ∞) to R2 . Property 2. There exists a positive integer k such that c(t) = c(t + k) for any t. Property 2 means that c(t) is a cyclic function. The smallest k that satisfies Property 2 is called the cycle of c(t). Property 3. c(t) is differentiable except at a finite number of points in [0, k], when k is the cycle of c(t). Hence, c(t) forms a smooth closed curve. Intuitively we may consider t as time, and c(t) as the position vector of a smoothly moving point at time t; then c (t) can be interpreted as the velocity of the moving point c(t). dc(t) , and let ||c (t)|| represents the Euclidean norm of Let us define c (t) ≡ dt the vector c (t). Property 4. For any t ∈ [0, ∞), c(t) satisfies c (t + ε) c (t + ε) = lim ε→+0 ||c (t + ε)|| ε→−0 ||c (t + ε)|| lim

(4)

where ε → +0 means that ε is positive and approaches to 0, and ε → −0 means that ε is negative and approaches to 0. If c(t) is differentiable at t, eq. (4) is trivialy satisfied. Property 4 says that, even if c(t) is not differentiable at t, the tangent direction at t is defined. In this sense, Property 4 implies that the trajectory of c(t) is smooth even though c(t) itself is not necessarily differentiable. Let arg(c (t)) denote the angle of the direction c (t) measured counterclockwise with respect to the positive x direction; arg(c (t)) Always exists because of Property 4. The last property is as follows. Property 5. arg(c (t)) = 2πt for any t. Property 5 implies many facts. First, substituting t = 0, we get arg(c (0)) = 0, and consequently the velocity at t = 0 is in the positive x direction. Secondly, the parameter t of the curve c(t) is chosen in such a way that t corresponds to the direction of the tangent line at c(t), that is, t represents the angle arg(c (t)) divided by 2π. Thirdly, arg(c (t)) is monotone increasing. Therefore, as t increases, the point c(t) changes its motion direction to the left continuously. Fourthly, Properties 2 and 5 imply that, after changing the motion direction by 2kπ (where k

236

Kokichi Sugihara

is the cycle of the curve), the point c(t) reaches the start position and repeats the same trajectory. Let us denote by M + the set of all the curves that satisfy Properties 1∼5. From now on, we denote the element of M + by c or {c(t)}, and denote the point on this curve corresponding to the parameter value t by c(t). For any two curves c1 , c2 ∈ M + , we define c1 ⊕ c2 ≡ {c1 (t) + c2 (t)},

(5)

and call it the Minkowski sum of c1 and c2 ; note that c1 ⊕ c2 also belongs to M + . An example of the Minkowski sum based on this new definition is shown in Fig. 4. Let c1 and c2 be the two closed curves in (a) and (b), respectively, whose start points c1 (0) and c2 (0) are as shown in the figure. Then, the Minkowski sum c = c1 ⊕ c2 is as shown in Fig. 4(c). The cycle of the curve c1 is equal to 2, and the cycle of the curve c2 is equal to 1; the cycle of the Minkowski sum c is equal to 2. Let c1 and c2 be the curves shown in Fig. 4(a) and (b), respectively, and let A1 and A2 be the regions bounded by the outermost parts of c1 and c2 . Then, the region bounded by the outermost part of c = c1 ⊕ c2 corresponds to the conventional Minkowski sum A1 ⊕ A2 . In this sense, our definition is a generalization of the conventional one.

4

Hyperfigures

From the definition, the following properties hold for any c1 , c2 , c3 ∈ M + .

c1(0) (a)

c2(0) (b)

(c)

Fig. 4. Relation between the new Minkowski sum and the conventional one


237

Property 6. c1 ⊕ c2 = c2 ⊕ c1 (commutativety), Property 7. (c1 ⊕ c2 ) ⊕ c3 = c1 ⊕ (c2 ⊕ c3 ) (associativity), Property 8. c1 ⊕ c2 = c1 ⊕ c3 implies c2 = c3 (injectivity). It is known in algebra that, if Properties 6, 7 and 8 are satisfied, the set M + can be augmented in such a way that all the elements have their inverses in it [3]. This augmentation is done formally in the following way. Let us define M + × M + ≡ {(c, d) | c, d ∈ M + }. The operation ⊕ is induced to M + × M + in such a way that (c1 , d1 ) ⊕ (c2 , d2 ) ≡ (c1 ⊕ c2 , d1 ⊕ d2 ). We define binary relation ∼ in M + × M + by (c1 , d1 ) ∼ (c2 , d2 )

if and only if

c1 ⊕ d2 = c2 ⊕ d1 .

Then, the relation ∼ is an equivalence relation in M + × M + . Moreover, this relation is consistent with ⊕, i.e., if (c1 , d1 ) ∼ (c2 , d2 ) and (c3 , d3 ) ∼ (c4 , d4 ), then we have (c1 ⊕ c3 , d1 ⊕ d3 ) ∼ (c2 ⊕ c4 , d2 ⊕ d4 ). Hence we can obtain the quotient algebra: M ≡ (M + × M + )/ ∼ . That is, M is the partition of M + × M + into subsets consisting of mutually equivalent elements together with the induced operation ⊕. This augmentation process can be understood more easily if we compare it to the augmentation from the set Z+ of positive integers to the set Q+ of positive rationals with respect to the multiplication operation. In this augmentation, element (a, b) ∈ Z+ × Z+ is usually denoted by a/b. Hence, the equivalence relation (a, b) ∼ (c, d) is nothing but a/b = c/d (i.e., a · d = b · c). Let 0 represent the curve that stays at the origin of the coordinate system for any t. For c ∈ M + , c corresponds to the equivalence class of (c, 0), and thus we can identify the equivalence class of (c, 0) as c itself. In this sense M is an extension of M + . For c ∈ M + , the equivalence class of (0, c) behaves as the inverse of (c, 0) because (c, 0) ⊕ (0, c) = (c, c) = 0. We denote the equivalence class of (0, c) by c−1 . According to our original definition, c−1 = (0, c) ≡ (−c, 0) = −{c(t)}.

(6)

Hence the inverse of c can be considered as the curve obtained when we rotate c by π around the origin of the coordinate system. The equivalence class of (c1 , c2 ) ∈ M + × M + can be identified as c1 ⊕ c−1 2 = {c1 (t) − c2 (t)}.

(7)

Note that c1 (t) and c2 (t) represent the points on the curves c1 and c2 respectively for the particular parameter values t, and hence c1 (t) − c2 (t) represents the

238

Kokichi Sugihara

Fig. 5. Hyperfigure difference of the radial vectors of these two points, and {c1 (t)−c2 (t)} is a function of t (i.e., a curve). Hence, a general element of M can be represented by c1 − c2 for some c1 , c2 ∈ M + . In particular, 0 corresponds to the equivalence class of (c, c) for any c ∈ M + . Consider the figure in Fig. 4 again. Let c1 and c2 be the closed curves in (a) and (b) respectively. Then c1 ⊕c−1 2 is as shown in Fig. 5. This is a typical example of an element in M \M + , where M \M + denotes the set of all the elements in M not included in M + . We call elements in M \M + hyperfigures. Thus, we can augment the set M + of closed curves to the set M of closed curves and hyperfigures. M forms a group and hence we can take the Minkowski sum and its inverse operation freely. We borrow the name “hyperfigure”, because our extension of the algebraic system is similar to the extension from functions to hyperfunctions. Another possible name might be “rational figures”, because the structure of the extension is similar to the extension from integers to rationals. However, the terms “rational curves” and “rational surfaces” have already been used in different meanings [4]. So, in order to avoid the confusion, we decided to use the term “hyperfigure”.

5

Algebra for Polygons +

Here we shall extend the class M + of closed curves to a larger class, say M , that includes polygons. + Let M be the set of closed curves each of which consists of a finite number of (straight) line segments and a finite number of (nonstraight) curved segment in such a way that each curved segment turns always to the left just like the


Fig. 6. Element of M

239

+

+

curves in M + . A typical example of an element of M is shown in Fig. 6, where the arrow represents the direction in which a point moves as the parameter increases. + Let c(s) denote a general element of M . We keep the symbol t to represent the special parameter that satisfies Property 5, and use the symbol s for a general + parameter. A general element c(s) of M differs from an element of M + only in that it contains straight parts, and the curve can turn to the right at the + corners. Now, we will show that for every curve c(s) in M , there exists a curve + d(t) ∈ M whose trajectory is arbitrarily close to that of c(s). Let c be an + element of M . The approximation procedure consists of three steps. In the first step, we round left-turn corners of c. Suppose that, as shown in Fig. 7(a), p is a corner point of the curve c at which the curve turns to the left. Then, as shown in Fig. 7(b), we add a small circle, say a circle with radius ε, touching the two segments incident to the corner c, and as shown in Fig. 9(c), we change the sharp corner p by a rounded corner. Similarly we replace all the sharp left-turn corners of c by small circular arcs.

p (a)

(b)

(c)

Fig. 7. Elimination of a left-turn corner

240

Kokichi Sugihara

p

(a)

(b)

(c)

Fig. 8. Elimination of a right-turn corner

In the second step we round right-turn corners of c. Suppose that, as shown in Fig. 8(a), p is a corner point of the curve, at which the curve turns to the right. Then, as shown in Fig. 8(b), we extend the two segments incident to p and place a small circle with radius ε touching the extended part of the segments. We replace the corner point by both of the extended segments and the circular arc, as shown in Fig. 8(c). We do the same procedure at all right-turn corners. By these two steps, we change the curve so that there is no sharp corners. Hence, the resulting curve is smooth and consists of straight line segments and nonstraight curved segments. In the third step, we replace each straight line segment with a curved segment. Suppose that l is a straight part of the curve. Then, as shown in Fig. 9(a), l is connected at both of the end points to left-turning circular arcs. So, as shown by the broken line in Fig. 9(b), we generate a very large circle, say with radius 1/ε, touching this part of the curve at both of the circular arcs. We replace the part of the curve between the two tangent points with this circular arc, as shown in Fig. 9(c). We carry out the same replacement for all of the straight line segments. Then, we get a curve belonging to M + , and this curve approximates the original curve c well if ε is very small. Therefore, the Minkowski algebra for + elements of M can be approximately carried out by the Minkowski algebra for elements of M + .

6

Interpretation of Hyperfigures

Here we consider physical interpretation of hyperfigures through examples. Example 1 (insectifuge-spread ability) Suppose that the region A in Fig. 10(a) represents the shape of a vehicle that can translate in any direction but cannot rotate. This vehicle can spread insectifuge from the source point indicated by the small dot; the region B in


241

l

(a)

(b)

(c)

Fig. 9. Elimination of a straight part of the closed curve Fig. 10(a) represents the area which the insectifuge can reach. Without loss of generality, we choose the origin of the coordinate system at the source point. Now, for example, suppose that we are given a corn field whose shape is represented by C in Fig. 10(b). We want to spread insectifuge over the field using the vehicle. The vehicle can move by translation outside the field freely but it cannot move inside the corn field. We want to know whether the insectifuge can reach the whole area of the field, and if not, we want to know which area the insectifuge cannot reach. Conventionally this problem is solved in the following way. As shown in Fig. 10(c), first we compute D = C ⊕ r(A). As we saw in the previous section, D represents the region in which the source point of the vehicle cannot enter. Next, we compute E = D B, i.e., we remove from D the region to which the insectifuge can be spread. The resulting figure E represents the region where the insectifuge can not reach, as shown by the shaded area in Fig. 10(c). The above procedure is summarized by E = (C ⊕ r(A)) B = D B.

(8)

Let cr(A) , cB , cD , cE denote the boundary curves of the regions r(A), B, C, D, E, respectively. Then, eq. (8) can be expressed by our notation as −1 cE = (cC ⊕ cr(A) ) ⊕ c−1 B = cD ⊕ cB .

(9)

Since ⊕ is associative, eq. (9) is equivalent to cE = cC ⊕ (cr(A) ⊕ c−1 B ) = cC ⊕ cF ,

(10)

242

Kokichi Sugihara

B

A

C

(a)

(b)

C D

F E

(c)

(d)

Fig. 10. Power for spreading the insectifuge

where cF is defined by

cF = cr(A) ⊕ c−1 B .

Since M is a group, cF ∈ M , but cF ∈ / M + and hence cF is a hyperfigure. If we dare to draw the boundary of cF , we get the trajectory as shown in Fig.10(d). The boundary cE of the region E is obtained either by eq. (9) or by eq. (10). The advantage of the latter equation is that, cF is determined by the shape A of the vehicle and the shape B of the insectifuge-spread region. Once we compute cF , we can apply it to any shape of the corn field to get cE = cC ⊕ cF . Thus, we can interpret that “cF represents the ability of the vehicle to spread the insectifuge.” In this way we can give a concrete meaning to a general element of M .

✷


243

Example 2 (insectifuge-spread in on a windy day) Readers might feel that the unrotatable vehicle is unnatural. Indeed an actual vehicle usually can rotate as well as translate. More natural situation to interpret eq. (10) is the following. Suppose that d is the smallest distance from the source point to the boundary of the vehicle shape A. As shown in Fig. 11(a), let A be the circle with the radius d centered at the source point. Then A represents the behavior of the vehicle A when it can both translate and rotate, because by appropriate translation and rotation the source point of the vehicle can approach to the corn field as close as d. On the other hand, assume that it is a windy day, and that the wind blows from the east to the west constantly. Then the region to which the insectifuge can be spread forms a circle, say B , whose center is to the east of the source point, as shown in Fig. 11. Let us define cF ≡ cA ⊕ c−1 B . Note that cF ∈ M but cF ∈ / M +.

B’

A’

C

(a)

(b)

D

E

(c)

C

F

(d)

Fig. 11. Power for spreading the insectifuge in a windy day

244

Kokichi Sugihara

By the same argument as the previous example, we can interpret that cF represents the ability of the vehicle to spread the insectifuge on this windy day. ✷ By these two examples, we have seen that a general element of M can have a physical meaning (i.e., the ability of the insectifuge-spread vehicle) even though it does not belong to M + . Similar interpretation is also possible in the case where a man (=vehicle) tries to reach a floating ball on the pond (=corn field) by a long stick (=insectifuge-spread circle), and in the case where a robot arm (=vehicle) moves to processes material (=corn field) by a cutter (=insectifuge-spread circle) in its hand.

7

Concluding Remarks

In this paper we have reformulated the Minkowski algebra so that it forms a group and contains many new objects, which we call hyperfigures. We have also shown that we can have a concrete physical meaning of hyperfigures. From a theoretical point of view, this recognition gives a new insight to the algebraic structure of the Minkowski sum. Our new recognition is also useful from a practical point of view. Recall that the hyperfigure cF can be applied to various shapes of the corn field. The precomputation of such a hyperfigure can save the total computational cost of the morphological operation. To find an interpretation of a hyperfigure obtained by the Minkowski sum of nonconvex figures is a challenging future problem. Other problems in future include clarification of the algebraic structure of the Minkowski sum for nonconvex figures, and the construction of an efficient method for representing hyperfigures.

Acknowledgement This work is supported by the Grant-in-Aid for Scientific Research of the Ministry of Education, Science, Sports and Culture of Japan and by Yazaki Memorial Foundation for Science and Technology.

References [1] Barrera, A. H.: Computing the Minkowski sum of monotone polygons. Technical Report of the Special Interest Group on Algorithms of the Information Processing Society of Japan, 96-AL-50-9, 1996. 231 [2] Boissonnat, J.-D., de Lange, E., and Teillaud, M.: Minkowski operations for satellite antenna layout. Proceedings of the13th Annual ACM Symposium on Computational Geometry, pp. 67–76, 1997. 231 ´ ements de Mathématique, Algébre 1. Hermann, Paris, 1964. 237 [3] Bourbaki, N.: El´ [4] Farin, G.: Curves and Surfaces for Computer Aided Geometric Design, 4th Edition, Academic Press, Boston, 1999. 238


245

[5] Ghosh, P. K.: A mathematical model for shape description using Minkowski operators. Computer Vision, Graphics and Image Processing, vol. 44, pp. 239–269, 1988. 231 [6] Ghosh, P. K.: A solution of polygon containment, spatial planning, and other related problems using Minkowski operators. Computer Vision, Graphics and Image Processing, vol. 49, pp. 1–35, 1990. 231, 232 [7] Ghosh, P. K.: An algebra of polygons through the notion of negative shapes. CVGIP: Image Understanding, vol. 54, pp. 119–144, 1991. 231, 234 [8] Ghosh, P. K.: Vision, geometry, and Minkowski operators. Contemporary Mathematics, vol. 119, pp. 63–83, 1991. 231 [9] Ghosh, P. K., and Haralick, R. M.: Mathematical morphological operations of boundary-represented geometric objects. Journal of Mathematical Imaging and Vision, vol. 6, pp. 199-222, 1996. 232 [10] Guibas, L. J., Ramshaw, L., and Stolfi, J.: A kinetic framework for computational geometry. Proceedings of the 24th Annual IEEE Symposium on Foundation of Computer Sciences, pp. 100–111, 1983. 231 [11] Guibas, L. J., and Seidel, R.: Computing convolutions by reciprocal search. Discrete and Computational Geometry, vol. 2, pp. 157–193, 1987. 231 [12] Haralick, R. M., Sternbery, S. R., and Zhuang, X.: Image analysis using mathematical morphology. IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. PAMI-9, pp. 532–550, 1987. 231 [13] Har-Peled, S., Chan, T. M., Aronov, B., Halperin, D., and Snoeyink, J.: The complexity of a single face of a Minkowski sum. Proc. of the 7th Canadian Conference on Computational Geometry, pp. 91–96, 1995. 231 [14] Hataguchi, T., and Sugihara, K.: Exact algorithm for Minkowski operators. Proc. of the 2nd Asian Conference on Computer Vision, vol. 3, pp. 392–396, 1995. 231 [15] Kaul, A., O’Connor, M. A., and Srinivasan, V.: Computing Minkowski sums of regular polygons. Proc. of the 3rd Canadian Conf. on Computational Geometry, pp. 74–77, 1991. 231 [16] Kaul, A., and Farouki, R. T.: Computing Minkowski sums of plane curves. International Journal of Computational Geometry and Applications, vol. 5 (1995), pp. 413–432. 231 [17] Kedem, K., Livne, R., Pach, J., and Sharir, M.: On the union of Jordan regions and collision-free translational motion amidst polygonal obstacles. Discrete and Computational Geometry, vol. 1, pp. 59–71, 1986. 231, 232 [18] Leven, D., and Sharir, M.: Planning a purely translational motion for a convex object in two-dimensional space using generalized Voronoi diagram. Discrete and computational Geometry, vol. 2, pp. 9–31, 1987. 231, 232 [19] Lozano-Perez, T., and Wesley, M. A.: An algorithm for planning collision-free paths among polyhedral obstacles. Commun. of the ACM, vol. 22, pp. 560–570, 1979. 231, 232 [20] Mikusi´ nski, J.: Operational Calculus (in Japanese), Shokabo, Tokyo, 1963. 232 [21] Mount, D., and Silverman, R.: Combinatorial and computational aspects of Minkowski decompositions. Contemporary Mathematics, vol. 119, pp. 107–124, 1991. 231 [22] Ramkumar, G. D.: An algorithm to compute the Minkowski sum outer-face of two simple polygons. Proceedings of the12th Annual ACM Symposium on Computational Geometry, pp. 234–241, 1996. 231 [23] Rossignac, J. R., and Requicha, A. A. G.: Offseting operations in solid modeling. Computer Aided Geometric Modeling, vol. 3, pp. 129–148, 1986. 231

246

Kokichi Sugihara

[24] Schmitt, M.: Support function and Minkowski addition of non-convex sets. P. Maragos, R. W. Schafer and M. A. Butt (eds.): Mathematical Morphology and its Applications to Image and Signal Processing, Kluwer Academic Publishers, 1996, pp. 15–22. 232 [25] Schwartz, J. T.: Finding the minimum distance between two convex polygons. Information Processing Letters, vol. 13 pp. 168–170, 1981. 231 [26] Serra, J.: Image Analysis and Mathematical Morphology, 2nd Edition, Academic Press, 1988. 231 [27] Sugihara, K., Imai, T., and Hataguchi, T.: An algebra for slope-monotone closed curves. International Journal of Shape Modeling, vol. 3, pp. 167–183, 1997. 231, 232

Polygon Decomposition Based on the Straight Line Skeleton Mirela T˘ anase and Remco C. Veltkamp Institute of Information and Computing Sciences Utrecht University, Padualaan 14, 3584 CH Utrecht, The Netherlands {mirela,remco.veltkamp}@cs.uu.nl

Abstract. We propose a novel type of decomposition for polygonal shapes. It is thought that, for the task of object recognition, the human visual system uses a part-based representation. Decompositions based on skeletons have been previously proposed in computer vision. Our method is the first one, however, based on the straight line skeleton. Compared to the medial axis, the straight line skeleton has a few advantages: it contains only straight segments and has a lower combinatorial complexity. The skeletal nodes and the way they are generated are the basis for our decomposition, which has two stages that result in a hierarchical decomposition into overlapping parts. First, a number of visually striking parts are identified, then their boundaries are successively simplified, by locally removing detail. Our method runs in time O((n + r22 ) log2 n), after the skeleton construction, where r2 is the number of reflex edge annihilations. The decomposition is invariant to rigid motions and uniform scalings. We present results indicating that it provides natural decompositions for a variety of shapes. This makes it attractive for shape based matching in content-based image retrieval, for example.

1

Introduction

It is thought that, for the task of object recognition, the human visual system uses a part-based representation. In computer vision, many recent approaches to the decomposition problem draw inspiration from the human perception theories. Existing decomposition methods can be classified into those that are boundarybased, using only contour information for extracting parts, and those that are region-based, using information about the interior of the shape. Probably the most known and influential approach in the first category is that proposed by Hoffman and Richards [1]. Their minima rule states that for any silhouette, the negative curvature minima of the contour are boundary points that separate different parts. This rule led to a representation of the shape boundary based on codons, or pieces of boundary bounded by curvature minima. Although the minima rule indicates a precise set of points on the shape boundary that mark the demarcation between parts, it does not indicate how to join these points in order to obtain the actual parts. When Blum introduced the medial axis [2] back in 1967, a decomposition at its branching points was also suggested. The medial axis is region-based and can T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 247–268, 2003. c Springer-Verlag Berlin Heidelberg 2003

248

Mirela T˘ anase and Remco C. Veltkamp

be defined as the locus of centers of maximally inscribable disks. Variations of it include smoothed local symmetries [3] and, for polygonal shapes, the straight line skeleton [4]. The medial axis is capable of capturing important visual cues of the shape, such as symmetry and complexity (reflected by branching points). It also allows recovery of the original shape and provides rich local support (local boundary modifications affect only a small portion of the skeleton). These make it attractive for shape description. However, one serious disadvantage of it is its sensitivity to noise. Moreover, its computation on raster data also poses severe problems: it often leads to non-connected axes and it may require extensive preprocessing (smoothing of the boundary) or post-processing (pruning of spurious edges). Some of these problems can be overcome by a computation based on Voronoi diagrams of the boundary points, together with a hierarchical clustering of the skeleton components [5]. Along with the medial axis branching points, local minima of the maximally inscribed disk radii have been suggested as useful for a decomposition into parts. A different approach [6] to using skeletons for shape decomposition, associates a weighted graph, called axial shape graph, to the medial axis. Weights are capturing both local and global information about the shape. The decomposition is derived by repeatedly partitioning the axial shape graph into two sets of nodes of roughly equal weights. Siddiqi and Kimia [7] proposed a way of combining extremal curvature boundary points with region information, since using only boundary information seems insufficient. Their decomposition involves two types of parts, limbbased and neck-based. Limb-based parts are the result of a partitioning through a pair of negative curvature minima with a smooth continuation of their boundary tangents. Neck-based parts, on the other hand, arise from narrowings in the shape, characterized by local minima of the maximally inscribed disk radii. Limbs and necks are also related to singularities (shocks) in a curve evolution process, based on a reaction-diffusion equation, introduced by Kimia, Tannenbaum and Zucker [8]. A contour evolution under constant deformation (contour points move along the normal with constant speed) is equivalent to Blum’s grassfire interpretation of the medial axis. The locus of shocks formed under constant deformation of the boundary is thus the medial axis. The formation of the shocks however provides additional information to the axis, and this is exploited in an organization of shocks into a shock-graph [9]. A discrete contour evolution is the base for the hierarchical decomposition method proposed by Latecki and Lak¨ amper [10]. They propose a simple contour simplification process: at every step, the two consecutive line segments with the least relevance, as defined by some relevance measure, are substituted with a single line segment joining their endpoints. This process yields a hierarchy of simplifications, which are used next in the decomposition, starting from the highest level in the hierarchy (corresponding to the simplest shape in the evolution) down to the lowest level (corresponding to the original shape). The parts of the contour at a given level of the shape hierarchy are determined by the maximal convex arcs with respect to the object.

Polygon Decomposition Based on the Straight Line Skeleton

249

Polygon decomposition is a well-established research area in Computational Geometry. There is a substantial body of literature that focuses on developing efficient algorithms for partitioning or covering a polygon with the smallest number of a particular type of subpolygons (triangles, convex, monotone, spiral or star-shaped subpolygons), see [11] for a survey. The emphasis on the optimality of the decomposition comes from the fact that polygon decomposition often serves as a preprocessing step for many geometric algorithms. Many geometric problems have simpler and faster solutions on such a restricted type of polygon, so the strategy of solving these problems for general polygons is to decompose them into simpler parts, solve them on each part and combine the partial solutions. For polygon recognition purposes, however, this may not be a desirable property of the decomposition. Moreover, the results of a decomposition into components of a fixed type, convex polygons included, do often correlate poorly with the human judgement. 1.1

Our Contribution

We propose here a novel type of decomposition for polygonal shapes. It is based on the straight line skeleton, a variant of the medial axis, introduced by Aichholzer and Aurenhammer [4]. The general advantages of the straight line skeleton, as compared to the medial axis, are its composition of straight segments only and its lower combinatorial complexity. Similar to the grass-fire analogy of the medial axis, the straight line skeleton is defined by a wavefront propagation process. In this process, edges of the polygon move inward at a fixed rate. The counterparts of the medial axis branching points, called nodes, are induced by combinatorial changes of the wavefront (events). The nodes and the way they are generated are central to our decomposition method. The wavefront events associated with the nodes endows them with additional information, in the same way as shocks add a coloring to the medial axis points in shock graphs [9]. We exploit this information in deciding how the node contributes to our decomposition (globally, locally or not at all). This results in a 2-step decomposition. Nodes corresponding to splittings of the propagating wavefront lead to a decomposition into non-overlapping parts. These are usually the most striking parts of the polygon. Nodes associated with edge events lead to a local simplification of the boundary of these parts, together with a removal of smaller protrusions. The successive boundary simplifications constitute the second step of our decomposition. The final output is a hierarchical decomposition into overlapping parts. In both stages, the actual computation of the parts is based on a backward propagation process, similar to the inward propagation that generates the skeleton. The decomposition is invariant to rigid motions and uniform scalings. It can be computed in O((n+r22 ) log2 n) time, after the straight line skeleton computation, where n is the number of vertices in the polygon, and r2 is the number of reflex edge annihilations in the skeleton construction. We present results indicating that it provides intuitive decompositions. The rest of this paper is organized as follows. In section 2 we present the straight line skeleton. Specifically, we discuss its relation with the medial axis,

250


give a complete categorization of the events occurring in the wavefront propagation and its properties. We detail the proposed decomposition method in section 3 and analyze its computational complexity in section 4. Examples of the implementation results are presented in section 5, followed by some concluding remarks in section 6.

2

Straight Line Skeleton

Aichholzer and Aurenhammer [4] introduced the straight line skeleton, a new type of skeleton for polygons. It is closely related to the medial axis, being also defined based on a wavefront propagation. In the grass-fire analogy of the medial axis, a wavefront is defined as the set of points having some fixed distance to the original polygon. It consists of straight line segments and circular arcs (see figure 1(a)) and, as the distance to the polygon increases, the breakpoints between consecutive line segments and circular arcs trace the Voronoi diagram of the polygon (a process also called wavefront propagation). By removing the segments in the diagram incident to the reflex vertices, we obtain the medial axis. To construct the straight line skeleton, we also let wavefront edges move parallel to the polygon sides; edges incident to a reflex vertex will grow in length. The front remains a polygon, whose vertices during the process trace out the skeleton (see figure 1(b)). As its name suggests, it consists of straight line segments only, as opposed to the medial axis which also contains parabolic arcs. It also has a smaller combinatorial complexity than the medial axis. In this section we take a closer look at the straight line skeleton definition, identifying all the possible morphological changes in the wavefront, as these events are of prime importance for our decomposition. We also review the known properties of the straight line skeleton and extend a previous result.

(a)

(b)

Fig. 1. Medial Axis (a) vs. Straight Line Skeleton (b). In (b) the black disk marks a reflex edge annihilation, while gray disks mark convex edge annihilations. An edge-edge collision generates the arc between the black box (vertexedge collision) and a gray disk (convex edge annihilation)


2.1

251

Preliminaries

Let P be a simple polygon, with a counterclockwise order defined on its edges. The orientation of the polygon is of no consequence for our decomposition, but simply a convention for ease in proving its properties. If ei is an edge of P , let lei denote its supporting line. The bisector Bij of ei and ej is the half-line angular bisector of lei and lej lying to the left of both ei and ej . As we do not assume the polygon to be in general position, we need to define two degenerate bisectors as in figure 2. For two disjoint edges, ei and ej , having the same orientation and supporting line, Bij is the ray perpendicular to lei , originating in a point of lei , equally distanced to ei and ej and to the left of both (see figure 2, left). If ei and ej have opposite orientation and parallel supporting lines, Bij is the line parallel to lei and lej , and equally distanced to them (see figure 2, right). We call an edge whose endpoints are convex vertices of P , a convex edge. An edge with at least one endpoint being a reflex vertex of P , will be called a reflex edge. The following propagation process is defined on P : edges are translated in a self parallel manner and at constant speed, keeping sharp corners at reflex vertices (see figure 1(b)). As we already mentioned, such a shrinked version of P is called a wavefront. During propagation, the wavefront can split, after which it is a union of simple, disjoint polygons. Wavefront vertices move, in the propagation, on angular bisectors of wavefront edges, which in turn may increase or decrease in length. If x is a point inside P , we denote by F (x) the wavefront that has x on its boundary. Definition 1. The straight line skeleton of P , S(P ), is defined as the union of pieces of angular bisectors traced out by the wavefront vertices during the above propagation process. These pieces of bisectors are called arcs. The endpoints of the arcs in S(P ) that are not vertices of P are called nodes. 2.2

Event Taxonomy

In the propagation, the wavefront repeatedly changes continuously until events occur. The following is a complete categorization of these events: ej Bij

ei

Bij

ej

Fig. 2. Degenerate bisectors

ei

252


1. Edge Events. A wavefront edge may collapse into a point, its neighbouring edges becoming adjacent. We distinguish: (a) Convex Edge Annihilation. If the collapsing edge is a convex edge of P . (b) Reflex Edge Annihilation. If the collapsing edge is a reflex edge of P . In this case, also a reflex vertex disappears from the front. See figure 1(b) for examples of convex and reflex edge annihilations. (c) Front Annihilation. An entire simple polygon of the wavefront may collapse into a point (equivalent with at least three edge annihilations at the same location), into a line segment, or into two or more line segments. (See figure 3(b)). 2. Split Events. A simple polygon of the wavefront may be split into a number of polygons following one or a combination of the following: (a) Vertex-Edge Collision. A reflex vertex of the front may collide into an edge thus splitting the front into two (see figure 3(a)). (b) Vertex-Vertex Collision. Two vertices (at most one being convex) may meet at the same location, splitting the front into two. See figure 3(a), where the event, marked with a gray box, is generated by two reflex vertices that reach the same point (the event) simultaneously. Unlike edge annihilations or vertex-edge collisions, this kind of event can introduce a new reflex vertex to the wavefront. (c) Edge-Edge Collision. Two parallel front edges may collapse into each other. This can give rise to a front splitting at one or both endpoints of the arc in the S(P ) generated by the event. See figure 1(b) where such an event generates a splitting of the front into a simple polygon and a degenerate rectangle. Note that in [4] a split event is only what we call a vertex-edge collision. In [12] a vertex-vertex collision between two reflex vertices, is called a vertex event. Vertex-vertex collisions involving a convex vertex and edge-edge collisions do not

(a)

(b)

Fig. 3. (a) Events in the wavefront propagation: gray disks mark convex edge annihilations, the black box marks a vertex-edge collision, while the gray box is a vertex-vertex collision. (b) Front annihilation: the front collapses down to two line segments


253

appear in their classification. The reason for this is that a small perturbation of the polygon removes these types of events from the skeleton without altering its structure. Vertex-vertex collisions between two reflex vertices, on the other hand, cannot be removed by perturbations of the original polygon without changing the structure of the skeleton. 2.3

Properties of the Straight Line Skeleton

We know [4] that, for a simple polygon with n vertices, S(P ) has at most n − 2 nodes and 2n − 3 arcs, and both bounds are smaller than those for the medial axis (n + r − 2 and 2n + r − 3, respectively, where r is the number of reflex vertices). S(P ) gives rise to a polygonal partition of the interior of P . Each cell in this partition is the region swept out during the propagation by one of the edges of P . For the rest of this paper, we say that an edge e is involved in an event, if the node introduced in S(P ) by that event is on the boundary of the cell swept out by e. We denote the cell swept out by ei by C(ei ). Aichholzer and Aurenhammer [4] observed that the cells of S(P ) are monotone polygons. We extend this result by showing that for edges not involved in vertex-vertex collisions, the corresponding cells are pseudo-normal histograms. If H = {q0 , q1 , . . . , qn+1 } is a simple polygon in counterclockwise order, and (we can assume without loss of generality) the edge e = qn+1 q0 is horizontal, then H is called a normal histogram iff: x , i = 0 . . . n; 1) qix ≥ qi+1 y y 2) qi > q0 , i = 1 . . . n. If the first condition is weakened to q1 x ≥ q2 x ≥ . . . ≥ qn x , thus allowing x q0 < q1 x and/or qn+1 x > qn x , we call H a pseudo-normal histogram (see figure 4, right). The following lemma is used in proving the bounds on the number of Steiner points added to the decomposition in the first phase of the proposed method. Lemma 1. If an edge of P is not involved in any vertex-vertex collision, then its corresponding cell is a pseudo-normal histogram. Proof. Let e be such an edge of P and C(e) its corresponding cell. Let C(e) = {q0 , q1 , ..., qn+1 }, with e = qn+1 q0 . We can assume without loss of generality qn qn

q1

q1

qn+1

q0

qn+1

q0

Fig. 4. A normal histogram (left) and a pseudo-normal histogram (right)

254


x that e is horizontal and q0x > qn+1 . The first observation is that no reflex vertex, incident to the wavefront edge corresponding to e, can appear in the wavefront propagation, because new reflex vertices may appear in the wavefront only following a vertex-vertex collision. We have to show that q1 x ≥ q2 x ≥ . . . ≥ qn x . If there exists i ∈ {1, . . . , n − 1} so that qi x < qi+1 x , then let e be the edge in P so that qi qi+1 is a piece of the bisector of e and e . This implies that at the moment that the front reaches qi , the wavefront edges corresponding to e and e become adjacent and form a reflex vertex. This contradicts the above observation, so C(e) must be a pseudo-normal histogram.

3

Proposed Decomposition

We saw in the previous section that in the propagation process, the wavefront suffers two types of modifications: it can be split into two or more parts and edges of the front can disappear. The first type of modifications are generated by split events. They give information about the number and the configuration of the most striking parts of the polygon. The second type of modifications are generated by edge events. Through these events the boundary of the shape is simplified by removing the edges roughly in an increasing significance order. A special type of edge event is the reflex edge annihilation, because it corresponds to a protrusion of the shape. All events are related to changes in the shape of the wavefront during propagation. But while split events are induced by drastic modifications of the wavefront, reflex edge annihilations are caused by smaller protrusions of the polygon, that annihilate ’smoothly’ in the propagation. The events generated in the propagation process capture thus two different types of information about the shape. It is natural then to use the split events first in order to identify the most striking parts, and then use the reflex edge annihilations to simplify those parts, by removing detail (protrusions). Our decomposition process has these two stages.

Fig. 5. Straight line skeleton and a decomposition induced by its split and reflex edge events


255

The basic idea for computing the parts in our decomposition is to reverse the process that caused the events in the first place, by back-propagating the wavefront at the moment that any of these events occur. As explained in detail below, each stage of the decomposition process handles the events sequentially, in the increasing chronological order of their occurrence. In the first stage, each split event induces a division of one of the parts in the current decomposition. In the second stage, reflex edge annihilations induce simplifications of the parts computed in the first stage, removing some of their smaller details (protrusions). In figure 5, we see a very simple example of such a decomposition. The vertex-edge collision, marked with a square box, is a split event that induces a decomposition of the polygon into two parts. Next, the protrusion in the larger right part is removed with the handling of the reflex edge annihilation marked with a disk in the figure. In the rest of this section we concentrate on the computation of the proposed decomposition. 3.1

Phase 1: Global Splitting

The handling of all split events results in a decomposition of P into non-overlapping parts. As we already mentioned, the split events are treated in the order that they occur in the inward propagation process. Degenerate cases (simultaneous split events at the same location) can be treated in arbitrary order. A split-chain is a polygonal chain with both endpoints on the boundary of a part p in the current decomposition of P and all the other points inside p, that divides p into two components. Each split event induces such a split-chain that will divide one of the parts in the previously computed decomposition into two. In our decomposition, we construct the split-chain in a back-propagation process, similar to the one that leads to the straight line skeleton. We prove that any split-chain consists of at most three segments, with one or two new Steiner points being added to the decomposition. Let x be a point inside P so that when the wavefront reaches x it is split. The wavefront at the moment x is reached, F (x), contains a union of two simple polygons, with disjoint interiors and having x as a common vertex. We propagate backwards these polygons simultaneously and at the same speed until the edges of F (x) adjacent to x reach their counterparts on the original polygon. The split-chain induced by x is the trace of the intersection of the back-propagating polygons. Before detailing the computation of the split-chain for each type of split event, we make a few observations. First, the inward propagation process defined in Section 2.1 is not a reversible one. The result of the backward propagation process, a polygon P , is usually not the original polygon P , but a simplified version of it. Edges of P lost in edge events during propagation are not in P . Secondly, an edge-edge collision is handled by handling its endpoints. If an edgeedge collision splits the wavefront at one of its endpoints then this endpoint is handled as a vertex-edge collision, otherwise an edge of the wavefront must also collapse into that endpoint, so the endpoint is treated as an edge event.

256

Mirela T˘ anase and Remco C. Veltkamp e2 e1 f1

f2 e3

f3

e4

f4

b12

e2

f2

e1 b34

b12 x

f1

f3

e3

x b34

f4

e4

y

Fig. 6. Vertex-vertex collisions and the split-chains they generate

Handling a Vertex - Vertex Collision. Let x be a vertex-vertex collision: two vertices of the front and nothing else meet at x, and split the front. In figure 6, at the moment that x is reached, the wavefront F (x) is split into two polygons. They share a vertex (x) in F (x), f1 and f4 are adjacent in one polygon, while f2 and f3 are adjacent in the other. The split-chain is defined as the trace of the intersection of these two polygons in their propagation backwards to the original polygon. The growth of the edges in the back-propagation is restricted by the interaction with each other, in the same way as in the inward propagation defining the skeleton. Let p be the part in the current decomposition of P that has x in its interior. This is the part that will be divided through a split-chain in the handling of the event. We now prove a bound on the length of the split-chain induced by a vertex-vertex collision. The bound on the length of the part-chain, in this case, is given by the following lemma. Lemma 2. The split-chain induced by a vertex-vertex collision consists of two line segments, introducing one Steiner point (the event itself ) to the decomposition. Proof. If x is our vertex-vertex collision, let fi , i = 1 . . . 4, be the edges of F (x) incident to x so that, prior to the event, f1 was incident to f2 and f3 to f4 , respectively. Let ei , i = 1 . . . 4, be their counterparts on P . Shortly before reaching x, the two front vertices that caused the event were moving along arcs of S(P ). Let b12 be the arc of S(P ), piece of the bisector B12 of e1 and e2 , and b34 be the arc of S(P ), piece of the bisector B34 (see figure 6). One of these arcs (b12 , for example) can have length zero, when the vertex-vertex collision coincides with the moment that the front edges corresponding to e1 and e2 become incident. In other words, when x is also an edge event. We distinguish two cases: – Reflex-Reflex Collision. If the vertices that caused the event were both reflex (see figure 6, left), b12 has one endpoint at x, and one on P , or at a vertexvertex collision, wherein the wavefront edges corresponding to e1 and e2 became adjacent. In either case this endpoint of b12 is on the boundary of


257

e2 e1

e2

e1

u b12

b12

e

x y

u x

e

y e3

e3

Fig. 7. The reflex vertex generating a vertex-edge collision at x can originate in a vertex of P (left) or may have appeared in the propagation process (right). In both cases, the split-chain contains on this side of x only one segment (xu = b12 ). On the other side of x, the split-chain contains one segment (xy) also, but only when y, the projection of x onto the edge that is split, falls inside it a part in the current decomposition. The same holds for b34 . So the splitchain in this case is included in b12 ∪ b34 . – Reflex-Convex Collision. If one of the vertices is convex, assume this is the one that moved previously along b34 (see figure 6, right). Based on the above mentioned argument, b12 has its endpoint other than x on the boundary of a part in the current decomposition. The other endpoint of b34 , though, may or may not lie inside p. The split-chain induced by x in this case is included in b12 ∪ B34 . In figure 6 right, the split-chain is given by b12 ∪ xy, where y is the intersection of the bisector B34 with the boundary of p. So in both cases the split-chain consists of two segments.

Handling a Vertex - Edge Collision. Let x be a vertex-edge collision: at the moment that the wavefront reaches x, x is a reflex vertex of F (x) that collides with and splits a wavefront edge. As in the previous case, the split-chain is the trace of the intersection of two polygons in F (x) in their back-propagation towards P . Lemma 3. The split-chain induced by a vertex-edge collision consists of two or three line segments, introducing one or two Steiner points to the decomposition. Proof. If x is a vertex-edge collision, let e1 and e2 be the edges of P that propagated and gave rise to the reflex vertex x in F (x). If e denotes the wavefront edge that is split in the event, let e3 be its counterpart on P . Finally, let p be the part in the current decomposition that has x in its interior. 1. Trace the intersection point of the fronts backwards to the vertex. On this side of x, that intersection point moves along the arc b12 in S(P ), piece of the

258


bisector B12 of e1 and e2 . This has one endpoint in x, the other lies on P , or is a vertex-vertex collision prior to x in the propagation (see figure 7). In both cases it is on the boundary of a part in the current decomposition. So the intersection trace contains on this side of x at most one segment. 2. Trace the intersection of the polygons backwards to the edge. On this side of x, the intersection is traced along the line perpendicular to the supporting line le3 of e3 . If the projection of x on le3 falls inside e3 (see figure 7), there are no morphological changes in the back-propagating front along the intersection trace, so the split-chain on this side contains only one segment. Let’s suppose now that the projection of x on le3 falls outside e3 (see figure 8). Because x ∈ e, which originates in e3 , x lies inside the cell C(e3 ) in the subdivision of P induced by S(P ). It follows that in projecting x on le3 we meet first the boundary of C(e3 ). It also follows that the closest endpoint of e3 to the above projection must be a reflex vertex w. Let B denote the arc of S(P ) incident to w. If e3 is not involved in any vertex-vertex collision previous to x (see figure 8 left), from Lemma 1 we have that in projecting x on le3 , we must intersect B. If we denote this intersection by y, it’s interpretation is the following: one of the pieces of e shrinks to y in the back-propagation, and the other piece becomes adjacent with the other edge of P incident to w. This way, if y is not already on the boundary of p, our intersection is further traced along B towards w. In this case, the trace on this side of x contains two segments sharing y as endpoint. If e3 is involved in at least one vertex-vertex collision, which must be previous to x (see figure 8 right), then in projecting x onto le3 we may meet any of the segments on the boundary of C(e3 ) between w and x. The intersection trace again may contain at most two segments, as all the vertex-vertex collisions on the boundary of C(e3 ) between w and the last vertex-vertex collision (z in figure 8 right) prior to x were handled, so all these segments in C(e3 ) between w and z are edges of parts in our decomposition.

e3

e3

e

y

e1 e2

u

x

B

e

w

w

y

z x

e2 u

e1

Fig. 8. The split-chain may contain, on the side of the edge that is split (e3 ), two segments (xy and yw on the left, xy and yz on the right). This may happen only when the projection of the event x onto le3 falls outside e3


3.2

259

Phase 2: Locally Removing Detail

A decomposition of P into r1 non-overlapping parts is the outcome of the first phase of the algorithm, where r1 is the number of split events in S(P ). Let Pi , i = 1, . . . , r1 , denote these parts. We see three examples of such decompositions induced by split events in the left column of figure 11. We can further simplify some of the parts in these decompositions. Reflex edge annihilations, we said, are related to protrusions of the shape. In the second step of the algorithm, these events are handled in the increasing chronological order of their occurrence, in order to locally simplify the boundary by removing detail (protrusions). We use for this purpose the events in the straight line skeleton of the original polygon. A recomputation of the skeletons for Pi would induce new splittings that are not significant for the original polygon. In the rest of this section, we give a simple method for locally simplifying the boundary of the original polygon based on reflex edge annihilations. Local Boundary Simplification Let x be a reflex edge annihilation: a wavefront edge incident to a reflex vertex collapses down to x, its neighbouring edges becoming adjacent and forming a convex angle in F (x) (see figure 9). Let f1 and f2 (in this order along the wavefront) denote the edges in F (x) incident to x, and e1 , and e2 their counterparts on P . We also denote by e the edge in P collapsing in the event. On P , e1 and e may not be adjacent, but then the edges in between e1 and e collapse before e during the wavefront propagation. The same holds for e and e2 . We call the chain P(x) = {e1 , . . . , e2 } a protrusion-chain. Our purpose is to find a simplification of this chain that cuts the protrusion off. A protrusion-cut is a segment with its endpoints resting on the extremal edges, e1 and e2 , of the protrusion-chain. Each reflex edge annihilation induces, in our decomposition, such a protrusion-cut that locally simplifies the boundary by removing detail (the protrusion). In figure 9, we have a simple example of a boundary simplification. The protrusion-chain here consists of only e e1 u

f1

v x

e2

f2

Fig. 9. A reflex edge annihilation (x) and the protrusion-cut it generates (the dashed line)

260


3 edges (e1 , e and e2 ), and the dashed line shows how its corresponding protrusion is separated from the part. We now describe a way of computing such a protrusion-cut, which is also based on a back-propagation. We consider this process only in the close vicinity of the event x, as we are interested in simplifying only a portion of the boundary. It can be shortly described as follows. We back-propagate the two wavefront edges in F (x) incident to x until we hit both their counterparts on P . Among the points of e1 and e2 that were reached by the back-propagating chain we select the two points, u ∈ e1 and v ∈ e2 , that are closest along P(x). They define the endpoints of the protrusion-cut segment induced by x. The back-propagation of these two edges is done in the same way as the inward propagation that defines the straight line skeleton, their growing, however, is restricted by the intersection with P . Their intersection point moves along B12 , the bisector of e1 and e2 , towards its origin {q} = le1 ∩ le2 . We will show that defined in this way, the computation of the protrusion-cut is very simple, requiring little information, namely only the relative position of e1 and e2 with respect to q. We first note that the areas swept in the inward propagation by e1 and e2 , before x is reached, are disjoint. As a consequence, only the following configurations can occur (see figure 10): 1. 2. 3. 4. 5.

q q q q q

∈ e1 and e2 is to the ∈ e2 and e1 is to the / e2 , e1 is to ∈ / e1 , q ∈ ∈ / e1 , q ∈ / e2 , e1 is to / e2 , e1 is to ∈ / e1 , q ∈

left of le1 (as defined by the orientation of e1 ); left of le2 ; the left of le2 and e2 is to the left of le1 ; the left of le2 and e2 is to the right of le1 ; the right of le2 , and e2 is to the left of le1 .

When back-propagating f1 and f2 in the way defined above, one of the following cases can occur: – e1 and e2 are hit simultaneously. This happens in cases 1-3 from above. If q ∈ / e1 and q ∈ / e2 (case 3), all of their points are reached by the backpropagating chain (see figure 10 (3)). The protrusion-cut is then the segment between the end point of e1 and the start point of e2 , as these are closest along P(x). In the other two cases, there is a part of the edge containing q not reached by the back-propagating chain, all points on the other edge being reached. The protrusion-cut in these cases will connect q with the closest point along P(x) on the edge not containing q (see figure 10 (1) and (2)). – e1 and e2 are not hit simultaneously. This happens in cases 4 and 5 from above. Let e1 be the one that is hit first (case 4, see figure 10 (4)), all of its points being reached. We continue the back-propagation, inside P , until the target v of e2 is hit. The point on e1 closest to v along P(x), is then the end point u of e1 and uv is then the protrusion-cut segment.

Hierarchical Decomposition into Overlapping Parts In the previous subsection, we described a method for locally simplifying the boundary of P , in order to remove smaller protrusions that give rise to reflex edge annihilations.


e1 q

q

v

f2

e2

x

e2

(1)

u

f1

u

B12

x

e1

e1

f1

q v

e2

f2

(2)

B12

f2

(3)

f1

u v q

f1 x

B12

e1 e2

261

f1 x

B12

f2 (4)

e1

q u

x v

e2

B12

f2

(5)

Fig. 10. There are only 5 possible configurations of e1 and e2 with respect to {q} = le1 ∩ le2 , that propagated become adjacent in a reflex edge annihilation x. The protrusion-cut induced by the event, in each case, is the dashed segment We now want to relate these simplifications with the parts Pi , i = 1 . . . r1 , resulted after splitting. A protrusion-cut segment often has both endpoints on the boundary of the Pi that contains the event generating it. In this case, it can be viewed as a simplification of Pi ’s own boundary. With a simple rule like this, we shift the boundary simplification, defined by a reflex edge annihilation x, from P to the part Pi containing x. Handling the reflex edge annihilations in the order of their occurrence leads to a hierarchy of parts. All such events inside a part Pi induce a series of simplifications of Pi , along with the possible separation of some smaller parts. We can then associate with the decomposition a tree structure. The root of the tree corresponds to the original polygon and its children correspond to the parts Pi resulted from the splitting. The subtree rooted at the node corresponding to Pi comes from the removing of detail phase of the algorithm. Each level is the result of the handling of a reflex edge annihilation inside Pi . The nodes in the level correspond to a simplification of Pi , together with the smaller parts the protrusion-cut removes from Pi . The decomposition leads to overlapping parts. A simplification of Pi may overlap protrusions previously computed or other parts resulted from split-

262


ting. Secondly, a protrusion-cut may remove more smaller parts, because the protrusion-cut segment may intersects edges on the protrusion-chain that collapse previously to the current event.

4

Time Complexity

The polygonal decomposition we described in the previous section requires the computation of the straight line skeleton as a preprocessing step. Despite its similarity with the medial axis, which can be computed in linear time [13], the fastest known algorithms for the straight line skeleton are slower. The first subquadratic algorithm was proposed by Eppstein and Erickson [12]. It runs in O(n1+ + n8/11+ r9/11+ ) time with a similar space complexity, where r is the number of reflex vertices and is an arbitrarily small positive constant. A more recent algorithm by Cheng and Vigneron [14] runs in O(n log2 n + r17/11+ ). The simpler and more practical algorithms given in [4] and [16] run in O(n2 log n) and O(nr + n log n), respectively. The basic approach of [12], [4] and [16] is to simulate the sequence of split and edge events that define the skeleton. With no additional computational complexity we can modify these algorithms in order to retain the information required by our decomposition process. So, for the rest of this section, we assume that, after the preprocessing, along with the straight line skeleton, we have a list of split events and a list of reflex edge annihilations, ordered according to their occurrence in time. Also, for each event we have pointers to the edges involved in the event. In the first phase of our decomposition, split events are handled in chronological order of their occurrence, the outcome being a decomposition into nonoverlapping parts. In the handling of a split event, a part in the current decomposition of P is divided by a chain (split-chain) of at most three segments. The first difficulty here is finding the endpoints of this split-chain, as defined in section 3.1, on the current part. The naive approach (checking the split-chain for intersection with the current part boundary) leads, over all split events, to O(r1 (n− r1 )) time in the worst case, where n is the total number of vertices in P and r1 is the number of the reflex ones. From a computational geometry point of view, however, this is a ray-shooting problem in a dynamic planar subdivision. A solution for this problem is found in the dynamic data structure introduced by Goodrich and Tamassia [15]. It uses linear space and supports ray-shooting queries and updates, like vertex and edge insertions and deletions, in O(log2 N ) time, where N is the current complexity of the subdivision. Point locations can also be performed in O(log2 N ) time, and this solves the second difficulty of this step, which is identifying the part that gets divided by the event. This is the part that has the event in its interior. In conclusion, the first step of the decomposition takes O(r1 log2 n) time, after the construction of the above mentioned ray shooting data structure for P in O(n log2 n) time.


263

The handling of the reflex edge annihilations in the second step leads to a hierarchy of simplifications of Pi , i = 1, . . . , r1 , the parts resulted from splitting. When handling such an event x, the part Pj that has x in its interior is further simplified, if the protrusion-cut segment has both its endpoints on Pj . Identifying the part Pj takes O(log2 n) time using the above mentioned data structure for r1 i=1 Pi . The construction of the protrusion-cut segment takes constant time, but we have to check its intersection with the protrusion-chain. This is again a ray shooting problem that can be solved, over all reflex edge annihilations, in r22 log2 n time, where r2 is the number of such events in S(P ). Theorem 1. For a simple polygon with n vertices, r1 split events and r2 reflex edge annihilations in its straight line skeleton construction, the proposed decomposition can be computed in O((n + r22 ) log2 n) time, after O(n1+ + n8/11+ r9/11+ ) preprocessing time.

5

Results

We have implemented the method described in section 3 and present in this section a few results. We opted for a simple implementation. That is, we avoided theoretically efficient solutions when they have the disadvantage of using complicated data structures. For the straight line skeleton computation, a simple, straightforward method [16] was preferred to the faster algorithm in [12]. Rayshooting in dynamic subdivisions, as introduced by Goodrich and Tamassia [15], requires the maintenance of balanced decompositions of simple polygons via geodesic triangles. Instead, we used the arrangement package of CGAL, the Computational Geometry Algorithms Library [17]. The examples we present in figure 11 come from the SQUID database [18] which contains 1100 images of contours of marine animals. After thinning these contours to one pixel width, a polygonal contour is extracted from the image. In this contour, each pixel corresponds to a vertex. To decrease the number of vertices, we applied the Douglas-Peucker [19] polygon approximation algorithm. Figure 11 illustrates the results of the proposed decomposition technique for three such contours. The output of the first stage (global splitting) in the decomposition is on the left. The parts in this non-overlapping decomposition are further locally simplified in the second stage. These successive local simplifications, are shown on the right. In figure 12, a comparison with the recent method of Latecki and Lak¨ amper [10] is provided. Their decomposition is based on a discrete evolution of the contour. Boundary points with a small relevance measure are iteratively removed from the contour. This induces a hierarchy of shape simplifications, that serves next as a base for the decomposition. Maximally convex arcs of the shapes in the hierarchy define parts on the original contour. These parts are simply cut by joining the endpoints of the maximal convex arcs. It is unclear from [10], however, how the selection of levels in the hierarchy is done. We can’t associate each iteration in the contour simplification with a hierarchy level, even for the

264


Fig. 11. Examples of the proposed decomposition: the parts resulted after the global splitting (left) are further locally simplified (right), by means of protrusion-cut segments. The boundary of the shape is drawn in thick lines, the split-chains in thin solid lines and the protrusions cut segments in dotted lines


265

contours in figure 12, because this would lead to a very fragmented decomposition. The way the hierarchy levels are selected among the great number of iterations in the simplification step influences greatly the resulting decomposition. A second drawback of their method comes from the fact that the proposed contour evolution may lead to self-intersections. Finally, only boundary information is used for decomposition. The contour evolution is intended to capture region information, but it often fails. Parts like the kangaroo foot and the tails of the rabbit and the donkey cannot be properly partitioned in any hierarchy selection, since only contour convexity information is not sufficient here. The counter-intuitive partitioning of the rabbit’s head, in their decomposition, has other causes. A different simplification hierarchy could lead here to a different partitioning. Our method is region-based and, as can be seen also from figure 11, the first stage of the decomposition is powerful enough to extract the main visual parts of the shape. The decompositions appear natural for a variety of biological shapes.

6

Concluding Remarks

We presented a novel type of decomposition for polygons. Decompositions based on various types of skeletons have been previously proposed in computer vision. This is the first one based on the straight line skeleton. The skeletal nodes and the way they are generated are the basis for our method. The wavefront propagation events associated with the skeletal nodes indicate not only where the partitioning should be done, but also how it should be done (globally, locally or not at all). The first step in our decomposition was motivated by the observation that split events are usually related with the most perceptually significant parts of the contour. This is consistent with the results obtained from the implementation. Our method is invariant to rigid motions and uniform scalings. The decomposition can be computed in O((n + r22 ) log2 n) time, after the straight line skeleton computation, where n is the number of vertices in the polygon, and r2 is the number of reflex edge annihilations in the skeleton construction. We have presented results that show that it provides natural decompositions for a variety of shapes. This makes it attractive for shape based matching in content-based retrieval, and it is our intention to use it for this purpose. After decomposing all the shapes in the database, it is easy to search for particular shapes based on only a part of the shape.

Acknowledgements We thank Petr Felkel for kindly providing the code for the straight line skeleton computation [16].

266


Fig. 12. A comparison of our decomposition (left) with that proposed by Latecki and Lak¨ amper (right). The contours and the decomposition are taken from their paper [10]


267

References [1] Hoffman, D., Richards, W.: Parts of Recognition. Cognition 18 (1984) 65–96 247 [2] Blum, H.: A Transformation for Extracting New Descriptors of Shape. Symposium Models for Speech and Visual Form ed: W. Wathen-Dunn MIT Press (1967) 362–381 247 [3] Brady, M., Asada, H.: Smoothed Local Symmetries and their Implementation. The International Journal of Robotics Research 3(3) (1984) 36–61 248 [4] Aichholzer, O., Aurenhammer, F.: Straight Skeletons for General Polygonal Figures in the Plane. In: Proc. 2nd International Computing and Combinatorics Conference COCOON ’96. Lecture Notes in Computer Science, Vol. 1090. Springer-Verlag (1996) 117–126 248, 249, 250, 252, 253, 262 [5] Ogniewicz, R. L., Kubler, O.: Hierarchic Voronoi Skeletons. Pattern Recognition 28(3) (1995) 343–359 248 [6] Simmons, M., Séquin,C. H.: 2D Shape Decomposition and the Automatic Generation of Hierarchical Representations. International Journal of Shape Modeling 4 (1998) 63–78 248 [7] Siddiqi, K., Kimia, B.: Parts of Visual Form: Computational Aspects. IEEE Transactions on Pattern Analysis and Machine Intelligence 17(3) (1995) 239– 251 248 [8] Kimia, B. B., Tannenbaum, A. R., Zucker, S. W.: Shapes, Shocks, and Deformations I: The Components of Two-Dimensional Shape and the Reaction-Diffusion Space. International Journal of Computer Vision 15(3) (1995) 189–224 248 [9] Siddiqi, K., Shokoufandeh, A., Dickinson, S. J., Zucker, S. W. Shock Graphs and Shape Matching. International Journal of Computer Vision 35(1) (1999) 13–32 248, 249 [10] Latecki, L. J., Lak¨ amper, R.: Convexity Rule for Shape Decomposition Based on Discrete Contour Evolution. Computer Vision and Image Understanding 73(3) (1999) 441–454 248, 263, 266 [11] Keil, J. M.: Polygon Decomposition In: J.-R. Sack and J. Urrutia, editors, Handbook of Computational Geometry. Elsevier Science Publishers B. V. NorthHolland, Amsterdam, 1999 491–518 249 [12] Eppstein, D., Erickson, J.: Raising Roofs, Crashing Cycles, and playing Pool: Applications of a Data Structure for Finding Pairwise Interactions. Discrete and Computational Geometry 22(4) (1999) 569–592 252, 262, 263 [13] Chin, F., Snoeyink, J., Wang, C.-A.: Finding the Medial Axis of a Simple Polygon in Linear Time. Discrete Computational Geometry 21(3) (1999) 405–420 262 [14] Cheng, S.-W., Vigneron, A: Motorcycle Graphs and Straight Skeletons. In: Proc. 13th ACM-SIAM Symp. Discrete Algorithms 156–165 2002 262 [15] Goodrich, M. T., Tamassia, R.: Dynamic Ray Shooting and Shortest Paths in Planar Subdivisions via Balanced Geodesic Triangulations. Journal of Algorithms 23(1) (1997) 51–73 262, 263 ˇ Straight Skeleton Implementation. In: Proc. of Spring [16] Felkel, P., Obdrˇza ´lek, S.: Conference on Computer Graphics, Budmerice, Slovakia 1998 210–218 262, 263, 265 [17] The Computational Geometry Algorithms Library. http://www.cgal.org/ 263 [18] SQUID database. http://www.ee.surrey.ac.uk/Research/VSSP/imagedb/demo.html 263

268


[19] Douglas, D. H., Peucker, T. K.: Algorithms for the Reduction of the Number of Points Required to Represent a Digitized Line or its Caricature. The Canadian Cartographer 10(2) (1973) 112–122 263

Reestablishing Consistency of Uncertain Geometric Relations in Digital Images Peter Veelaert Hogent, Schoonmeersstraat 52, 9000 Ghent, Belgium [email protected]

Abstract. In uncertain geometry in the 2D plane, points are replaced by uncertainty regions. By allowing uncertainty several geometric notions such as parallelism and concurrency become inconsistent with Euclidean geometry. In previous work we explained how consistency can be partially restored by graph-theoretical grouping algorithms. In this paper we study inconsistencies at a higher-level, e.g., the violation of Desargues’s Theorem or Pappus’s Theorem. We provide a simple algorithm that completely restores Euclidean consistency. Although the algorithm may not give optimal results with respect to grouping, it shows a way to develop more sophisticated algorithms to obtain global consistency in uncertain geometry.

1

Introduction

Uncertainty is an important issue in digital image processing. On the one hand, there is the uncertainty due to the imaging process itself. Capturing and manipulating an image introduces several forms of uncertainty, such as a digitization error, the positional error of edge detection algorithms, or the misclassification of pixels. On the other hand, when we perceive an unknown or partially known environment then uncertainty and likelihood measures are inherent to any model of the environment. As a result any method that extracts geometric structure from an image must cope with the possibility that some of the information will be inconsistent or invalid. In previous work we proposed a simple, universal scheme to model the uncertainty of pixel positions [10]. To each pixel we attach an uncertainty region, which represents the locus of all possible positions of the original real point. An uncertain version for notions such as straightness or parallelism has been developed following the same guidelines. For example, a set of pixels is considered to be straight if we can select in each uncertain region a real point, so that the selected points lie on a common straight line [10]. Once we allow for geometric uncertainty, geometric inconsistencies become unavoidable. For example, parallelism will no longer be an equivalence relation, and concurrency and/or collinearity relations may contradict parallel relationships. In previous work we used grouping algorithms to restore consistency [10, 11, 12]. For example, to derive a consistent list of parallel relations, T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 268–281, 2003. c Springer-Verlag Berlin Heidelberg 2003

Reestablishing Consistency of Uncertain Geometric Relations

269

we extract cliques from a so-called graph of parallel pairs. Similarly, graph algorithms were used to detect concurrency and collinearity relations. In this work we have a closer look at some of the inconsistencies that can still arise when we combine the different grouping algorithms, which have been developed previously. Furthermore, we give a more precise definition of what is meant by consistency in the Euclidean sense. The purpose of this and previous papers is to work out a methodology for geometric reasoning when the position and parameter vectors are unprecise or uncertain. On the one hand, this work builds further on results obtained in digital geometry. For example, we use the “discretization by dilation” scheme, which has been formalized by Heijmans and Toet [5], and our definition of straightness and thickness coincide with what has been defined for digital straight lines [8]. On the other hand, uncertain geometry was found to be strongly linked to graph theory. That is, many optimization problems in uncertain geometry can be formulated as minimum clique covering, minimum coloring or minimum dominating set problems [10, 9, 12]. In fact, graphs are used more and more to represent complex relationships between objects in an image. For example, self-organization has been viewed as a graph partitioning problem [3], the extraction of polytope faces from a line drawing has been formulated as a maximum weight clique problem [7], and the evaluation of similarity in image databases has been modeled as a subgraph isomorphism problem with limited error correction [1]. One of the strong points of uncertain geometry is that there is a direct link between the uncertainty model and the graph-theoretical optimization problems. That is, the solution of a given optimization problem can always be interpreted in terms of uncertainty regions and parameter domains. In Section 2 we recall previous results, and we define Euclidean consistency. In Section 3 we explain how grouping is used to restore Euclidean consistency up to a certain level. Section 4 shows how inconsistencies can still arise, even after grouping. In Section 5 we give a first, simple version of an algorithm that can be used to restore consistency completely. We conclude the paper in Section 6.

2

Derivation of Uncertain Geometric Relations

In this section we recall some results from previous work [10, 9, 12]. Since image pixels lie on a rectangular grid, we assume that all pixels are part of the digital plane ZZ 2 , although this is not strictly necessary for what follows. A digital set S is a subset of the digital plane. We model uncertainty by an uncertainty region that we associate with each grid point. The discretization process that coincides naturally with this notion of uncertainty is the discretization by dilation scheme developed by Heijmans and Toet [5]. Let U denote a set in IR2 , let A be a second subset of IR2 , called the structuring element, and let Ap be the translate of A by p. Then the discretization by dilation of U consists of all points p ∈ ZZ 2 for which Ap ∩ U is non-empty [5]. In previous work and also here we use one particular form of discretization by dilation, that is, we use a simple variant of grid-intersect discretization. To model

270

Peter Veelaert

Fig. 1. Digital collinearity and parallelism the uncertainty of its position, for each digital point p = (x, y), we introduce as translate of the structuring element the vertical line segment Cp (τ ), which comprises all points (x, b) ∈ IR2 that satisfy y − τ /2 ≤ b < y + τ /2. Here τ is a positive real number, called the acceptable thickness. To avoid confusion between a vertical “segment” and a digital straight line “segment” we shall call a vertical segment an uncertainty segment. The reason for using this particular form of uncertainty region is computationally efficiency. Uncertain geometry inherits its properties from Euclidean geometry in a straightforward way. Definition 1. A digital set S is called digitally straight if there is a continuous straight line that cuts all uncertainty segments Cp (τ ), p ∈ S. Definition 2. Let S1 , . . . , Sn a finite collection of finite digitally straight sets. We define the following digital geometric properties: – The sets {Si : i = 1, . . . , n} are called digitally collinear if there exists a common Euclidean straight line A that cuts the uncertainty segments of all the sets Si ; – The sets in {Si : i = 1, . . . , n} are called digitally parallel if there exist n Euclidean straight lines A1 , . . . , An that are parallel and such that for i = 1, . . . , n the line Ai cuts the uncertainty segments of the set Si ; – The sets in {Si : i = 1, . . . , n} are called digitally concurrent if there exist n Euclidean straight lines A1 , . . . , An that meet in at least one common point and such that for i = 1, . . . , n the line Ai cuts the uncertainty segments of the set Si ; A finite digitally straight set will also be called a digital straight line segment. We shall call two or more digital sets simply “collinear, parallel or concurrent’ when it is clear we either mean digitally collinear, parallel or concurrent. Digital collinearity and parallelism are illustrated in Figure 1. S1 and S2 are collinear since their uncertainty segments are cut by the common Euclidean line A. Likewise, S2 and S3 are collinear because of the common line transversal B. Moreover, S2 is digitally parallel with S4 , since B||C in the Euclidean


271

sense. Finally note that although S1 is collinear with S2 , S2 with S3 , as well as S1 with S4 , S1 is not collinear with S3 because there is no common line cutting all their uncertainty segments. For the same reason is S4 not collinear with S2 , nor with S3 . Evidently, in the digital plane collinearity and parallelism are not transitive. Definition 3. Let S be a finite digital set that contains at least two points with distinct x-coordinates, and let τ be a chosen acceptable thickness with 0 < τ . Then the domain of S, denoted as domx (S; τ ), is the set of all parameter points (α, β) ∈ IR2 that satisfy the following system of inequalities: − τ /2 < αxi + β − yi ≤ τ /2, (xi , yi ) ∈ S.

(1)

In other words, the domain contains the parameters of all the Euclidean lines that cut the uncertainty segments of S. It follows almost immediately that a collection of digital line segments is digitally collinear if and only if their domains have a common non-empty intersection [10, 9]. As an example, consider the collection of digital line segments in Fig. 5. For the line segments of slope −1 < α < 1, the domains are shown in Fig. 2, for an acceptable thickness τ = 25. A domain of a digital set with N points is a convex bounded set, and can be computed in O(N log N ) time as an intersection of 2N halfplanes [9]. When we let τ vary, a domain gets larger for increasing values of τ . If the acceptable thickness is too small, however, the domain of a set will be empty. For a given set S ⊂ ZZ 2 , let D ⊂ IR be the set of acceptable thicknesses for which the domain of S is non-empty. The thickness of S is defined as the infimum of D [10]. Let S = {A, . . . , } be a finite set of n digital line segments, and let P be a finite set of properties called incidences between these line segments, where the incidences are restricted to the following: collinearity of pairs, parallelism of pairs and concurrency of triples. For example, for S = {A, B, C, D, E} we may have P = {col(A, B), B||C, conc(A, C, D)}, which means that A and B are collinear, that B and C are parallel, and that A,C and D are concurrent. Definition 4. We call the set of incidences P consistent in the Euclidean sense provided the incidences P can also be realized in the Euclidean plane, i.e., each digital line segment E in S can be replaced by a Euclidean line lE , such that lE cuts all the uncertainty regions of points of E, and such that all the properties in P also hold for the set of Euclidean lines S = {lE | E ∈ S}, and that all the incidences that are not listed in P do not hold for the set of Euclidean lines S . Clearly, in uncertain geometry a set of incidences derived in accordance with Definition 2 need not be consistent, since for a set of three line segments S = {A, B, C}, it is possible to have P = {A||B, B||C} [10]. Since this list does not include A||C, it is not consistent in the Euclidean sense. The derivation of a consistent set of incidences from an inconsistent set may therefore be formulated as follows. Given a set of S of digital line segments and a set P of incidences, find a subset C ⊆ P such that C is consistent in the Euclidean sense and C is

272

Peter Veelaert

Fig. 2. Domains for a given set of digital line segments as large as possible. For example, suppose we are given P = {conc(A, B, C), conc(A, B, D), conc(A, C, D), conc(B, C, D), conc(A, B, E)}. Then P cannot be consistent, since conc(A, C, E) is missing (amongst other incidences). However, in principle the set C = R \ {conc(A, B, E)} could be consistent in the Euclidean sense, provided we can find Euclidean lines that satisfy these properties and that cut the uncertainty regions of the digital line segments. This puts strong constraints on the original list of incidences P . That is, we must develop an algorithm producing a list P , such that P contains large consistent sublists, even when P itself is inconsistent.

3

Euclidean Consistency through Grouping

Suppose we are given a collection of digital line segments and a list P of uncertain incidences. In the previous section the restoration of consistency was formulated as an optimization problem in which we discard as few incidences as possible so that the set of incidences becomes consistent in the Euclidean sense. The problem with this formulation is that determining the existence of a Euclidean realization is not straightforward. That is, the computational complexity of the Euclidean realization problem is the same as that of a theorem prover, as will be explained later. Therefore, the fact that the realization problem can be reformulated as a completely combinatorial problem, is very useful. In [10] the following result is derived as a Helly-type theorem:


273

Theorem 1. Let S = {S1 , . . . , Sn } be a finite collection of finite digital straight line segments, with n > 1. Then the collection S is digitally parallel if each subcollection containing two sets of S is digitally parallel. The consistency problem can now be reformulated as a purely combinatorial optimization problem. Suppose we represent parallel relationships in a so-called graph of parallel pairs G, in which each vertex represents a line segment and two vertices are connected by an edge provided the corresponding line segments are digitally parallel [12]. Then, according to Theorem 1, we must delete as few edges from G as possible so that the remaining graph G becomes isomorphic to a collection of disjoint cliques. Next, we can derive from G a list of consistent parallel relations. For each clique C in G , we include all pairs of vertices in C as a parallel pair in P . For example, suppose that G consists of to disjoint cliques {A, B, C} and {D, E}. Then we set P = {A||B, A||C, B||C, D||E}, which will be consistent in the Euclidean sense, according to the following result. Corollary 1. Let S be a finite collection of digital line segments, and let G be the graph representing the parallel relations between pairs of line segments of S. Furthermore, let P be the set of parallel incidences derived from a minimum clique covering of G. Then the incidences P are consistent in the Euclidean sense. Proof. First, according to Theorem 1 for each clique in the minimum clique covering of G we can find a set of parallel Euclidean lines such that each Euclidean line cuts all the uncertainty regions of the corresponding digital set. Hence, we can find a realization of Euclidean lines such that the relations P hold for these lines. It remains to prove that no other parallel relationships exist, that is, we must prove that we can choose the Euclidean lines in such a way that a line in one clique cannot be parallel to a line in another clique. For each clique C the Euclidean lines cutting the uncertainty regions of theline segments must have a slope which lies in the intersection of the intervals i πα dom(Ai ), where the intersection includes each digital line segment Ai in C, and where πα dom(Ai ) denotes the projection of dom(Ai ) upon the α-axis. Now suppose that we have a realization such that there is Euclidean line A representing a clique C1 , and a second Euclidean line B representing a second clique C2 , and that we have A||B. It follows that ( i πα dom(Ai ))∩( i πα dom) = ∅. Or in other words, in the graph G each vertex in C1 is adjacent to each vertex in C2 , which means that the union of C1 and C2 forms a larger clique and that C1 and C2 are therefore not part of a minimum clique covering. Figure 3 shows a hand-made drawing of squares. The graph of parallel pairs is shown in Figure 4, for uncertainty threshold τ = 4. A minimum clique covering is given by {A, B, C}, {D, E, F }, {G, H, I, J, K, L}. Therefore, the list of consistent parallel incidences contains 3 + 3 + 15 = 21 parallel pairs. The graph shown in Figure 4 contains 27 edges. The 6 dotted edges represent parallel pairs that have been discarded to make the list of parallel incidences consistent. Note that the given covering is not unique; also {A, B}, {C, D, E, F }, {G, H, I, J, K, L} is

274

Peter Veelaert

Fig. 3. Scanned hand-drawn squares

a minimum clique covering, which would give rise to another consistent list of parallel incidences.

4

Euclidean Inconsistency for Higher Order Theorems

Results similar to Corollary 1 can be proven for collinearity and concurrency; for all the details we refer to [13, 14]. Using these results we can restore consistency for each type of property separately. That is, apart from a set of parallel incidences P1 , we can construct two other sets of incidences P2 and P3 , for collinearity and concurrency respectively. Each set will be consistent on its own, in the Euclidean sense. The union P = P1 ∪ P2 ∪ P3 , may not be consistent, however, as shown in the following example. In Figure 5, for τ = 9, the following relations have been found: P1 = {{G, H}} is a collinear pair; P2 = {{A, B},{C, D},{F, G},{I, J}} are the parallel pairs; and, P3 = {{B, C, G}, {B, C, J},{B, G, J},{C, G, J},{A, E, G}, {B, E, F }, {C, F, I}, {A, D, J}, {A, D, I}} are concurrent triples (here we used the fact that G and H can be considered as one line segment). If considered separately, each set of incidences is consistent. However, if we consider the union P = P1 ∪ P2 ∪ P3 , then according to Desargues’s Theorem we should also have that the triple {C, D, E} is either parallel or concurrent, which in the uncertain context is not detected as a valid incidence relation. That is, the list of incidences P is inconsistent in the Euclidean sense. Figure 6 shows another example, with a line drawing meant to illustrate Pappus’s Theorem. All obvious concurrency properties have been detected for τ = 4, such as the concurrency of {A, B, C, D}, and {B, G, F }. However, {E, F } is not detected as a collinear pair of line segments although, according to Pappus’s Theorem these two line segments should be collinear.


275

Fig. 4. Graph representing the parallel relationships between the line segments of the squares

In fact, Corollary 1 is based upon a Helly-type theorem and it can be used to restore consistency at a local level, i.e., by minimum clique covering for parallelism. Together with similar properties for collinearity and concurrency, Corollary 1 yields consistent incidence properties, but it fails to take into account Desargues’s and other higher geometric theorems, such as the ones shown. To avoid higher level inconsistencies, several approaches are possible. A first possibility is to add more properties to the theory, including an uncertain version of the notion of congruence, since congruence is also needed in Euclidean geometry to prove Desargues’s Theorem. This approach has the disadvantage, however, that there is no guarantee that we can exclude further higher level inconsistencies, unless we add an uncertain version for each axiom of Euclidean geometry, including notions such as betweenness and the congruence of angles [6, 4]. A second approach, is to feed the set P of incidence properties into a geometric theorem prover, which can then find out whether the properties are realizable in the Euclidean plane [2]. By computing the solution space of a system of nonlinear equations, such a theorem prover will find out whether the given set of incidences can be realized in the Euclidean plane. The use of a theorem prover has the drawback that it is time-consuming and that some of the information about realizations in the Euclidean plane that is already available is discarded. In fact, the domains of the lines contain already useful information about possible realizations. To find out how the information of the domains can be used, we examine the cause of the inconsistency found in Figures 5 and 6 more closely. Figure 7(a) provides an alternative view of the uncertainty of the line parameters. Figure 7(a) shows the region swept out by all possible Euclidean lines that pass through the uncertainty regions of a given digital line segment. Such region has been called the preimage of the digital line segment [10]. To be precise, the preimage 2 dig−1 x (S) is the set of all points (x, y) ∈ IR for which there exist parameters (α, β) ∈ domx (S) such that y = αx + β. Thus, each Euclidean line which lies in

276

Peter Veelaert

Fig. 5. Hand-drawn illustration of Desargues’s Theorem

the preimage, corresponds to a parameter point in the domain of the digital line segment. One can prove that a preimage has the shape of a butterfly tie [10]. Figure 7(b) shows the preimages for a set of digital lines segments of a handmade drawing of Desargues’s Theorem. Figure 7(c) shows how we can determine geometric properties using the alternative setting of preimages. Suppose we are given two digital line segments, if we can find two parallel Euclidean lines, where each line lies in one of the two preimages, then the two corresponding digital line segments are digitally parallel [10]. Thus in Figure 7(c), A and B form a parallel pair, as well as C and D. Concurrency, is handled in a similar way. If we can find three concurrent Euclidean lines, each line lying in a preimage corresponding to a particular digital line segment, then the digital line segments are concurrent. Figure 7(d) shows how this may lead to inconsistencies. In Figure 7(d) we have two lines A and D, and a concurrent triple E, F and G. The problem, however, is that the Euclidean lines A and D, which are used to determine parallelism, are not the same as the lines F and G that are used to determine concurrency, although for example, D and F are representing the same digital line segment. If we would use in each preimage a single Euclidean line to determine all properties, then this would automatically lead to Euclidean consistency. The reason why no single representant is used, is due to the computational complexity of the problem caused by the fact that a result such as Corollary 1 can only be used to determine a single kind of property, in this case parallelism. To summarize, because of computational efficiency, the general idea that in uncertain geometry oâll properties must be inherited from Euclidean geometry¨ o has only been applied to each type of incidence relation separately. The inconsistencies illustrated in Figure 7 show that the currently used combinatorial methods fail as soon as we combine different types of incidences. Furthermore,


277

Fig. 6. Hand-drawn illustration of Pappus's Theorem to avoid such inconsistencies, it is clear that we must include more properties of the Euclidean plane, either in a direct way by extending uncertain geometry with more concepts such as congruence, or indirectly by verifying the existence of realizations of lines and points in the plane R2,as done by a geometric theorem prover. The solution that we propose here will determine consistent realizations in reasonable time. However, in contrast with a theorem prover which would consider all possible realizations, we start from the positional information provided by the uncertainty regions and domains. More precisely, we propose to maintain a list P of consistent incidences and to adapt the shape of the domains each time a new incidence relation is added to this list. Thus we make use of the information available in the domains of the digital line segments. In this view, each domain represents the realization space of one particular Euclidean line.

5

Induced Domain Adaptations

Since domains are subsets of the a@-plane, domains contain information about possible realizations. In fact, suppose that we would reduce beforehand each domain to one parameter point (ai,Pi). Since each parameter point represents a single Euclidean line, then the geometric incidences that we find for the single point domains would certainly be consistent. Therefore, the method that we use here is the reduction of the domains so that only consistent properties can arise.

278

Peter Veelaert

Fig. 7. Higher-level inconsistencies: (a) The preimage of a digital line segment, (b) preimages for Desargues’s Theorem, (c) how parallelism arises, and (d) the inconsistency between parallelism and concurrency

Several different approaches to domain reduction are possible, however. We start with the most straightforward, but not necessarily optimal, kind of reduction, as described in the following algorithm. Derivation of a set of incidences, consistent in the Euclidean sense. Input: A collection of m digital line segments Si , and their domains Di . Output: A list P of incidences, and a set of m Euclidean straight lines Ri , such the incidences P are valid for the lines Ri , and such that the line Ri cuts all the uncertainty regions of one of the digital line segment Si . First set P = {}. Next we look for an incidence between the line segments. Since parallelism and collinearity involve two segments, and concurrency three segments, the total amount of combinations that must be verified is bounded polynomially. Each time we add an incidence to P , the domains Di are reduced as follows: 1. For a collinear incidence col(Si , Sj ) replace Di and Dj both by Di ∩ Dj ; 2. For a parallel incidence Si ||Sj , find a vertical Euclidean line L : α = α0 in the αβ-plane that cuts the domains Di and Dj , and replace Di by Di ∩ L, and Dj by Dj ∩ L;


279

3. For a concurrent incidence conc(Si , Sj , Sk ), find a Euclidean line L in the αβ-plane that cuts the domains Di , Dj and Dk , and replace each domain Di by Di ∩ L, Dj by Dj ∩ L, and Dk by Dk ∩ L; This process must be repeated until no other incidences can be found. Finally, if no additional incidences exist, we have the following result. Some domains may still have their original shape, some domains may have been reduced to a Euclidean line segment, and some may have been reduced to a single point. Therefore, for all domains that have not been reduced to a single point, we select an arbitrary point in the domain. Thus we obtain a collection of m parameter points. Then we let Ri be the Euclidean line that corresponds to the i-th parameter point. It is clear that the list P will be consistent, since the incidences hold for real Euclidean lines Ri . Furthermore, since the i-th parameter point belongs to the domain Di , the line Ri cuts all the uncertainty domains of the digital line segment Si . Finally, the list will also be complete, since the algorithm only stops when all incidence relations have been found. Figure 8 shows how the reduction is done for a given set of five digital line segments. Figure 8(a) shows the domains of the line segments A, . . . , E in the αβ-plane. We start with an empty list P of detected incidences. Since there is a straight line P1 cutting the domains of A, B and C, these three line segments are concurrent, and therefore we add conc(A, B, C) to the list P . Figure 8(b) shows how the domains of A, B and C have been replaced by the new domains A , B and C . Next, since there is a vertical line P2 cutting the domains C and D, we add C||D to the list P , and we replace the domains C and D by the reduced domains C and D , which are shown in Figure 8(c). Note, that C now consists of a single point. Then, since there is a vertical line P3 cutting B and E, we add B||E to the list P , and we replace B and E by B and E , as shown in Figure 8(d). Finally, since no other properties can be found, we select arbitrary parameter points in all the domains that contain more than one point. The five parameter points represent Euclidean straight lines, which cut the uncertainty regions of each of the five digital line segments, and which satisfy the incidences P = {conc(A, B, C), C||D, B||E}. The list P is not the largest possible, however. If we would have used the lines Q2 , Q3 , as shown in Figure 8(a), instead of P2 and P3 , then it would have been possible to find a fourth line Q4 cutting the reduced domains A and E and D , which would have led to a fourth incidence conc(A, D, E). The parameter points that would have been chosen instead, are shown also in Figure 8(a). The drawback of this algorithm is that it is not optimal. ¿From the algorithm it is clear, however, that it is possible to extend the above algorithm by adding multiple incidences simultaneously. For example, suppose that at a certain stage of the algorithm, we have found a consistent set of parallel groups by using minimum clique covering. Then it is clear that all the induced parallel incidences can be added at once. Furthermore, it is also possible to reduce the domains in a less strict way. That is, we can keep a larger part of each reduced domain, thereby increasing the probability of incidences. This requires more bookkeeping, however, and the extended algorithm will not be discussed in this work.

280

Peter Veelaert

Fig. 8. Illustration of the algorithm used to remove inconsistencies

6

Concluding Remarks

In this work we have seen that the combinatorial level is not sufficient to obtain consistency. Consistency can be obtained, however, if we use the information contained in the parameter domains of the line segments, i.e., for each digital line segment we must look at the realization space of the Euclidean line representing it. The difference with the combinatorial approach as employed in [12, 13, 14], is smaller than one might expect, however. Even when consistency is reestablished at a purely combinatorial level, it is perfectly possible to explain inconsistencies in terms of domains. For example, the so-called weak collinearity graph yields inconsistent groupings when there is a cyclic chain of domains such that the intersection of the entire chain is empty, although the intersection of any pair of domains in the chain is non-empty [13]. That is, also in this case the relative positions and shape of the domains can be used to explain the inconsistency. To conclude, the subject of uncertain geometry is to study configurations of uncertainty regions, domains and preimages, which replace the points and lines of Euclidean geometry. Sometimes, it is possible to make an abstraction of the actual positions and shapes of these regions, and to process geometric


281

relationships at a purely combinatorial level, which results in greater computational efficiency. For more involved problems, however, we still need the precise geometry of the domains, and we obtain consistency by manipulating the shape of the domains.

References [1] S. Berretti, A. Del Bimbo, and E. Vicario. Efficient matching and indexing of graph models in content-based retrieval. IEEE Trans. Pattern Anal. Machine Intell., 23:1089–1105, 2001. 269 [2] D. Cox, J. Little, and D. O’Shea. Ideals, Varieties and Algorithms: an Introduction to Computational Algebraic Geometry and Commutative Algebra. Springer, New York, 1992. 275 [3] Y. Gdalyahu, D. Weinshall, and M. Werman. Self-organization in vision: Stochastic clustering for image segmentation, perceptual grouping, and image database organization. IEEE Trans. Pattern Anal. Machine Intell., 23:1053–1074, 2001. 269 [4] R. Hartshorne. Geometry: Euclid and beyond. Springer, New York, 2000. 275 [5] H. J. A. M. Heijmans and A. Toet. Morphological sampling. CVGIP: Image Understanding, 54:384–400, 1991. 269 [6] R. Hilbert. Foundations of geometry. Open Court, La Salle, 1971. 275 [7] J. Liu and Y. T. Lee. A graph-based method for face identification from a single 2D line drawing. IEEE Trans. Pattern Anal. Machine Intell., 23:1106–1119, 2001. 269 [8] J.-P. Reveillès. Géométrie discrète, calcul en nombres entiers et algorithmique. Thèse d’Etat, Université Louis Pasteur, Strasbourg, 1991. 269 [9] P. Veelaert. Algorithms that measure parallelism and concurrency of lines in digital images. In Proceedings of SPIE’s Conference on Vision Geometry VIII, pages 69–79, Denver, 1999. SPIE. 269, 271 [10] P. Veelaert. Geometric constructions in the digital plane. J. Math. Imaging and Vision, 11:99–118, 1999. 268, 269, 271, 272, 275, 276 [11] P. Veelaert. Line grouping based on uncertainty modeling of parallelism and collinearity. In Proceedings of SPIE’s Conference on Vision Geometry IX, pages 36–45, San Diego, 2000. SPIE. 268 [12] P. Veelaert. Parallel line grouping based on interval graphs, volume 1953 of Lecture Notes in Computer Science, pages 530–541. Springer, Uppsala, Sweden, 2000. 268, 269, 273, 280 [13] P. Veelaert. Collinearity and weak collinearity in the digital plane, volume Digital and Image Geometry of Lecture Notes in Computer Science, pages 439–453. Springer, Dagstuhl, Germany, 2001. 274, 280 [14] P. Veelaert. Concurrency of line segments in uncertain geometry, volume 2301 of Lecture Notes in Computer Science, pages 289–300. Springer, Bordeaux, France, 2002. 274, 280

Inverse Quantization for Resolution Conversion A. Torii1 , T. Ichinose1 , Y. Wakazono1 , and A. Imiya2,3 1

3

School of Science and Technology, Chiba University, Japan 2 National Institute of Informatics, Japan Institute of Media and Information Technology, Chiba University, Japan

Abstract. In this paper, we introduce a resolution-conversion method for two- and three-dimensional discrete objects. We first derive a method for boundary extraction, second, introduce a method for the estimation of a smooth boundary, and third, construct an algorithm for resolution conversion.

1

Introduction

In this paper, we propose a requantization process based on an inverse quantization method for binary digital images on a plane in a space. If a shape is sampled and expressed as a digital shape, it is impossible to reconstruct a high-resolution boundary. Since resolution conversion produces high-resolution digital images from low-resolution ones, we can only register low-resolution images and objects in the memory of computers. Therefore, the resolution-conversion method enables us to reduce the amount of memory required for data storage. In Figure 1, we present our concept for resolution conversion as an application of digital data archiving. In the previous paper [1], we introduced a diffusion equation on graphs and applied it for the deformation of polyhedrons. Furthermore, we proposed a resolution-conversion method for planar discrete shapes, using the curvaturedriven flow of planar polygons. In this paper, we deal with the resolution conversion of discrete objects, employing the two results which we introduced previously. Therefore, for the reconstruction of a shape with high resolution from a digital shape, we are required to solve an inverse problem which estimates the original boundary. The inverse quantization of digital terrain data for the recovery of a smooth terrain surface and a series of iso-level counters on it are solved using variational problems. This is a surface reconstruction method which is common in computer vision and aerial data processing [2, 3, 5]. The expansion and super-resolution of digital binary images are mathematically equivalent problems because, for the achievement of these processes, we are required to construct a smooth boundary curve and surface as an estimation of the original boundary, from a digitized object which is expressed as a collection of pixels and voxels. Spline curves and surfaces are described as the solution of a variational problem for the fitting of smooth functions to a sequence of samples along a curve and an array of sample points on a surface. Therefore, splines are utilized for the T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 282–300, 2003. c Springer-Verlag Berlin Heidelberg 2003

Inverse Quantization for Resolution Conversion

283

estimation of the smooth boundary from a collection of samples [4, 5, 6, 7]. The families of splines are attended on by several authors in the computer vision community with respect to curve fitting [8], corner detection [9], shape recovery [3], and detection of discontinuities along the boundary [10]. The spline curves has also been closely examined in meteorology for the description of iso-level curves on weather charts [5, 6]. Furthermore, a family of splines has recently been studied theoretically in the context of wavelets and practically in shape description for the application of shape expression for data transmission through the internet [11, 12]. These applications partially refer to the application of splines in computer vision for the data compression of boundary information. If we could estimate the boundary of the original object from a discrete object, it would be possible to express the original object using smaller pixels and voxels. Therefore, the first step of resolution conversion is the estimation of the boundary of the original object. We call this process the boundary extraction of binary objects. In the previous paper [13], we proposed the inverse-quantizationbased resolution conversion for planar objects. In this paper, we extend the idea to three-dimensional objects in a space and terrain. The generation of a high-resolution object for two- and three-dimensional objects is mathematically achieved by computing a binary set, applying a sampling scheme to its expanded set, and reducing the length of the edges of pixels and voxels, if we define the resolution by using the length of the pixels and voxels. We derive an algorithm for the generation of high-resolution digital objects from the set which is obtained using the sampling scheme with ordinary pixels and voxels. Our resolution-conversion method first estimates the smooth boundaries of objects from discrete objects produced by a sampling procedure. We first derive a method for the extraction of a boundary as a collection of edgel and surfel from a 4-connected and a 6-connected object in a plane and a space, respectively. Our method in a space is a parallel version of Herman’s surfel extraction method, that is, our method first extracts a surfel slice-by-slice in three orthogonal directions which are parallel to the axes of the coordinate system and second, constructs the union of these surfels as the boundary of an object. Next, we introduce a method for the estimation of a smooth boundary using the deformation process for these orthogonal polygons and polyhedrons. Finally, using the deformed boundary, we construct a high-resolution object by resampling the object with the estimated smooth boundary.

2 2.1

Boundary Estimation and Resolution Conversion Boundary Extraction

The resolution conversion introduced in the previous paper [13] requires edges which separate the 0-pixels and 1-pixels to determine the boundary of a digital binary image. The three-dimensional analogy of this algorithm requires the surfels as the boundary which separates 0-voxels and 1-pixels. In this section, we

284

A. Torii et al.

Fig. 1. Concept of resolution conversion for image data retrieval

discuss a uniformed framework for the extraction of edge and surfel set in twoand three-dimensional spaces, respectively. We deal with the two- and three-dimensional discrete spaces Z2 and Z3 , respectively. Hereafter, we call R2 and R3 two- and three-dimensional spaces, respectively. Therefore, Z2 and Z3 are two- and three-dimensional discrete spaces, respectively. For (k, m, n) ∈ Z3 , we set Z21 (k), Z22 (m), and Z23 (n) as twodimensional planes x = k, y = m, and z = n, respectively. Plane Zi (α) is perpendicular to ei for i = 1, 2, 3, for e1 = (1, 0, 0) , e2 = (0, 1, 0) , and e3 = (0, 0, 1) . For points (m, n) and (k, m, n) in Z2 and Z3 , respectively, (m , n ) and (k , m , n ) such that (m − m)2 + (n − n)2 ≤ 1, (k − k)2 + (m − m)2 + (n − n)2 ≤ 1,

(1)

are 4-connected and 6-connected points on a plane and in a space, respectively. Furthermore, for points (m, n) and (k, m, n) in Z2 and Z3 , respectively, (m , n ) and (k , m , n ) such that (m − m)2 + (n − n)2 ≤ 2, (k − k)2 + (m − m)2 + (n − n)2 ≤ 3,

(2)

are 8-connected and 26-connected points on a plane and in a space, respectively. We express them as N4 (x), N6 (x), N8 (x), and N26 (x). In Zα (β), we express the 4-connected points as N4α (β). For x = (k, m, n) , the neighborhood in a space and on the planes satisfies the relation (3) N6 (x) = N41 (k) N42 (m) N43 (n).


285

Hereafter, we affix 0 and 1 to points in Z2 and Z3 , and our object is the collection of pixels and voxels in two- and three-dimensional spaces whose centers are points in Z2 and Z3 . We set x = (m, n) , 1, |m − 12 | ≤ 1, and |n − 12 | ≤ 1, u(x) = (4) 0, otherwise, and x = (k, m, n) , 1, if |k − 12 | ≤ 1, |m − 12 | ≤ 1, and |n − 12 | ≤ 1, v(x) = 0, otherwise,

(5)

respectively. u(x) and v(x) are a pixel and a voxel, respectively, whose centers are at x = (m, n) and x = (k, m, n) . Therefore, setting F to be the set of 1-points, our object is expressed as in two- and three-dimensional spaces: D= u(x), D = v(x). (6) x∈F x∈F We call the boundary of D the edge polygon and the surface polyhedron, for the collection of pixels and voxels, respectively. The edge polygon and surface polyhedron are extracted as (7) ∆F = {(F ⊕ N8 ) \ F } {F \ (F N8 )}, (8) ∆F = {(F ⊕ N26 ) \ F } {F \ (F N26 )}, where ⊕ and are the Minkowski addition and subtraction, respectively, of two sets in a vector space. Next, we define edge polygon and surfel polyhedron in Z2 and Z3 , respectively, for a set of points F . The edge polygon is extracted as follows. – Search for a pair of vertices p1 and p2 on u(x) for the boundary of ∆F . – Follow points which satisfy the relations |pi+1 −pi | = |pi −pi−1 |, and (pi+1 − pi ) (pi − pi−1 ) is 0 or 1, for i ≤ 2. The surfel polyhedron of the three-dimensional discrete object is obtained by applying the procedure slice by slice in the directions e1 = (1, 0, 0) , e2 = (0, 1, 0) , and e3 = (0, 0, 1). Since we are dealing with the 6-connected discrete objects, for a plane Pi (k) which is a perpendicular vector ei for i = 1, 2, 3 and which passes through point kei , the vertices of object O lie on the cross sections of object O with respect to Pi (k), and the degree of vertices is three or four. Furthermore, vertices adjacent to a vertex exist in the 6-neighborhood of the vertex, since we are dealing with 6-connected discrete objects. The surface polyhedron is extracted by applying this algorithm slice-by-slice in each Pi (k). For a terrain such that z = f (x, y), we assume that we are dealing with discrete objects which are infinite in the direction of −e3 = (0, 0, −1) . Figure 2 shows the boundary edgels and surfels of objects on a plane and in a space.

286

A. Torii et al.

Fig. 2. Edgels and surfels of objects in 2D and 3D, respectively

2.2

Generation of High-Resolution Images

We set an object f (x), where x = (x, y) and x = (x, y, z) for two- and threedimensional objects, respectively. We define the set of points A = {x | f (x) = 1, x ∈ Rn } in Rn for n = 2, 3. We also define λF = {λx | x ∈ F , λ > 0} for the point set F in Zn for n = 2, 3. Setting fm , m ∈ Zn for n = 2, 3 to be the average volume of f (x) in a pixel and a voxel in two- and three-dimensional spaces. The boundary extraction is to estimate A from F = {m | fm > 12 } and 1 resolution conversion is described as the computation of m F m , where F m is 1 Fm a binary set computed from the binary object f (mx). Furthermore, set m enables us to generate an approximation of high-resolution images of f (x) for an arbitrary resolution. If A and its boundary ∂A are estimated from F , it is easy to generate F m by computing the average in the pixels and the voxels where 1 edge length is m unit. In the previous section, we proposed an algorithm for the estimation of boundary curve ∂A from digital set F . Therefore, using the estimation of D, 1 F m according to the following steps. we generate set m 1. Compute D from F . 2. Compute the B-spline curve from mD, and adopt its closure as the estimator of m∂A. 3. Apply the sampling scheme to the closure of the curve using unit pixels and voxels. 4. Reduce the size of pixels and voxels uniformly.

3 3.1

Boundary Estimation for Planar Object Boundary Curve

Setting f (x, y) to be the value of a binary image defined on R2 , where f (x, y) is the value of an image at a point x = (x, y) , let fmn be the average of


287

the value of the binary image f (x, y) in u((m, n) ). For a collection of vectors F = {(m, n) | fmn ≥ 12 }, we define three sets, A = {(x, y) |f (x, y) = 1}, D= u((m, n) ), (m,n) ∈F u((m, n) ). B= (m,n) ∈∆F

(9) (10) (11)

We assume that boundary ∂A of region A is a continuous smooth simple curve. Since we assume a 4-connected boundary, the original boundary curve ∂A of an image is contained in region B, which is an orthogonal polygonal curve of finite width. Our problem is the reconstruction of boundary ∂A from a binary digital image D. We adopt set B as the first estimation of the original boundary ∂A. Furthermore, as the second approximation of ∂A, we adopt the boundary of D. The boundary of set D is an orthogonal polygon curve d(s) which lies within the finite closed set B. We assume that a set of points {pα }nα=1 on curve d(s) is ordered in the counterclockwise direction and points pi±1 and pi are 4-connected. For the third approximation of ∂A, we construct a B-spline curve p(s) of order three, using ordered points {pα }nα=1 along polygonal curve d(s). This curve p(s) passes through the area encircled by polygon torus P , which is the union of the convex hulls defined by the four successive points pi , pi+1 , pi+2 , and pi+3 , on ∂A, where pi = pn+i for i = 1, 2, · · · , n [14]. This polygonal region P is contained in region B. Therefore, the estimation curve p(s) might be closer to the original curve than the orthogonal polygon curve d(s). The vertex angles of an orthogonal polygon are π/2 and 3π/2 and the distance between each pair of control points is one. These configurations of points on a curve yield small smooth vibrations on the B-spline curve whose control points lie on an orthogonal polygon. If we deform the polygonal boundary curve P using a discrete curvature flow, pα (t + 1) − pα (t) = F (θα ),

(12)

where θα is the discrete curvature of point pα for the t-th iteration on a polygonal curve and F (x) is a function, we can write the minimization criterion as S 2 n d p(s) 2 |p(sα ) − pα (t)|2 + λ dt. (13) E(p(s), pα (t)) = ds2 0 s=t α=1 3.2

Deformation of Polygonal Boundary Curve

The geometric configuration of sets P and B implies that, if we can generate a series of polygonal regions P (t) such that – P (t + 1) ⊂ P (t) for t = 0, 1, 2, · · · , and P (0) = P ;

288

A. Torii et al.

– limt→∞ |P (t)| = 0, where |P (t)| is the areal measure of polygonal set P (t), using the curvature of each point on a polygonal curve, we can define an operation which approximately generates this sequence of polygonal curves. From the vertices of polygonal curve {pi }nα=1 , we define the vectors and their average: uki (t) =

n

1 1 k {(pi−k (t) − pi (t)) + (pi+k (t) − pi (t))}, v ni (t) = ui (t), (14) 2k n k=1

for k = 1, 2, · · · , n. Vector v ki expresses global configurations of points for k ≥ 1, since vector v ki is defined from (2k + 1) successive points {pα }i+k α=i−k for point pi . The angle between vectors (pi−k − pi ) and (pi+k − pi ), and the average of these angles, for k = 1, 2, · · · , n, are defined as n (pi−k − pi ) (pi+k − pi ) 1 k θi . (15) θik = cos−1 , φni = |pi−k − pi | · |pi+k − pi | n k=1

The angle θik expresses the local turn of a planar polygonal curve for k = 1. We call θik the vertex angle of the order k. The average of vertex angles describes the global turns of a planar polygonal curve. We call φni the average vertex-angle of the order n. If the average vertex-angle φni is larger than the threshold τ , the polygonal boundary curve is not locally smooth. Conversely, if the average vertex-angle φni is smaller than a threshold τ , the polygonal boundary curve is locally smooth. If a curve is not locally smooth at a point, we deform a vertex inward. Furthermore, if a curve is smooth globally, we deform a vertex to enhance the global shape within a finite region. Based on these rules, we can describe the equation for the deformation of a polygon as α(t)u1i (t), if φni > τ , (16) pi (t + 1) − pi (t) = 1 n α(t)ui (t) − β(t)v i (t), otherwise, for a pair of monotonically decreasing positive functions α(t) and β(t), such that limt→∞ α(t) = 0 and limt→∞ β(t) = 0, respectively. These requirements for the coefficients of the recursive form might preserve the condition pi (t + 1) = pi (t) for large t. If this equality is satisfied, the points on P (t) remain within a finite region along a finite polygonal curve. For point pi , which is defined as pi = limt→∞ pi (t), setting P = {pi }ni=1 , the boundary ∂A of the support of the binary function f (x, y) is estimated as the B-spline curve whose control points are elements of P . Here, we set n = 3, since the configuration of seven successive points determines the local shape of a curve expressed by B-spline polynomials of the order three. One of the advantages of B-spline polynomials for the expression of curves is that B-spline polynomials approximate the original curve using fewer control points. Therefore, we derive an algorithm for the reduction of the number of


289

control points {pi }ni=1 to P o = {poj }j∈I , where I is a subset of integers from 1 to n. Setting φni to be the average vertex angle of polygonal curve P , for the generation of P o from P , we adopt point pi which satisfies one of the following conditions. – The average vertex angle φni is larger than a predetermined threshold φ. – If pi is an element of point set P o , then point pi+3 is an element and points pi+1 and pi+2 are not. We set n = 3 for the computation of the average vertex angle, since we use B-spline polynomials of order three.

4 4.1

Boundary Estimation in a Space Detection of Boundary

Setting f (x, y, z) to be the value of a binary object defined on R3 , where f (x, y, z) is the value of an object at a point x = (x, y, z) , let fkmn be the average of binary object f (x, y, z) in v((k, m, n) ). For a collection of vectors F = {(k, m, n) | fkmn = 1}, we define three sets, A = {(x, y, z) |f (x, y, z) = 1}, D= v((k, m, n) ), (k,m,n) ∈F B= v((k, m, n) ). (k,m,n) ∈∆F

(17) (18) (19)

We assume that boundary ∂A of region A is a continuous smooth simple surface. Since we assume a 6-connected boundary, the original boundary surface ∂A of an object is contained in region B, which is an orthogonal polyhedron surface of finite width. Our problem in the three-dimensional space is the estimation of ∂A starting from B. 4.2

Curvature Flow in a Space

Setting a finite number of three-dimensional vectors {pi }ni=0 such that V = {p1 , p2 , · · · , pn } to be the vertices of a polyhedron, we define the neighbors of each vertex pi as the adjacent vertices of pi such that V(i) = {pj }ij=i(1) (j), where pi(j) and pi(j+1) . These notations show that a polyhedron consists of a triplet of graphs with indices on vertices which have the same topological structure. We define vi =

i(k) j=i(1)

v ij , f i =

i(k) j=i(1)

f ij , tmn (i) = (pi − pm ) + (pi − pn ),

(20)

290

A. Torii et al.

where pm and pn do not lie on the same face. We call v i , f i , and tmn (i) the vertex normal, the face normal, and a path normal of vertex pi . The vertex normal is the sum of all edge vectors which are connected to a vertex. The face normal is the sum of all normal vectors of the triangles formed by a point and two points in its neighborhood which are connected by an edge. The path normal is the sum of a pair of edge vectors. These notations derive the following definitions. Definition 1 The path normal and face normal vectors classify the geometric properties of a vertex in each neighborhood as follows. 1. 2. 3. 4.

For all m and n in V(i), if tmn (i) f i > 0, then pi For all m and n in V(i), if tmn (i) f i = 0, then pi For all m and n in V(i), if tmn (i) f i < 0, then pi For all m and n in V(i), if sgntmn (i) f i depends a saddle point.

is quasi-convex. is flat. is quasi-concave. on m and n, then pi is

We define the gradient on the polyhedral surfaces by ∇pi = [pi(1)i , · · · , pi(k)i ] , pji = pj − pi .

(21)

Since pi is vector, eq. (21) is the vector gradient and ∇pi is a i(k) × 3 matrix. The dimension i(k) of this matrix depends on the degree of vertex pi . Since the divergence of a vector-valued function is the sum of the elements of vector values, we define the divergent operation on polyhedrons as divf (pi ) =

i(k)

q ni

(22)

n=1

for a matrix function which is defined on the vertices of a polyhedron such that f (pi ) = Qi for Qi = [q i1 , q i2 , · · · , q ni ] , where q ik s are three-dimensional vectors and n is the degree of a vertex pi . From eqs. (21) and (22), we obtain the Laplacian operation on the polyhedral vertices, 1 ∆pi = ∇pj (23) j∈V (i) since the divergent of the gradient of a scaler function yields the results of the application of the Laplacian. The definition implies the same expression with ∆pi = pj − pi (24) j∈V (i) which is introduced by Fisher [15] and Collatz [16, 17, 18]. 1

e p

eQ

For i = 1, 2, 3 correspond to the gradient and a scaler function i ∇ i and i on a graph, respectively, since a polyhedron is a labelled graph at each node, which determine positions of nodes in Euclidean space.


291

The total sum of the lengths of the gradients on a surface acts as a total roughness measure of this surface. Then, the length of the gradient at each point characterizes the local roughness of the surface at each point. The length of a path which passes through a vertex corresponds to the curvature along this path at this point, and the path length is computed by elements of the gradient. Therefore, we define the mean curvature of vertex pi using the lengths of the gradients on a polyhedron. Definition 2 Setting ri to be the average of the elements of the gradient at vertex pi such that i(k) 1 ri = |p |, (25) i(k) j=1 i(j)i we define a discrete version of the mean curvature on each vertex as ki = sgn(v i f i ) × ri .

(26)

From this definition, we derive a discrete version of the mean curvature flow [1], pi (t + 1) − pi (t) = ki (t) × sgn(v i f i)

v i (t) , |v i (t)|

(27)

i (t) where t is the step of the iterations, since ki and |v v i (t)| are the discrete mean curvature of vertex v i and the outer normal of vertex v i , respectively. v i is the outward vector if pi is quasi-convex, otherwise v i is the inward vector. Therefore, v i (t) determines the outward normal vector at vertex p (t). sgn(v i (t) f i (t)) |v i i (t)| A linearization of eq. (27) is

pi (t + 1) − pi (t) = αv i ,

(28)

for a nonzero constant α. Furthermore, setting P = [p1 , p2 , · · · , pn ] , the Laplacian operation for a polyhedron is expressed as P := LP , where L = A − D for the adjacency matrix A and the degree matrix D of a polyhedral graph. 4.3

Curvature Flow in the Discrete Space

Setting p1i = (0, yi , zi ) , p2i = (xi , 0, zi ) , and p3i = (xi , yi , 0) , for pi = (xi , yi , zi ) , the Laplacian operation is decomposed as LP =

1 2 (D P 1 + D 2 P 2 + D 2 P 3 ), 2

(29)

where D 2 is the second order derivation for a discrete sequence and α α P α = [pα 1 , p2 , · · · , pn ]

(30)

for α = 1, 2, 3. This expression allows us to decompose deformation in a space to the sum of deformations on three mutually orthogonal planes.

292

A. Torii et al.

On each plane, deformation is expressed as 2 α α pα i (t + 1) − pi (t) = c(t)D v i (t).

(31)

In this paper, we set c(t) = we−λt , where λ and w are positive constants. Next, employing the same concept for planar shape deformation, we replace the righthand side of eq. (31) with the averages of the second-order derivative with different grid lengths. Finally, also using the same concept for resolution conversion for planar binary shapes, we compute the high-resolution binary objects. 4.4

Deformation of Terrain

For vector pij = (i, j, f (i, j)) , i, j = 1, 2, · · · , n, setting D12 pij = pi+1 j − 2pij + pi−1 j , D22 pij = pi j+1 − 2pij + pij−1 ,

(32)

we have the equation D12 pij + D22 pij = (0, 0, ∆4 f (i, j)) ,

(33)

where ∆4 is the 4-connected discrete Laplacian operation for the two-valued discrete function f (i, j). Assuming that function f (i, j) is a function of time t, we have the relation pij (t + 1) − pij (t) = (0, 0, f (i, j, t + 1) − f (i, j, t)) .

(34)

Therefore, the equation pij (t + 1) − pij (t) = c(D12 pij + D22 pij ),

(35)

where c is a constant, implies the equation f (i, j, t + 1) − f (i, j, t) = ∆4 f (i, j, t). These mathematical properties of the deformation for discrete terrain data lead to the conclusion that our deformation for terrain data based on the vertex Laplacian operation for a polyhedron is equivalent to the deformation by the linear diffusion equation, if we consider the height of each point as the gray-level of each point. Furthermore, eq. (35) implies that, in the numerical computation, a serial application of the operations D12 and D22 is possible. For the deformation of the discrete height model of a topographical map, it is necessary to preserve the height data of a map, since it depends on the measurement of heights. Therefore, we add new control points between the data points of the discrete height model. Then, using these points as control points, we deform the model to derive a high-resolution level map. Since a terrain is an open surface, its slices along the x and y axes are open curves in a space. Therefore, we consider the boundary conditions of the Laplacian operator to be open curves. The Laplacian operator for open curves is derived in the next section. The new control points are generated according to the following rules.


293

Fig. 3. Control Point Generation 1. For i, j = 1, 2, · · · , n − 1, pij + = (i + k, j + k, f (i, j)) , where 0 < k < 1. 2. For i, j = 2, 3, · · · , n, pij − = (i − k, j − k, f (i, j)) , where 0 < k < 1. 3. For i, j = 1, 2, · · · , n − 1, and if |f (i + 1, j) − f (i, j)| ≥ 2k, then pij α = (i,j)| (i + k, j + k, f (i, j) + αl) , where α = 1, 2, · · · , l − 1 and l = |f (i+1,j)−f . k 4. For i, j = 1, 2, · · · , n − 1, and if |f (i, j + 1) − f (i, j)| ≥ 2k, then pij β = (i + k + βm, j + kβm, f (i, j) + βm) , where β = 1, 2, · · · , m − 1 and m = |f (i,j+1)−f (i,j)| . k We extract every point of intended resolution from the estimated discrete curves of each decomposed row and column and reconfigurate it as a form of the DTM matrix. 4.5

Deformation of Curve in the Discrete Space

Let pi be the centroid of a voxel i. Considering the 6-neighborhood in the discrete space, we assume that a curve in the discrete space is a path. Since we consider the 6-connectivity in the discrete space, we have the following proposition for curves in space. Proposition 1 Setting pi for i = 1, 2, · · · , n to be a path in the space, there exist at most two points in the neighborhood of each point. If a curve is closed, there exist two points in the neighborhood of each point. If a curve is open, there only exists one point in the neighborhood of each endpoint.

294

A. Torii et al.

Setting P (t) = [p1 (t), p2 (t), · · · , pn (t)] for points on a curve, the discrete deformation is described as P (t + 1) − P (t) = cLP (t), where c is a positive constant and L is given as    −1 1 · · · −2 1 · · · 1  1 −2 · · · 0   1 −2 · · ·     . . . ,   0 .. . . . ..   0 .. . . . 1 · · · 1 −2

(36)

1 0 .. .

   , 

(37)

1 · · · 1 −1

for closed and open curves, respectively. Next, employing the same concept for planar shape deformation, we replace the right-hand side of eq. (36) with the averages of the second-order derivative with different grid lengths. Finally, using the same concept with resolution conversion for planar binary shapes, we compute the high-resolution binary objects. In Figure 4, we show our concept for the deformation of boundaries of a curve in the discrete space. The modification preserves the positions of both endpoints p1 and pn−1 , then computes the average of global configuration vector v ni as i+1 1 k if i < n+1 k=1 ui (t), 2 , (38) v ni (t) = i 1 n−i k n+1 u (t), if i ≥ k=1 i n−i 2 for i = 1, 2, · · · n − 1. Using these modifications, discrete space curves are estimated via the same method for a planar polygonal curve.

Fig. 4. Estimation of a discrete spatial curve


5

295

Numerical Examples

In Figure 5, we show examples for the generation of high-resolution images from digital binary images using the algorithm derived in the previous sections. Figure (a) shows the binary image obtained from (b) using the sampling scheme described in section 3. Figures (c) and (d) show the estimated boundary curve and the original boundary curve, respectively. In figure (e), the region in which the original boundary exists is shown. Points marked by + in figure (f) express lattice points on the boundary of the support of the shape shown in figure (a). Points marked by + in figure (g) express the configurations of control points after the deformation of the polygonal curve shown in (g). The curve in figure (h) is the estimated boundary curve after the reduction of the control points. Points marked by + in figure (i) represent control points for the expanded images. These figures show that our algorithm accurately generates a high-resolution image from a given digital image. Here, we set parameters τ = π/3, φ = π/18, and T = 100, where T is the maximum number of iterations for the flow computation. Furthermore, we set α = 2/5, β = 2/5, and n0 = 332. Then we have n = 198 for α(t) = αt−2 and β(t) = βt−2 , and n0 and ne are the numbers of initial control points and the reduced control points, respectively. In these examples, we could reduce the number of sample points. Figure 6 shows the results of resolution conversion for geometric data. Figures in the left, middle and right columns are high-resolution original data, lowresolution data, and reconstructed high-resolution data, respectively. For the quantitative evaluation of the reconstructed data, we have evaluated the ratio d = |V ∩ Vˆ |/|V | × 100, where |V ∩ Vˆ | and |V | are volumes of the original data and the common regions of the original and reconstructed data, respectively. For these examples, d-s are around 85. Since the method utilizes curvature flow, the reconstructed shapes are smaller than the original ones. We set λ = 0.05, w = 1, α(t) = 1 β(t) = 1/10. Furthermore, we stop the iteration if e−βt ≤ 5/100. The original objects are 100×100×100 voxels. The low-resolution images are sampled to 50 × 50 × 50 voxels, that is, the low-resolution objects are sampled by eighttimes-larger voxels. Assuming these low-resolution objects as the input objects, our algorithm yields the high-resolution object in the right-hand column. In Figure 7, (a) shows the original open curve. Figure 7 (b) shows the lowresolution curve obtained from the curve in (a). Figure 7 (c) shows the curve estimated from the curve in (b). Table 1 shows the parameters of this experiment and the lengths of the curves in Figure 7 (a) and (c), respectively. The length in (a) is 253.84 units and the length in (c) is 248.28 units. For the quantitative evaluation of the reconstructed curve, we have evaluated the ratio LL × 100 = 96.5, where L and L are the lengths of the original and the estimated curves, respectively. Figure 8 shows the results of the inverse quantization of terrain data. The original topographical map in Figure 8 (a) is from a 1/5000 digital height map of Japan. Figure 8 (b) shows a lower resolution map obtained from (a) by reducing the data points of the original one to 1/16.

296

A. Torii et al.

(a)

(b)

(c)

0

20

y

40

60

80

100 0

20

40

60

80

100

x

(d)

(e)

0

(f)

0

0 100

40

200 300 y

40 y

20

y

20

60

60

80

80

100 0

100 0

400 500 600 700

20

40

60

80

100

20

40

60

x

x

(g)

(h)

80

100

800 0

200

400 x

600

800

(i)

Fig. 5. Resolution Conversion for Geographical Data

The appearance of the reconstructed topographical map in figure (c) is better than that of the one interpolated in figure (d) by B-spline functions. For the comparison, we interpolated slice curves using splines.


(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

297

Fig. 6. Resolution Conversion for Geometric Data

Since the positions and heights of local minima and maxima are cue-features for the evaluation of roughness of the interpolated functions, we have evaluated

298

A. Torii et al.

Table 1. Parameters of a Spatial Curve iteration (T) α(t) β(t) τ φ length of (a) length of (c) matching rate (%)

iteration (T) α(t) β(t) τ φ dif . (a) & (c) dif . (a) & (d)

100 3/t2 1/2t2 π/3 π/18 253.84 248.28 96.5

the sum ratio rαα (i, j) =

Table 2. Parameters of a Terrain

|f¯αα (i, j) − fαα (i, j)| × 100 |fαα (i, j)|

such that rαα = (i,j)

rαα (ij)

10 45/t2 15/t2 π/3 π/18 1.229288 1.691865

(39)

(40)

in the region of interest

for points fαα (i, j) = 0, where α ∈ {x, y}. Table 2 shows the sum of rαβ for all points in the region of interest and the parameters.

6

Conclusions

In this paper, using a morphological operation and curvature flow, we constructed an algorithm for the estimation of a smooth boundary curve of the original image from an isotactic polygon which is the boundary of connected pixels in a plane and voxels in a space from given digital objects through which the original boundary curve should pass. Using this estimation of the boundary curve, we can generate binary digital objects at any resolution. Numerical examples confirmed the suitable performance of the proposed method. A two-dimensional gray-level image is a topographical map and generally, an n-dimensional graylevel image is a topographical map in (n + 1) dimensional space, if we consider the gray levels as the height data of points. Therefore, using our method for the terrain data, we can obtain high-resolution gray-level images of two- and three-dimensional variables.

Acknowledgements The implementation of the boundary extraction algorithm in 3D objects is based on the thesis of K. Tatara, and parts of Section 3 are based on the thesis of A. Ito, both are at Chiba University. Parts of Section 4 are based on the research performed by A. Imiya during his visit to the University of Hamburg in 1997 and 1998. He expresses thanks to Professor Dr. Ulrich Eckhardt and the Department of Applied Mathematics of University of Hamburg for their hospitality.


"ispring.txt"

40 35 30 25 20 15 10 5 0

"spring2.txt"

40 35 30 25 20 15 10 5 0 20

20

15 5 -15

-10

0 -5

-5 0

5

-10 10

15

-15 20

-20

(a)

"opensp.txt"

400 350 300 250 200 150 100 50 0 150

15

10 -20

299

100

10 -20

50

5 -15

-10

0 -5

-5 0

5

-150

-100

-10 10

15

-15 20

0 -50

0

-20

(b)

50

100

150

-50 -100 -150 200

(c)

Fig. 7. Resolution Conversion for a Curve in Discrete Space

(a)

(b)

(c)

(d)

Fig. 8. Resolution Conversion for a Terrain

300

A. Torii et al.

References [1] Imiya, A., Eckhardt, U., Curvature flow in discrete space, Lecture Notes in Computer Science, 2001, Vol. 2243, 229-249. 282, 291 [2] Isomichi, Y., Inverse-quantization method for digital signals and images: Pointapproximation type, Trans. IECE, 63A, 815-821, 1980. 282 [3] Terzopoulos, D., The computation of visible-surface representations, IEEE, Trans, PAMI, 10, 417-438, 1988. 282, 283 [4] Lu, F., Milios, E. E., Optimal spline fitting to plane shape, Signal Processing 37 129-140, 1994. 283 [5] Wahba, G., Surface fitting with scattered noisy data on Euclidean D-space and on the sphere, Rocky Mountain Journal of Mathematics, 14, 281-299, 1984. 282, 283 [6] Wahba, G., Johnson, D. R., Partial spline models for the inclusion of tropopause and frontal boundary information in otherwise smooth two- and threedimensional objective analysis, J. Atmospheric and Oceanic Technology, 3, 714725, 1986. 283 [7] Chen, M. H., Chin, R. T., Partial smoothing spline for noisy+boundary with corners, IEEE Trans. PAMI, 15, 1208-1216, 1993. 283 [8] Paglieroni, D., Jain, A. K., Control point transformation for shape representation and measurement, Computer Graphics and Image Processing, 42, 87-111, 1988. 283 [9] Medioni, G., Yasumoto, Y., Corner detection and curve representation using cubic B-spline, Computer Graphics and Image Processing, 39, 267-278, 1987. 283 [10] Langridge, D. J., Curve encoding and the detection of discontinuities, Computer Graphics and Image Processing, 20, 58-71, 1982. 283 [11] Daubechies, I., Guskov, I., Sweldens, W., Regularity of irregular subdivision, Constructive Approximation, 15, 381-426, 1999. 283 [12] Daubechies, I., Guskov, I., Schr¨ oder, P., Sweldens, W., Wavelets on irregular point sets, Phil. Trans. R. Soc. Lond. A, to be published. 283 [13] Imiya, A., Ito, A., Kenmochi, Y., Inverse quantization of digital binary images for resolution conversion, LNCS 2106 Springer: Berlin, 426-434. 283 [14] Boechm, W., Prautzsch, H., Numerical Methods, AK Peters, MA, 1992. 287 [15] Fisher, M., On hearing the shape of a dram, Journal of Combinatorial Theory, 1, pp.105-125, 1966. 290 [16] Collatz, L.,Eigenwertaufgeben mit Technichen Anwendungen, Akademische Verlagsgesellshaft, Leipzig 1949. 290 [17] Collatz, L., Sinogowitz, U., Spektren sndicher Grafen, Abhandlungen aus dem Mathematischen Seminar der Universtaet Hamburg, Band 21, pp.63-77, 1957. 290 [18] Collatz, L.,Numerische Behandlung von Differntalgleichhaungen, Springer, Berlin, 1955. 290 English translation: The Numerical Treatment of Differential Equations, Springer, Berlin, 1960.

Discrepancy-Based Digital Halftoning: Automatic Evaluation and Optimization Kunihiko Sadakane, Nadia Takki Chebihi, and Takeshi Tokuyama Graduate School of Information Sciences, Tohoku University Aoba-ku, Sendai, Miyagi, 980-8579 Japan {sada,nadia,tokuyama}@dais.is.tohoku.ac.jp

Abstract. Digital halftoning is a problem of computing a binary image approximating an input gray (or color) image. We consider two problems on digital halftoning: mathematical evaluation of a halftoning image and design of optimization-based halftoning algorithms. We propose an efficient automatic evaluation system of halftoning images by using quality evaluation functions based on discrepancy measures. Our experimental results on the evaluation system infer that the discrepancy corresponding to a regional error is a good evaluation measurement, and thus we design algorithms to reduce this discrepancy measure. Keywords: Digital halftoning, Discrepancy, Rounding, Quality evaluation.

1

Introduction

Digital halftoning is an active research field in image processing. It is a wellknown technique for computing a binary image approximating an input gray image, so that the binary image looks similar to the input one. The main motivation of halftoning process is to keep impression of gray-level variation of the original image in displaying it on binary devices such as laser printers and fax machines. Therefore there is a need to design a halftoning algorithm with a good visual quality. So far, a large number of techniques have been presented in this field [6]. However those approaches don’t propose reasonable criteria for evaluating how it is similar a halftoning image to its original one and how its quality is good. Up to now, the most popular criterion to judge the quality is human vision system; this is inconvenient, since human’s judgement depends on individual sense of beauty. Therefore, it is desired to establish an automatic evaluation system (instead of the “human eye’s judgment”), and to handle the digital halftoning problem fully mathematically or algorithmically [1]. Consequently, the following questions should be answered in order to design a good digital halftoning system: – How to give a reasonable mathematical definition of a good halftoning image? – How to evaluate quality of a halftoning image automatically? T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 301–319, 2003. c Springer-Verlag Berlin Heidelberg 2003

302

Kunihiko Sadakane et al.

– How can we compute a good halftoning image with respect to the above mentioned quality evaluation? That is to say, we need to convert the digital halftoning problem as a rounding problem with a good optimization criterion. In order to evaluate quality of halftoning image automatically, we need to select a suitable quality measurement. Discrepancy is a hopeful candidate of mathematical measurement concept representing quality of halftoning images [1, 2, 11]. Intuitively, we consider a family F of regions in the pixel grid plane and a function f on F whose value at R ∈ F indicates the “difference” between an input picture and its output halftoning image within R. However, relation between quality of halftoning and discrepancy values has not been studied well. Moreover, there are several different discrepancy measures, and we need to know which discrepancy measures give quality measurements compatible with human’s judgment. In the literature, choice of a discrepancy measure is done from the aspect of computational complexity; that is, we choose a discrepancy measure for which the halftoning image minimizing it can be computed efficiently (say, in polynomial time). For example, discrepancy measures associated with unimodal hypergraphs are studied [2, 4]. In other words, it has not been systematically discussed how to choose a discrepancy measure from the aspect of quality of outputs. In this paper, we give a prototype system for evaluating a halftoning output based on several discrepancy measures utilizing an efficient algorithm for computing discrepancy values. By applying the system to outputs of popular halftoning methods, we examine which discrepancy measures are suitable for quality evaluation. Quality evaluation measures should be selected so that they detect difference between halftoning image and the original image locally near each pixel. Moreover, a quality evaluation system should detect scars and patterns in halftoning outputs that are not contained in the original pictures. In our experiment, we consider some natural measurement functions on region families consisting of fixed-size squares; moreover, our principal approach is also effective to a wide range of useful quality measures. Our experimental results using the evaluation system infer that the regional discrepancy on k × k rectangles is nicely compatible with human’s judgment. Thus, we next focus on devising a halftoning algorithm to output an image with small regional discrepancy. Unfortunately, computing the halftoning image minimizing the regional discrepancy for this family is NP-hard, and even its approximation is theoretically hard [3, 2]. Therefore, we provide several heuristic algorithms based on sequence rounding technique, for which we transform the pixel grid plane into a one-dimensional array by filling the grid plane with ordering curves and consider each curve as a sequence. We show experimental results of the proposed algorithms, in terms of halftoning outputs and their quality evaluations.

Discrepancy-Based Digital Halftoning

303

Although color image halftoning is practically important, we focus on monochromatic images, since a color image can be considered as an overlay of three monochromatic images representing RGB colors.

2

Matrix Rounding Problem and Quality Evaluation

We formulate the digital halftoning problem into a matrix rounding problem as follows: Let ai,j (i, j = 0, 1, . . . , n − 1) be a real value in the range [0,1] representing a gray level image at the (i, j)-th pixel of a monochromatic image displayed on an n × n pixel grid, where ai,j is the fraction of the gray level to the maximum possible brightness. Let bi,j be the binary value representing the color of the (i, j)-th pixel in the output halftoning image, where 1 represents white and 0 represents black. Consider the n-by-n matrices A = (ai,j )i,j=0,1,...,n−1 and B = (bi,j )i,j=0,1,...,n−1 . In this formulation, digital halftoning converts the real matrix A to a binary matrix B = (bi,j ); thus, B is obtained by rounding entries of the matrix A suitably. 2.1

One-Dimensional Discrepancy

Let us start with a one-dimensional analogue of the digital halftoning problem. Let A = (a0 , a1 , ..., an−1 ) be a sequence of real numbers between 0 and 1 and B = (b0 , b1 , ..., bn−1 ) be a binary sequence. For a subinterval I ⊂ [0, n − 1], let ai − bi is the absolute difference between A and B within I. Given i∈I i∈I a family I of itnervals, we define a distance between A and B based on the l∞ discrepancy by: ∞ ai − bi DistI (A, B) = max I∈I i∈I

i∈I

Dist∞ I (A, B)

satisfies the axiom of distance. If I is the It is clear that the above family of all intervals, we can compute Imax attaining the maximum absolute difference in O(n) time by using a simple scanning algoirthm. This is a classical topic in algorithm and programming, and the scanning algorithm is introduced as Kadane’s scanning algorithm in the famous Programming Pearl article by Bentley [5]. 2.2

Measurement for Matrix Rounding

Although Dist∞ (A, B) is a good distance between sequences, we need to measure the difference between two matrices. In a halftoning system, an output image B must look similar to the original image so that difference between A and B can be ignored by human’s eye as negligible noise; in other words, it should be avoided that B has patterns meaningful for human’s eye (e.g., line segments or curve segments) that are not in A. Such patterns are often called unexpected patterns.

304


For detecting an unexpected pattern that resembles to a horizontal (or vertical) line segment, we can use one-dimensional discrepancy considered in [10]. Indeed, we can detect such an unexpected pattern by applying the scanning algorithm for each column and row of the image. However, unexpected patterns need not resemble to such axis aligned line segments, and hence it is not sufficient to use one-dimensional discrepancy for evaluating halftoning images. Human being cannot detect difference of color of each pixel between input and ouput images; however, the difference of texture and/or average color in a region can be detected. Thus, our idea is to consider k × k subregions in the pixel grid, and judge that an output image is a good halftoning if it resembles to the input image in every such subregion according to a given optimization criterion. Therfore, for automatically judging whether a given output image B is good or not, we define a quality measurement function f defined on the set Fk×k of all k × k subregions and indicating discrepancy between A and B, and then design an efficient algorithm to enumerate regions R with large f -values. 2.3

Discrepancy Measures in a k × k Region.

We define six objective functions based on discrepancy measures for evaluating errors within a region R = I × J =∈ Fk×k , where I and J represents intervals of lengths k of row indices and column indices, respectively. – Absolute regional error: ARE ARE(R) = |

(ai,j − bi,j )|

(i,j)∈R

gives the difference of total gray levels between an input and its ] halftoning output within a region R. This is considered to be a quite natural difference, since it gives the difference of brightness between input and output within the region. – S quared pixel error: SPE (ai,j − bi,j )2 SP E(R) = (i,j)∈R

determines the aggregated deviation between each pixel’s gray value and its binary output. – S quared row error: SROE ( (ai,j − bi,j ))2 SROE(R) = i∈I j∈J

measures the aggregated deviation between the input gray level of each row and its output in R. We expect that it can capture horizontal stripe textures. If its value is large, we can detect unexpected horizontal stripes in R, especially of unexpected wide horizontal line segments.


305

– S quared column error: SCOE SCOE(R) =

( (ai,j − bi,j ))2 j∈J i∈I

measures the aggregated deviation within R between the input gray level of each column. We expect that it can capture vertical stripe textures. – S quared (downward and upward) diagonal errors: SDDE and SADE SDDE and SADE represent diagonal and off-diagonal features, respectively. We expect that they can capture diagonal/off-diagonal stripe textures. For R = [1, k] × [1, k], they are defined by ( (ai,j − bi,j ))2 SDDE(R) = −k+1≤y≤k−1 i−j=y,(i,j)∈R

and SADE(R) =

(

(ai,j − bi,j ))2

2≤y≤2k i+j=y,(i,j)∈R

2.4

Algorithm for Computing Evaluation Functions

Given a halftoning image, among regions in Fk×k , we want to list up regions for which the above functions have large values. We want to investigate the relation between local quality of the worst k × k region and the global quality of the image for the human’s eye, and also find out which of the six functions plays as the best indicator function for quality of an output. Proposition 1. We can compute ARE, SPE, SROE and SCOE for all the regions in Fk×k in O(n2 ) time using O(n) working space. Proof: We describe our method for ARE, since the other cases can be solved analogously. Instead of ARE, we consider the regional error RE(R) = (i,j)∈R ai,j − bi,j . Of course, ARE can be computed from RE in O(1) time for each R. Let Ri,j be the k × k subregion whose north-west corner pixel is at (i, j). We can computes RE values for Ri,1 for all i = 1, 2, . . . , n − k in O(nk) time. Suppose that we have computed RE values for Ri,ν−1 for all i = 1, 2, . . . , n − k. Now, we show that we can compute Ri,ν for all i = 1, 2, . . . , n − k in O(n) i+k i+k time. g(i, ν) = RE(Ri,ν ) − RE(Ri,ν−1 ) = t=i (at,ν+k − bt,ν+k ) − t=i (at,ν−1 − bt,ν−1 ). Naively, g(i, ν) is computed in O(k) time. However, g(i, ν) = g(i − 1, ν)+ {ai+k,ν+k − bi+k,ν+k − ai+k,ν−1 + bi+k,ν−1 } − {ai−1,ν+k − bi−1,ν+k − −ai−1,ν−1 + bi−1,ν−1 }. Thus, it can be computed in O(1) time from g(i − 1, ν). Therefore, the time complexity is O(n2 ) and the space complexity is O(n) (except space for outputting the function values).

Unfortunately, SDDE and SADE are more difficult to compute, since the diagonal segments in a k × k matrix have different lengths (from 1 to k), and we need to consume O(k) time instead of O(1) to update the information if we design a similar algorithm as above.

306


Proposition 2. SDDE and SADE for them can be computed in O(kn2 ) time using O(n) working space. We note that if we consider k × k rotated squares by 45 degree in the pixel grid instead of axis parallel squares, we can compute SDDE and SADE for them in O(n2 ) time 1 , although we have not done experiment on that version. 2.5

Evaluation of Known Halftoning Algorithms

By using our scanning algorithm we evaluate the quality of some well-known halftoning techniques. Througout all of our experiments, we use the Lena’s picture for the target gray-level picture, and the matrix size is set to be 512 × 512. Implementation have been done using C language, and the computing environment is a Linux PC with a Pentium III processor. We first describe the halftoning techniques briefly, together with human eye’s judgement of their outputs. Threshold Rounding. Given a matrix A of real numbers in range [0, 1], a naive method to obtain a binary matrix B approximating A is to round each entry ai,j to 1 if and only if it is larger than or equal to a fixed threshold value. In particular, we use 0.5 for the threshold value, so that the method minimizes the squared pixel error SPE. However, if we apply it to digital halftoning, its output (Figure 1) loses too much information since a uniform gray region tends to be transformed into totally white or totally black region, depending on slight difference of the brightness. For the human’s eye, we can capture the outline of the image, but lack of smoothness gives impression of bad quality. Randomized Rounding. In algorithm theory and optimization theory, a powerful tool to obtain an approximation algorithm with a mathematically provable quality is the randomized rounding [9]. This simple method rounds independently each input entry ai,j in a probabilistic fashion; precisely speaking, bi,j is 1 with probability ai,j and 0 with probability (1 − ai,j ). Unfortunately, an output (Figure 2) of this method suffers from apparent grainess that makes the image blurred. As a halftoning algorithm, randomized rounding is often a poorer method than the threshold rounding since its output is harder for human to capture the outlines of objects in the image. Ordered Dither. The ordered dither method [6] operates independently on blocks. It employs a spatial lookup-table called dither matrix D (see Figure 3), to perform the quantization of brightness values. We use a dither matrix of size 8×8 in our experiment, although larger size matrices are often used in practice. Pixel 1

This was pointed out by an anonymous reviewer.


Fig. 1. Threshold rounding

307

Fig. 2. Randomized rounding

positions (i, j) of an image are divided into 64 classes. Each pixel’s coordinate (i mod 8, j mod 8) belongs to a class k stored in D. The entire gray level image is partitioned into blocks of size 8 × 8, and each gray level pixel is compared to the threshold value at the corresponding position in the dither matrix. We process all pixel positions (i,j) belonging to class k and define the halftoned pixels as follows: If ai,j ≥ k+0.5 64 , bi,j = 1 else bi,j = 0. In other words, each pixel is rounded by using the corresponding entry of the dither matrix as the threshold. Its output (Figure 4) is much better for human’s eye than those of threshold rounding and randomized rounding. A defect of this method is that it generates visible texture inherited from the dither matrix, and thus its output image gives an artificial impression.

0 48 12 60 3 51 15 63

32 16 44 28 35 19 47 31

8 56 4 52 11 59 7 55

40 24 36 20 43 27 39 23

2 50 14 62 1 49 13 61

34 18 46 30 33 17 45 29

10 58 6 54 9 57 5 53

42 26 38 22 41 25 37 21

Fig. 3. Dither matrix

Fig. 4. Ordered dither

308


Fig. 5. One dim. error diffusion

Fig. 6. Two dim. error diffusion

One Dimensional Error Diffusion. Error diffusion is the algorithmic approach that has been extensively studied. The one-dimensional error diffusion algorithm computes a binary sequence B = (b0 , b1 , . . . , bn−1 ) from A = (a ) greedily for j = 0, 1, 2, . . . , n − 1 such that the inequality 0 , a1 , . . . , an−1 j i=0 (ai − bi ) ≤ 0.5 holds for each j. We process each row of the matrix by using the above algorithm. Its output (Figure 5) is almost comparative to that of ordered dither. It has no systematic pattern caused by dither matrix, but it is suffering from vertical wave (or stripe) patterns. Also, we can see some vertical linear scars. The reason of occurrence of such patterns is as follows: We apply the same greedy algorithm for each row and thus adjacent output rows have similar periodical feature. This causes unexpected scars and vertical wave patterns. Two Dimensional Error Diffusion. Two-dimensional error diffusion is a neighbor halftoning process given by Floyd and Steinberg [7]. It use a partition of 1 into four constants α, β, γ, δ to diffuse the error, which is directed proportionately to nearby elements whose B values have not yet been computed. Floyd-Steinberg suggested taking (α, β, γ, δ)=(7/16,3/16,5/16,1/16). The algorithm scans a matrix row-wise from its top row to the bottom one, and rounds entries greedily. It applies the threshold rounding for rounding each pixel, but the difference is that it propagates the side effect of the rounding to neighboring pixels. Suppose that we have rounded an entry ai,j into bi,j , and let di,j = ai,j − bi,j . Then, we update its neighborhood entries (which have not been rounded yet) as follows: ai,j+1 = ai,j+1 +α·di,j , ai+1,j−1 = ai+1,j−1 +β·di,j , ai+1,j = ai+1,j + γ · di,j , and ai+1,j+1 = ai+1,j+1 + β · di,j . As seen in Figure 6, two-dimensional error diffusion leads to higher halftoned image quality, and contouring artifacts are minimized. Although it still produces some vertical scars and artifacts looking like zebra stripes, its visual quality is best among the methods mentioned above.


309

Table 1. Maximum values of measurment functions Methods Threshold Randomized Ordered Dither 1D diffusion 2D diffusion

2.6

Row 100 37 107 0.99 17

Col 206 38 134 159 68

ARE 12.29 11.00 5.00 4.71 3.24

SPE 6.15 9.33 5.00 7.42 6.90

SROE SCOE 30.19 30.29 31.03 27.99 14.88 15.17 4.44 36.21 12.79 32.61

SDDE 20.56 23.61 24.77 22.34 22.25

SADE 20.51 25.27 24.03 22.53 22.13

Automatic Evaluation of Typical Halftoning Methods

In this section we show experimental results on the validity of our quality evaluation system by applying it to the outputs of the halftoning algorithms described in the previous subsection. Table 1 shows the maximum values over all regions in Fk×k for each of our measurment functions, where “Row” and “Col” represents maximum of 1dimensional discrepancies of rows and columns, respectively. The ranking of halftoning methods is highly dependent on choice of measurement functions. This implies that it is extremely important to find a suitable measurement function. Clearly, among these functions, ARE is most compatible with human’s judgment. The maximum values might be attained at few singular parts of the pictures; thus, one may worry that the maximum values given in Table 1 may fail to describe the visual feature of the outputs. We consider L1 and L2 measures which will be discussed in the next subsection for removing this anxiety. Due to space limitation, we omit to give the experiment result in this version; It will be give in a more detailed version [8], in which we also consider distribution of large values of the measurement function to confirm that the maximum values given in Table 1 also indicate the tendency of distribution of large values in the picture. In [8], features of the texture of the region maximizing each evaluation function are also reported for each halftoning method. 2.7

Variations of ARE

We have observed that ARE is very important for evaluating quality of halftoning measure. Thus, we examine ARE for different values of k, and also consider variants of ARE. We have considered the maximum value of ARE(R) over all regions R ∈ Fk×k . This is called the L∞ regional measure, and denoted by AREk∞ from now on; that is, AREk∞ = max ARE(R). R∈Fk×k

Similarly, we define AREk1 =

R∈Fk×k

|ARE(R)|/|Fk×k |

310


and AREk2 = [

|ARE(R)|2 /|Fk×k |]1/2 .

R∈Fk×k

They basically correspond to L1 and L2 regional measures, respectively. Tables 2, 3, and 4 give tendency of quality of halftoning methods for these three norms. For the L∞ ARE measure, 2D diffusion method is best for the all ranges of 5 ≤ k ≤ 50. However, for the L1 and L2 measurements, phase transition occurs around k = 30. If the resolution of the picuture becomes finer, ARE for a larger k becomes more important, since the resolution of human vision is fixed. Thus, for a future halftoning system, the authors guess that a method which minimizes ARE for a larger k will be more important.

3

Minimizing ARE by Using Global Roundings

Our exerimental results show that ARE discrepancy measure works well for evaluating halftoning quality, and we can also observe regions with large ARE values tend to have bad output patterns that human eyes can perceive [8]. Based

Table 2.

The infintiy norm for k = 5, 10, 20, 30, 40, 50

∞ Method ARE5∞ ARE10 Threshold 12.29 48.23 Randomized 11.00 21.16 Dither 5.00 7.42 1D diffusion 4.71 8.78 2D diffusion 3.24 5.98

Table 3. Method Threshold Randomized Dither 1D diffusion 2D diffusion

∞ ARE20 186.67 42.93 13.17 15.45 10.05

∞ ARE30 412.70 67.13 21.52 19.49 13.40

∞ ARE40 ARE ∞ (50) 726.91 1127.96 92.13 105.98 25.05 36.56 24.85 25.54 16.44 20.81

The L1 norm for k = 5, 10, 20, 30, 40, 50

1 1 1 1 ARE51 ARE10 ARE20 ARE30 ARE40 ARE 1 (50) 7.34 27.21 96.80 196.80 319.42 459.11 1.84 3.68 7.32 10.84 14.25 17.83 0.98 1.19 2.15 3.70 4.61 6.33 0.74 1.12 1.69 2.15 2.56 2.90 0.61 0.85 1.45 2.19 3.08 4.06

Table 4.

The L2 norm for k = 5, 10, 20, 30, 40, 50

Method Threshold Randomized Dither 1D diffusion 2D diffusion

2 2 2 2 2 ARE52 ARE10 ARE20 ARE30 ARE40 ARE50 7.87 29.66 108.39 224.13 368.18 534.91 2.31 4.63 9.18 13.70 18.23 22.75 1.24 1.50 2.77 4.67 5.85 7.98 0.97 1.48 2.26 2.89 3.43 3.89 0.76 1.08 1.84 2.76 3.88 5.11


311

on it, we hope that a halftoning method minimizing ARE value will create good halftoning images for human being. However, it is known to be NP-hard [3] to compute the binary matrix B minimizing AREk∞ even for k = 2, and hence we need to design a heuristic or approximation method. In this section, we modify the one-dimensional error diffusion algorithm by using the concept of global rounding given by Sadakane et. al. [10]. Consider a rounding B = (b0 , b1 , . . . , bn−1 ) of A = (a 0 , a1 , . . . , an−1 ). Given an integral subinterval I of [0, n − 1], |B(I) − A(I)| = | i∈I (bi − ai )| is called the interval error of the rounding within I. For a given family F of intervals over a sequence, the rounding B is called F -global rounding if |A(I) − B(I)| < 1 holds for every I ∈ F. The output of the 1-D error diffusion method is a global rounding for any family F of intervals. However, we have seen that we have vertical scars and wave patterns if applying the 1-D error diffusion method to every row independently, because of syncronization. A key fact is that there are exactly n + 1 global roundings (under a nondegeneracy condition) if we consider the set of all subintervals as the interval family. Moreover, we can design an efficient algorithm to enumerate all of them in O(n2 ) time [10]. Our strategy is, instead of 1-D error diffusion output, we choose a rounding from the set of all possible global roundings at each row, and try to break the syncronization, hoping to reduce scars and wave patterns consecuently. By selecting a global rounding randomly in each row, we can reduce the generation of vertical stripe patterns as shown in the output image Fig. 7 from the one dimensional error diffusion output (Fig. 5). Neglecting long intervals, we define It -global rounding where we consider the set of all subintervals of lengths at most t < n. This enables to reduce the O(n2 ) time complexity of the above method to O(tn) time. The set of all It -global rounding forms the set of all source-sink paths of a DAG of size O(nk) [10], and hence we can select an It -global rounding in an uniformly random fashion in O(tn) time; moreover, we can give side constraints or optimization conditions so that we can find the optimal It -global rounding by applying a shortest-path algorithm. The output images (for n = 512 and t = 9) are given in Figure 7, and we can see that they have few vertical scars. The computation time for all of n rows is O(tn2 ), which is less than 0.5t seconds in our experiment. Unfortunately, we see that we can see wave patters, and the waves are larger that that of 2-D error diffusion; thus, they give impression that the outputs are slightly rough. We evaluate quality of outputs of the method based on the It -global rounding for different values of t. Table 5 shows the value of our evaluation functions for the It -global rounding method. The “time” column indicates the computation time (seconds). The values of discrepancy measures tend to decrease slightly if we increase t, although the visual difference (via human eyes judgment) between the quality of output images does not significantly depend on the choices of t if t ≥ 7. For the 512 × 512 picture, t = 7 seems to be a threshold of the interval length t.

312


Fig. 7. In -global rounding (left) and I9 global rounding (right) The values of ARE5∞ do not depend on t much; however, for a larger k, AREk∞ depends on t heavily, as seen in the following table. We can observe that, for a larger k, the values are considerably smaller than the method given in the previous section if we take a large t.

4

Rounding Based on Ordering Curves

In the previous subsection, we consider a halftoning method combining sequence roundings of rows by introducing randomization. Although ARE values (for k ≥ 20) are improved, the method is clearly worse than the 2-D error diffusion method in terms of judgement by human’s vision for the Lena’s picture in the 512 × 512 resolution. An alternative approach is to apply sequence rounding along an ordering curve (often called space filling curve) that gives a sequential ordering of the pixels so that the consecutive pixels in the ordering is adjacent to each other in

Table 5. Evaluation of It -global rounding outputs t 3 5 7 9 11 15 19 23 n

Row 64 25 16 9 8 5 3 2 0.996

Col ARE5∞ 171 9.18 140 7.29 134 4.82 154 4.42 139 4.21 132 4.04 121 4.25 123 4.09 40 4.36

SPE 9.09 8.13 7.53 7.26 8.02 7.87 7.56 7.81 7.37

SROE SCOE 18.08 42.14 10.64 36.97 4.64 36.64 3.95 34.02 3.86 35.00 3.56 36.51 3.61 35.83 3.83 36.96 3.80 34.87

SDDE SADE time 27.50 31.30 1.70 23.41 23.04 2.42 22.55 25.19 2.87 22.44 23.84 3.72 22.12 23.14 4.18 22.40 25.08 5.56 22.24 24.28 6.85 23.03 25.84 9.99 23.11 22.51 30.43


313

Table 6. Dependency of t for various k t ARE5∞ ∞ ARE10 ∞ ARE20 ∞ ARE30 ∞ ARE40 ∞ ARE50

3 9.18 23.91 80.24 157.75 272.37 398.37

5 6.00 13.66 40.24 83.52 127.91 173.52

7 4.72 9.89 29.99 59.40 99.61 126.42

9 4.42 9.16 20.16 41.97 61.09 86.65

11 4.25 8.21 17.57 35.77 50.09 70.39

15 4.67 6.72 13.69 24.33 33.32 46.91

19 4.62 7.34 11.08 18.65 26.42 38.78

23 4.57 7.24 9.20 17.04 20.39 24.51

n 4.36 6.16 8.13 10.02 11.57 11.97

the grid. The use of space filling curves is indeed folklore in computer vision [12, 13]. We combined the idea of ordering curve with It -global rounding method. The method takes advantage of characteristics of an ordering curve to reduce a two-dimensional problem to one-dimensional problem. Instead of considering an ordering curve of the whole grid, we consider a partition of the grid into ordering curves, since the behavior of an ordering curve near the boundary of the grid often affects the quality of halftoning if we force to cover the whole grid by using a single connected curve. A Hilbert curve is a curve to fill pixel grid plane recursively. For our experiment we considered a non-recursive curve H2 having 16 vertices each at the center of the sixteenth of the unit square. This is illustrated in Figure 8. We partition the pixel plane into horizontal bands of size 4 × n. We order the entries of each band following the H2 curve. Doing so, we constitute a sequence of length 4n for each band. The combination of H2 and In -global rounding gives a good halftoning result shown in Figure 10 having low discrepancy measures. This algorithm outputs an image which takes a relatively small value for every evaluation function. One weak point of this algorithm is its O(n3 ) time complexity, but it can be reduced by using It -global rounding for a smaller t; see Figure 11 for t = 9. We also tried to order the entries of an input matrix following a curve shaped as a union of diagonal zigzags of with 3. For notational convenience let DZ3 denote this ordering curve. Figure 9 shows the partition of the picture into a set of sequences filling the pixel grid plane. Figure 14 is a halftoning image by the I9 -global rounding along the DZ3 curve. Unfortunately, it suffers from downward diagonal stripe patterns. 4.1

Global Rounding Minimizing the Sum of ARE

We have selected the sequence rounding from the set of all global roundings in a randomized fashion. However, we can control the selection by choosing the best one according to a given criterion. Indeed, we want to compute the It -global rounding along an ordering curve minimizing AREk2 . The problem seems to be difficult theoretically, and we give a heuristic algorithm for the curve DZ3 in order to test whether such kind of optimization is effective or not. We give a technical assumption that t ≥ 2k + 1; this assumption

314


Fig. 8. 4 × 4 Hilbert curve

Fig. 10. I4n -global rounding along H2

Fig. 9. DZ3 Ordering curve

Fig. 11. H2

I9 -global rounding along

can be removed, although it will make the algorithm complicated. Although k can be taken arbitrary, we explain the case where k = 3. The ordering curve DZ3 consists of connected components each of which covers a tridaigonal region. Figure 12 illustrates the behaviour of our heuristic algorithm. Our heuristic is as follows: For simplicity, we assume that n = 3m + 2 for a natural number m. Indeed, 512 = 3 × 170 + 2. For an integer s ∈ [−m, m], let Us be the tridiagonal region such that its central diagonal consists of the matrix entries ai,j satisfying that j − i = 3s. Let F i = {R|R ∈ F3×3 , R ⊂ ∪ij=−m Uj }. Starting with i = −m, we compute the It global rounding of Ui along DZ(3) minimizing R∈Fi ARE(R) under the condition that the roundings of entries in ∪i−1 j=0 Uj has been already computed. The computation of the rounding of Ui can be done as follows: The set of It roundings of the sequence DZ3 ∩ Ui (of length at most 3n) can be represented as a set of shortest path of a directed acyclic graph G with O(tn) nodes (see Figure 13 for an example). The structure of G is given in Sadakane et. al. [10],


315

Table 7. Evaluation of It -global roundings along H2 t 3 5 7 9 11 15 19 23 4n

Row 70 37 29 29 25 26 29 26 25

Col 86 47 30 35 46 30 37 26 25

ARE SPE 9.17 9.12 5.21 7.45 4.33 7.40 3.98 7.44 4.11 7.38 3.96 7.36 4.18 7.18 3.60 7.21 3.69 7.16

SROE SCOE 24.20 26.02 21.85 19.95 16.71 18.79 18.10 16.07 18.93 15.71 19.03 15.86 19.16 17.30 19.04 16.73 18.92 15.13

SDDE SADE time 26.05 30.00 2.93 23.22 23.26 4.48 22.58 23.55 5.67 22.37 23.81 7.09 22.27 22.63 8.39 22.15 22.79 10.86 22.26 24.05 13.61 22.41 22.72 16.32 22.04 21.99 107.1

and we give a brief outline here. Let us consider the set of binary sequences representing all the It roundings. There are exponential number of such sequences, and if we bundle each pair of sequences with the same prefix, we obtain a leveled tree T representing the sequences so that the set of paths in T gives the set of all It roundings: Each edge of the tree has a binary label, and we read the sequence of labels in a root-to-leaf path, we have an It rounding. Now, for each pair of vertices with the same level, if the paths of length t from the vertices towards the root are same, we can see that the structure (including the binary labels) of the subtrees from these nodes are identical. Thus, we contract these vertices into a node for each of such pairs to obtain our graph G. Of course, it takes exponential time if we compute G as above, but we can smartly compute it in O(tn) time [10]. Given the graph G representing the set of It global roundings, we can compute the global rounding of DZ(3) ∩ Ui minimizing R∈Fi ARE(R) as follows: We keep track of square regions which are in Fi but not in Fi−1 . Let S be the set of such regions. For each node v of G, we give a value Sum(v), which is the total

Fig. 12. Regions in F3×3 associated with pixels of a sequence following DZ3

316


Fig. 13. The I4 -global rounding graph for an input sequence (0.4,0.4,. . . ,0.4) sum of ARE values of regions in S whose all entries are rounded according to the labels of the path from the root to v; we do not add the ARE values of regions that contains an entry corresponding to the edges below the level of v. We sweep the leveled directed acyclic graph G from the root to leaves, and every time the rounding on a path newly determines ARE value of a square region, we update v. If two paths join at v, we take the parent so that the smaller value of Sum(v) is attained. The suffix of length 3k of the two paths has the same suffix of length t, and since t ≥ 2k + 1, the suffix only influence to the non-determined ARE values. When the process is completed, we take the leaf with the smaller Sum value, and retrieve the path by using backtracking. It

Fig. 14. DZ3

I9 -global rounding along

Fig. 15. With optimization


317

takes O(tn) computation time for this process for an i. Thus, in total, we obtain the rounding of the n × n matrix in O(tn2 ) time. The output halftoning image is given in Figure 15. The problem of the figure is that it has many small diagonal patterns, and we cannot say that optimization gives significant improvement of visual quality. Moreover, unfortunately, the gain of the optimization is not very large, we improved our objective function ARE3∞ from 0.86 to 0.79. As shown in tables 8 9 10, ARE values are mostly improved from the output of I9 global rounding along the curve, but not as good as In global rounding. 4.2

Comparison of Halftoning Algorithms

The following three tables shows AREkp errors for p = 1, 2, ∞ and k = 5, 10, 20, 30, 40, 50. For larger k, In global rounding, I4n global rounding along H2 , and In global rounding along DZ3 perform well. This suggests that these method will work well for a finer pictures. Table 11 shows the performance comparison on other measurement functions. We see that I4n (or I9 ) rounding along H2 gives good scores for every quality Table 8. Comparison of L∞ k ARE values ∞ Method ARE5∞ ARE10 Threshold 12.29 48.23 Randomized 11.00 21.16 Dither 5.00 7.42 1D diffusion 4.71 8.78 2D diffusion 3.24 5.98

∞ ARE20 186.67 42.93 13.17 15.45 10.05

∞ ARE30 412.70 67.13 21.52 19.49 13.40

∞ ARE40 726.91 92.13 25.05 24.85 16.44

∞ ARE50 1127.96 105.98 36.56 25.54 20.81

Table 9. Comparison of L1k ARE values Method Threshold Randomized Dither 1D diffusion 2D diffusion


Table 10. Comparison of L2k ARE values Method Threshold Randomized Dither 1D diffusion 2D diffusion


318


Table 11. Values of other measurement functions Method Threshold Dither 1D diffusion 2D diffusion

Row 100 107 0.996 16

Col ARE5∞ SPE SROE SCOE SDDE SADE time 206 12.29 6 21 32 30 20 0.6 134 5.00 7 14 15 25 24 0.6 159 4.71 7 4 36 25 25 0.6 68 3.24 7 13 33 22 22 0.7

evaluation function except the computation time. The CPU times for computing these image range from 0.6 to 107.1 seconds. Our algorithms are clearly slower than known methods, but not very slow if k is small.

5

Concluding Remarks

Based on our experiment, it is very important to select a suitable measurement function. We have succeeded to reduce AREkp values (for p = ∞, 1, or 2) for k > 20, and thus we hope that our method using global rouding works well for halftoning a fine image; we will do experiments on such fine images in our future work. ARE works well, but it is necessary to give a more effective measurement function to devise an evaluation system that simulate human’s eye more precisely. It is necessary that ARE value should be low; however, optimizing ARE doesn’t always give good looking output. Nevertheless, if we will find a more effective measurement function, we expect that we will be able to design a better halftoning method by using an algorithm for optimizing the measurement function. Instead of considering partitioning of the grid into ordering curves, we could use a covering of the grid by curves. That is, if two or more curves overlap at a pixel, we may choose the rounding of the pixel from the roundings given on the curves by majority and/or coin-flip. In our future work, we will implement our evaluation system by using other types of region families, for example, the 2-laminated family proposed by Asano et al.[2]. Also, we will try to detect unexpected scars and waves automatically and sensitively.

References [1] T. Asano, “Digital Halftoning: Challenges of Algorithm Engineers”, IEICE TRANS. Fundamentals, vol. E02-D, NO. 5 May 2002. 301, 302 [2] T. Asano, N. Katoh, K. Obokata, T. Tokuyama, “ Matrix Rounding under the Lp Discrepancy Measure and Its Application to Digital Halftoning,” Proc. 13th ACMSIAM Symp. on Discrete Algorithms, 2002, pp.896–904. 302, 318 [3] T. Asano, T. Matsui, and T. Tokuyama: “Optimal Roundings of Sequences and Matrices,” Nordic Journal of Computing, 7-3 (2000), pp. 241–256. 302, 311 [4] J. Beck and V. T. S¨ os, Discrepancy Theory, in Handbook of Combinatorics Volume II, (ed. R.Graham, M. Gr¨ otschel, and L Lov´ asz) 1995, Elsevier. 302


319

[5] J. Bentley, “Programming Pearls,” CACM 27 (1984), 865–871. 303 [6] Donald E. Knuth,”Digital Halftones by Dot Diffusion”, ACM Transactions on Graphics 6 (1987) pp. 245-273. 301, 306 [7] R. W. Floyd and L. Steinberg: “An adaptive algorithm for spatial gray scale,” SID 75 Digest, Society for Information Display (1975), pp. 36–37. 308 [8] N. Takki-Chebihi, “Discrepancy-Based Digital Halftoning: Automatic Evaluation and Optimization,” Master thesis, GSIS Tohoku University. In preparation for a journal pubilication. 309, 310 [9] P. Raghavan and C. Thompson, “Randomized Rounding,” Combinatorica 7 (1987), pp. 365–374. 306 [10] K. Sadakane, N. Takki-Chebihi, T. Tokuyama, “Combinatorics and Algorithms on Low-Discrepancy Roundings of a Real Sequences”, Proc. 28th ICALP, LNCS 2076 (2001), pp. 166-177. 304, 311, 314, 315 [11] V. R¨ odl and P. Winkler: “Concerning a Matrix Approximation Problem”, Crux Mathmaticorum (1990), pp. 76–79. 302 [12] L. Velho and J. Gomes: “Digital Halftoning with Space Filling Curves”, Proc. SIGGRAPH’91 (1991), pp. 81–90. 313 [13] I. Witten and N. Neal, “Using Peano Curves for Bilevel Display of ContinuousTone Images”, IEEE Computer Graphics and Applications (1982), pp. 47–52. 313

Deriving Topological Representations from Edge Images Ullrich Köthe University of Hamburg, Cognitive Systems Group Vogt-Kölln-Str. 30, 22527 Hamburg, Germany [email protected]

Abstract. In order to guarantee consistent descriptions of image structure, it is desirable to base such descriptions on topological principles. Thus, we want to be able to derive topological representations from segmented images. This paper discusses two methods to achieve this goal by means of the recently introduced XPMaps. First, it improves an existing algorithm that derives topological representations from region images and crack edges, and second, it presents a new algorithm that can be applied to standard 8-connected edge images.

1

Introduction

Many authors have argued that image segmentation should produce a topological image representation [2, 9, 11, 17]. Otherwise, it would be impossible to consistently answer basic questions such as “Which regions are neighbors of each other?”, “Which regions are contained in another one?”, or “Where precisely is the mutual boundary between two neighboring regions?” The connectivity paradox is the most infamous example for the inconsistencies that occur if a naïve image representation is used [11]. A topological representation defines a finite topological space made up of regions, arcs, and points (also called 2-, 1-, and 0-cells, or faces, edges, and nodes) which encode a particular partitioning of the image plane. Several structures have been proposed to encode such a partitioning, including cellular complexes [11], combinatorial maps [7], and the Khalimsky grid [8]. In a recent paper, I’ve introduced the concept of an extended planar map (XPMap) that subsumes the important characteristics of these structures and provides a unified approach to topological image representation [10]. In this paper, I’d like to fill a gap that remained open in the previous article: How can one actually derive a topological representation from a set of pixels? Or, put differently: How can we modify well known segmentation algorithms so that they produce the desired XPMap representation? Previous authors, e.g. [2, 17] have approached this problem solely on the basis of crack edges, that is, edges that are located between the pixels. We will review this work below. However, many standard image segmentation algorithms (such as Canny’s algorithm and the watershed algorithm) do not locate edges between pixels, but mark edges on the pixels themselves. As far as I can tell, an algorithm to transform this kind of edge image into T. Asano et al. (Eds): Geometry, Morphology, ... 2002, LNCS 2616, pp. 320-334, 2003.  Springer-Verlag Berlin Heidelberg 2003

Deriving Topological Representations from Edge Images

321

a topological representation without resorting to heuristics (as in [14, 15]) does not yet exist. The development of such an algorithm is the main contribution of this paper.

2

Finite Topological Spaces and Topological Transformations

2.1

XPMaps

From the point of view of finite topology, image segmentation is the partitioning of the image plane into points, arcs, and regions. In principle, these entities can be defined by geometric means: A point can be defined by its 2D coordinate. An arc is a mapping of the open interval (0, 1) onto the image plane, such that the images of 0 and 1 coincide with two of the points and no arc crosses another one or itself. And regions can be defined as the connected components of the complement of the boundary set, i.e. the complement of the union of points and arcs. However, the topology of the partitioning is only implicitly represented by this geometric definition. In the context of image analysis, making the topology explicit is much more desirable. A separation between topology and geometry can be achieved by means of abstract topological representations such as the XPMaps introduced in [10]. XPMaps are based on combinatorial maps [7]: Definition 1: A combinatorial map is a triple (D, σ, α) where D is a set of darts (also known as half-edges), σ is a permutation of the darts, and α is an involution (a permutation with cycle length 2) of the darts. A combinatorial map is said to be trivial if it doesn’t contain any dart. By definition, the trivial map contains a single face, the infinite face (which corresponds to the entire plane). The cycles or orbits of the α involution define the edges (1-cells) of the map, and the orbits of the σ permutation are its nodes (0-cells). The mapping ϕ = σ -1α is also a permutation whose orbits correspond to the faces (2-cells). A k-cell is said to bound an m-cell if k < m and the orbits associated with either cell have at least one dart in common. A combinatorial map fulfills the axioms of a topological space, if open sets are defined as follows: a set O of cells is open if, whenever cell c belongs to O, all cells bound by c do also belong to O. A combinatorial map is planar (encodes a partitioning of the plane) if the numbers n, e, f of nodes, edges and faces respectively fulfill Euler’s equation: (1) n−e+ f =2

However, this equation only applies if the map’s boundary set, i.e. the graph defined by considering only nodes and edges, is connected. Thus, combinatorial maps cannot represent partitionings of the plane with disconnected boundaries, such as a wall with windows. But disconnected boundaries are common in image analysis, so that the map concept must be extended in order to handle this case: Definition 2: An extended planar map (XPMap) is a tuple (C, c0, exterior, contains) where C is a set of non-trivial planar combinatorial maps (the components of the XPMap), c0 is a trivial map that represents the infinite face of the XPMap, exterior is a relation that labels one face of each component in C as the exterior face, and contains is a relation that assigns each exterior face to exactly one non-exterior face or the infinite face.

322

Ullrich Köthe

Fig. 1. Example of an XPMap (for clarity, only some edges and nodes are labeled in the drawing, and only the α orbits [opposite arrows] and the σ orbit for node N4 are shown)

Fig. 2. Illustration of the Euler operators “Merge Faces”, “Remove Bridge”, “Remove Isolated Node”, “Merge Edges” (left to right, top to bottom)

1

2

3

4

5

Fig. 3. Segmentation by Euler operators: 1) original image; 2) associated Khalimsky grid (one 2-cell per pixel); 3) after 7 applications of Merge Faces; 4) 1 application of Remove Bridge and Remove Isolated Node; 5) 13 applications of Merge Edges

Figure 1 shows an example of an XPMap. Details can be found in [10]. Since an XPMap may contain multiple components, Euler’s equation has to be modified to take this into account: n − e + f − c =1 where c denotes the number of components.

(2)


323

XPMaps can be modified by means of Euler operators. These operators are named so because they guarantee that Euler’s equation remains valid after the modification. Thus, Euler operators are able to transform a valid XPMap into another valid XPMap. In the present context of image segmentation, four operators are of primary interest (see figure 1, detailed algorithms and proofs can be found in [10]): MERGE FACES, REMOVE BRIDGE: These operators remove an edge. They first remove the edge’s darts from their σ-orbits and then delete the edge’s α-orbit and the darts. The desired modification of the ϕ -orbits follows automatically. Removal of a bridge (an edge that bounds only one face) creates a new XPMap component, so the exterior and contains relations must be adjusted as well. REMOVE ISOLATED NODE: Sometimes the new component resulting from “Remove Bridge” consists of a single node. “Remove Isolated Node” deletes this component from the XPMap and updates the contains relations accordingly. MERGE EDGES: Edge removal usually causes many nodes to have degree 2, i.e. to bound exactly two edges. These nodes are called links. “Merge Edges” simplifies an XPMap by replacing a connected sequence edge-link-edge with a single edge. All these operators reduce the number of cells. To use them for segmentation, we must start with an over-segmentation which is then successively reduced to the desired segmentation. 2.2

Creating a Topological Partitioning by Euler Operators

The simplest way to define a topologically consistent over-segmentation is to associate a face with every pixel. This is formalized by means of the Khalimsky grid [8]: Definition 3: A Khalimsky grid is defined on Z2 by denoting points with two even coordinates as faces, points with two odd coordinates as nodes, and mixed points as edges. Nodes bound their eight neighbors (four edges and four faces), and edges bound the two neighboring faces.

If open sets are defined as in the previous section Khalimsky grids also fulfill the axioms of a topological space. It is also easy to see that a Khalimsky grid defines a combinatorial map: we associate two darts with every edge, one pointing in increasing x or y direction respectively, the other pointing into the opposite direction. These pairs form the orbits of the α involution. The orbits of the σ permutation are defined by taking the four darts starting at the same node and sorting them in mathematically positive order. For any image there exists a corresponding Khalimsky grid such that the pixel at image coordinate (x, y) corresponds to the face at Khalimsky coordinate (2x, 2y). Figure 3 illustrates how an image segmentation can be obtained by applying a sequence of Euler operators to a Khalimsky grid. However, as images get larger, this method becomes inefficient. Therefore, we will not pursue this approach in the present paper. Instead, we will investigate two alternative algorithms that derive topological representations directly from traditional image segmentations.

324

Ullrich Köthe

3

The Crack Insertion Algorithm

Topological segmentation on the basis of Euler operators, as outlined in the previous sections, is a good way to theoretically prove the topological properties of the resulting representations. However, it is somewhat laborious as a practical algorithm. In practice, the Crack Insertion Algorithm is the easiest way to derive a topological representation. In similar form, phase 1 of the algorithm has been used in the segmentation methods of [2, 3, 17]. However, phase 2 (derivation of an explicit topological representation) was only carried out in [2]. The algorithm starts from a region image, i.e. a complete image partitioning into 4connected components. Region images typically result from region growing (e.g. [1]), split-and-merge (e.g. [12]) or 4-connected components labeling of binary images. As the name suggests, crack edges are located between neighboring pixels, if those pixels’ labels differ. In a region image, the crack edges are only coded implicitly because the image does not contain addressable entities which represent the cracks. The crack insertion algorithm makes the cracks explicit by inserting appropriate entities into the data representation (phase 1). After this an XPMap representation can easily be derived (phase 2). Compare figure 4 for illustration: Phase 1: Crack Insertion Given: Region image with labeled 4-connected components, size w × h.

1. Create an image of size (2w – 1) × (2h – 1). Following [3], we will call this image the super grid. Copy the labels from position (x, y) of the region image to position (2x, 2y) in the super grid. 2. For each cell in the super grid with coordinates (2m, 2n + 1): If the cell’s two vertical neighbors have the same label, copy this label into the cell. Otherwise, mark the cell with a special edge label. 3. For each cell in the super grid with coordinates (2m + 1, 2n): If the cell’s two horizontal neighbors have the same label, copy this label into the cell. Otherwise, mark the cell with a special edge label. 4. For each cell in the super grid with coordinates (2m+1, 2n+1): If any of the 4neighbors was labeled as an edge, mark the cell with a special node label. Otherwise, copy the label of the neighbors (which is necessarily unique). Phase 1 results in 4-connected regions whose cells have identical labels, and 4connected boundaries, whose cells are labeled as edges and nodes (the proof of 4connectedness is straightforward). Now, phase 2 derives an XPMap from the labeled super grid: Phase 2: Derivation of an XPMap from a Labeled Super Grid Given: Labeled super grid resulting from phase 1.

1.

2.

Augment the super grid with a one cell wide border whose cells are labeled as nodes if their coordinates are both odd, and as edges otherwise. This results in a super grid of size (2w + 1) × (2h + 1). Find the connected components of the boundary set (that is, of the set of cells marked as edges and nodes). For each component thus found create a component in the XPMap to be build.


325

3.

For each component: 3.1. For each edge in the component: Create a pair of darts and insert it as an orbit into the component’s α involution. The two darts of a pair are distinguished by their orientation (north/south and east/west respectively). 3.2. For each node in the component: Create an orbit in the component’s σ permutation that contains the darts adjacent to the node in counterclockwise order. 4. Establish the exterior relation of each component. In case of the outermost component (the one that is adjacent to the super grid’s border) the exterior face is always the infinite face (which is not explicitly represented in the super grid). To find the exterior face of the other components, traverse the super grid in scan-line order. When the scan first encounters a node of a particular component, the cell seen just before belongs to the sought for exterior face. 5. Establish the contains relation as follows: for each region, list the components where the present region was designated as the exterior face. Since ϕ = σ -1α it is not necessary to explicitly derive the ϕ permutation from the super grid – it is already uniquely defined by σ and α. It should be noted that the algorithm is essentially equivalent to a segmentation by means of Euler operators, as outlined in the last section: If we started with a Khalimsky grid on the subset [-1, w] × [-1, h] of Z2, the edges and nodes to be removed by Euler operators would be precisely the ones that were not marked with edge/node labels during crack insertion (compare figures 3 and 4). After phase 2 of the above algorithm, all edges consist of a single cell, and most nodes bound exactly two edges. We will call these nodes joints, whereas nodes of higher degree will be called junctions. In many applications, we are only interested in the junctions, not the joints. In this case, we may again use the Euler operator “Merge Edges” to transform connected cell sequences of the form “edge-joint-edge” into single edge cells. By applying the operator repeatedly, we can successively eliminate all joints (last step in figure 4). Details can again be found in [10].

Fig. 4. Application of the Crack Insertion Algorithm: 1) 3%3 region image; 2) Labeling of the even coordinate pixels in the associated super grid; 3) Labeling of the mixed coordinate pixels (the special edge label is indicated by a black line); 4) Labeling of the odd coordinate pixels and addition of the outer boundary (the special node label is indicated by a black ball); 5) Elimination of joints by “Merge Edges”. Note that, in contrast to figure 1, Euler operators cannot eliminate pixels from a super grid. Instead, pixels get re-labeled with their new cell type. In this case, black balls (nodes) are changed into black dots (odd coordinate pixels that belong to an edge)

326

Ullrich Köthe

4

Deriving an XPMap from an 8-Connected Boundary

Many segmentation algorithms do not present their results by means of region images or crack edges but rather mark some pixels as “edge pixels”. This kind of edge image typically results from the watershed algorithm [16] and from topological thinning [6]. Canny’s algorithm [4] is also commonly used to create edge images, although one can also directly produce a symbolic representation, e.g. a set of edgel chains (but in contrast to our new algorithm, existing algorithms to link edgels into chains heavily rely on heuristics, especially at junctions, e.g. [14, 15]). Edge images resulting from zero-crossing detection [15] occupy a middle ground: Since zero-crossings occur between pixels, it is natural to interpret them as crack edges, so that the crack insertion algorithm can be applied. But it is just as common to mark the pixel nearest to a zero-crossing as an edge pixel, in which case the resulting edges can be treated like Canny edges. As far as I’m aware of, a non-heuristic algorithm that can derive an XPMap or another topological representation from an edge image has not yet been developed. The new algorithm will be based on the following observations: • From the definition of the XPMap, it is clear that we need not only regions and edges, but also nodes (junctions and end points). Thus, a classification of the boundary pixels into edge and node pixels will be the core of our algorithm. Considering this, it is not really correct to call the images resulting from Canny’s algorithm “edge images” because this ignores the nodes. In the sequel, we use the term boundary image instead. • It is well known that one cannot in general define consistent topological relations on an image by using either the 4- or 8-neighborhood. However, one can use 4-neighborhood in the foreground, and 8-neighborhood in the background [13]. Therefore, we will adopt the 4-neighborhood to determine the connectivity of regions and the 8-neighborhood for the connectivity of the boundary. • We will define edges as junction-free chains of edge pixels. To be junctionfree, chains must have the following property: Every interior pixel of the chain must be 8-connected to exactly two other edge pixels. The two ends of the chain are adjacent to an edge pixel and a node pixel. A degenerate chain consists of a single edge pixel that is adjacent to two nodes. • We can determine whether a boundary pixel is a node or edge pixel by just looking at the 8-neighborhood of the pixel1. However, this requires the boundary to be irreducible: It must not contain simple points, i.e. boundary pixels that could be turned into region pixels without changing the connectivity of both the boundary and the regions. Removal of simple points is called thinning (therefore, we also call an irreducible boundary thin).

The watershed algorithm is an example of thinning, where a cost function (e.g. the image gradient) determines the order of simple point removal. Therefore, boundary images resulting from this algorithm fulfill the requirement. In contrast, boundaries 1

Formally, the 8-neighborhood of the pixel at (x0, y0) is defined by N8(x0, y0) = {(x, y): max(|x–x0|, |y–y0|) = 1}, i.e. the center pixel is not part of the neighborhood.


327

coming from Canny’s algorithm may still contain a few simple points which must be removed before the classification can start. Table 1. Possible configurations (modulo rotation and reflection) in the 8-neighborhood of a boundary pixel, along with the classification according to definition 4. Configurations marked with (*) can only occur if they are treated specially during thinning, and are then classified as “node pixels” (see text)

328

Ullrich Köthe

Fig. 5. Some patterns which show that configurations 22, 37, and 51 (located in the center of the example images, respectively) can actually by irreducible


329

When we analyze how region and boundary pixels can be distributed in the 8neighborhood, we obviously arrive at exactly 256 possible configurations. After removing rotated and reflected patterns, 51 unique configurations remain. They are shown in table 1. A number of these patterns cannot occur in an irreducible boundary because the center pixel would be a simple point2. In a few cases (marked with *) the decision is not clear cut: In a strict sense these patterns are reducible, but it is often desirable to modify the thinning algorithm in order to keep the points in the boundary. Consider, for example, configuration 5: This pattern occurs at the corners of an axes-parallel rectangle. Removal of the center point would “round” the corners. A similar situation is found in configuration 16 (and 21, 29, 34): These configurations mark T-junctions. Removal of the center point would result in a little bend in an otherwise straight edge. Configurations 2 and 3 represent another exceptional case: These patterns mark an end of a broken or dangling edge. Broken edges often result from noise or low contrast and can be “repaired” by higher level analysis and perceptual grouping. But this is only possible if the dangling ends are not removed during thinning. Since our algorithm does not depend on whether these configurations occur or not, there is no need to forbid them – the appropriate variant of thinning can be chosen according to the application context. In order to avoid special treatment at the image border (where the 8-neighborhood would be partially outside the image) it is useful to again augment the boundary image with a one pixel wide border whose pixels are all marked as boundary pixels. If addition of this boundary creates reducible pixels they should be removed by an additional thinning iteration. Then, if the 8-neighborhood is partially outside the enlarged image the missing pixels can always be considered as region pixels (namely as part of the infinite region), and classification proceeds as usual. Formally, the irreducible patterns are classified as follows (see table 1): Definition 4: A boundary pixel in a thin boundary image is classified as an edge pixel if its 8-neighborhood consists of exactly four 4-connected components, and neither of the components consisting of boundary pixels contains more than one 4-neighbor. Otherwise, the pixel is a node pixel. (If configuration 5 is allowed, it is treated exceptionally and marked as a node pixel as well.)

The first condition ensures that each edge pixel has exactly two neighbors, so that we can actually group edge pixels into chains. The second condition is necessary to avoid that no pixel is classified as a node pixel in configurations like figure 5 left. On the basis of the definition, we can specify our algorithm as follows: Algorithm: XPMap from 8-connected Irreducible Boundary:

Given: Boundary image with irreducible 8-connected boundary and 4-connected regions (as indicated above, several thinning variants might be used). 1. Augment the image with a one pixel wide border whose pixels are all marked as boundary pixels.

2

On first glance one might think that even more patterns should be reducible, e.g. numbers 22, 37 and 51. But, as figure 5 shows, this is not the case: there are irreducible configurations that contain these patterns.

330

Ullrich Köthe

2. Find the 8-connected components of the entire boundary, i.e. the set of all boundary pixels. Create an XPMap component for each boundary component. 3. Classify boundary pixels according to definition 4. 4. Perform 8-connected components labeling of the node pixels. If necessary reclassify pixels to make the components simply connected (see remark A below). Each resulting component becomes a node of the XPMap to be build. 5. Perform 8-connected components labeling of the edge pixels. In order not to merge different chains, a slight modification of the labeling algorithm is required: edge pixels that are adjacent to the same node pixel are not considered connected (see remark B below). Each resulting component is a chain and becomes an edge of the XPMap to be build. 6. For each XPMap component found in step 2: 6.1. For each chain in the component: Create a pair of darts and insert it as an orbit into the component’s α involution. Most chains have two unique end pixels, and the darts may be identified by those pixels. In case of a degenerated, one pixel chain (e.g. configuration 46), the darts are identified by their orientation. In case of a closed loop without node pixel (e.g. the boundary of a diagonal square), an arbitrary edge pixel in the chain must be re-classified as a node pixel. 6.2. For each node in the component: Create an orbit in the component’s σ permutation that contains the darts adjacent to the node in counterclockwise order. To find those darts, simply walk around the node by means of the well-known “left hand on the wall” algorithm and register all darts thus met. 7. Establish the exterior and contains relations as in steps 4 and 5 of phase 2 of the crack insertion algorithm.

Fig. 6. Some patterns which result in connected components of node pixels (black) that have holes and are thus not simply connected

Fig. 7. Example of the problematic case where diagonally adjacent edge pixels (gray) touch the same vertex pixel (black). The four “arms” of the pattern are not considered connected at the center pixel, despite their touching diagonally


331

Fig. 8. a, c. The 2 possible edge pixel configurations with neighbor groups having 2 or 3 boundary pixels (* denotes an arbitrary attribution, a: configurations 11, 12, 13, 25, 28, 35; c: 23, 24, 35, 46). b, d: Configurations obtained by shifting the window 1 pixel to the right. As is easily verified by table 1, in any irreducible configuration containing these patterns the center pixel is classified as a node pixel (b: 5, 16, 17, 21, 27, 29, 34; d: 16, 29, 31, 42, 47)

Remarks:

A. The nodes in an XPMap must be homeomorphic to a point. This means that the node components resulting from step 4 of the algorithm must be simply connected. Unfortunately, this is not guaranteed: it is possible to construct point configurations that result in connected components of node pixels that have holes. Figure 6 shows 3 examples. However, the problem is not very serious in practice since it can only arise if the boundary patterns are highly symmetric, and this is very unlikely in real images. In fact, the only configuration I have ever seen during experiments is figure 6 left which may occur if a seed has never got the opportunity to grow. In any case, the problem is easy to detect and surmount: measure the area enclosed by the node (i.e. the number of the node’s pixels plus the area of a possibly enclosed hole) by means of the expression A = ∑i (xi yi+1 – yi xi+1) which is evaluated along the outer crack edge of the node. Compare the resulting area with the number of the node’s node pixels. If the numbers agree, the node doesn’t have a hole. Otherwise merge all pixels inside the hole with the surrounding node. In the practically important case of figure 6 left this simply means to re-label the central pixel, otherwise it amounts to a standard flood-fill. Nodes that occupy a simply connected image region rather than a single pixel present no topological problems, since the bounding relation (which determines the topological structure) is independent of a node’s shape. B. The edges in an XPMap must be homeomorphic to a line. This idea is captured by our notion of a chain: Starting from an edge pixel adjacent to a node, we must be able to go to uniquely defined successor pixels, until we reach another edge pixel which is adjacent to a node. We will now prove that the modified 8-connected component algorithm in step 5 indeed creates such chains. First, we refer to figure 7 to justify the modification of the connected components algorithm: If two or more edge pixels are horizontally and vertically adjacent to a node pixel, they are also diagonally adjacent to each other. However, they clearly belong to different chains, and the modification explicitly handles this case in the desirable way. Now we show that the labeling indeed produces chains. According to definition 4, the neighborhood of an edge pixel consists of four 4-connected components, two of which contain boundary pixels. We will call the latter boundary groups. If we look at the configurations classified as edge pixels in table 1, we see that none of their neighboring boundary groups consists of more than three boundary pixels. It is also easy to see that the chain property could only be violated if some boundary group

332

Ullrich Köthe

would contain two or three edge pixels but no node pixel – only then the present edge pixel would not have a unique successor (or predecessor). But this is never the case: Whenever a boundary group consists of two or three pixels, at least one of them gets classified as a node pixel, so that the center pixel becomes the end of a chain. This fact is illustrated in figure 8. In other words, whenever an edge pixel is not the end of a chain, its two neighboring boundary groups consist of exactly one boundary pixel, and the chain property is always ensured. Although the description of the algorithm is somewhat complicated, its actual implementation is quite simple – it basically doesn’t involve anything beyond connected components labeling (with slight modifications), contour following around nodes, and classification of boundary pixels according to the 8-neighborhood. The algorithm has been implemented successfully, and figure 9 shows a result.

Fig. 9. Left: boundary image as might result from the watershed algorithm (but note the modified thinning indicated with an arrow); Right: resulting classification of the boundary pixels (black: edge pixels, gray: node pixels; note the addition of an outer boundary)

5

Conclusions

This paper presented two algorithms that derive a topological representation from the results of standard segmentation algorithms: region images and edge images. This is very useful because it allows to apply topological concepts without major modifications to the segmentation algorithms themselves. Questions concerning the boundaries and neighborhood of features can thus be answered consistently, without resorting to heuristics that work around topological problems found in traditional representations. It is easy to augment the new topological representation with geometric data defining the precise location of the nodes and edges. In fact, the clean separation of topology from geometry in the new framework provides very high flexibility, because different geometric models (e.g. straight lines, splines, sub-pixel accurate edgel chains) can be connected with the topological representation as the task requires. In contrast to topological representations used in computational geometry (e.g. [5, 7]), our approach establishes precise correspondences between the topological cells and the underlying raw pixels. Thus, one can always go back to the original pixel data when it becomes necessary to collect additional information about a cell’s properties.


333

Further research should systematically compare the two approaches (crack edges or 8-connected edges) in order to define their appropriate application domains. For example, while the crack insertion algorithm is simpler, it also requires four times as many pixels in order to store the inserted cells, unless specific efficient data structures are used [2]. This problem doesn’t arise with 8-connected edges. Also, the boundaries resulting from 8-connected edges look better visually. But making this statement objective is difficult because it is far from clear how segmentations should be compared. In fact, topological comparison criteria have rarely been used in the past. The presented results open up interesting new roads in this direction.

References [1] [2] [3] [4] [5] [6] [7] [8]

[9] [10]

[11] [12] [13] [14] [15]

R. Adams, L. Bischof: “Seeded Region Growing”, IEEE Trans. Pattern Analysis and Machine Intelligence, 16(6), pp. 641-647, 1994 J.-P. Braquelaire, J.-P. Domenger: “Representation of Segmented Images with Discrete Geometric Maps”, Image and Vision Computing, 17(10),715-735, 1999 C. Brice, C. Fennema: “Scene Analysis Using Regions”, Artificial Intelligence, 1(3), pp. 205-226, 1970 J. Canny: “A Computational Approach to Edge Detection”, IEEE Trans. Pattern Analysis and Machine Intelligence, 8(6), pp. 679-698, 1986 “CGAL – Computational Geometry Algorithms Library”, http://www.cgal.org/, 2002 M. Couprie, G. Bertrand: “Topological Grayscale Watershed Transformation”, in: Proc. of SPIE Vision Geometry V, SPIE vol. 3168, pp. 136-146, 1997 J.-F. Dufourd, F. Puitg: “Functional specification and prototyping with oriented combinatorial maps”, Computational Geometry 16 (2000) 129-156 E. Khalimsky, R. Kopperman, P. Meyer: “Computer Graphics and Connected Topologies on Finite Ordered Sets”, J. Topology and its Applications, vol. 36, pp. 1-27, 1990 U. Köthe: “Generische Programmierung für die Bildverarbeitung”, PhD thesis, Computer Science Department, University of Hamburg, 2000 U. Köthe: "XPMaps and Topological Segmentation - a Unified Approach to Finite Topologies in the Plane", in: A. Braquelaire, J.-O. Lachaud, A. Vialard (eds.): Proc. of 10th Intl. Conf. Discrete Geometry for Computer Imagery (DGCI 2002), Lecture Notes in Computer Science 2310, pp. 22-33, Berlin: Springer, 2002; longer version appeared as: Univ. Hamburg, Dept. of Informatics Technical Report FBI-HH-M-308/0, 2001 V. Kovalevsky: “Finite Topology as Applied to Image Analysis”, Computer Vision, Graphics, and Image Processing, 46(2), pp. 141-161, 1989 T. Pavlidis: “Structural Pattern Recognition”, New York: Springer, 1977 Rosenfeld: “Adjacency in Digital Pictures”, Information and Control vol. 26, pp. 24-33, 1974 Rothwell, J. Mundy, W. Hoffman, V.-D. Nguyen: “Driving Vision By Topology”, in: IEEE Intl. Symposium on Computer Vision, pp. 395-400, 1995 M. Sonka, V. Hlavac, R. Boyle: “Image processing, Analysis, and Machine Vision”, Brooks/Cole Publishing Comp., 1998

334

Ullrich Köthe

[16]

L. Vincent, P. Soille: “Watersheds in digital spaces: an efficient algorithm based on immersion simulations”, IEEE Trans. Pattern Analysis and Machine Intelligence, 13(6), pp. 583-598, 1991 S. Winter: “Topological Relations between Discrete Regions”, in: M. Egenhofer, J. Herring (eds.): Advances in Spatial Databases, pp. 310-327, Lecture Notes in Computer Science vol. 951, Berlin: Springer, 1995

[17]

Calibration of Panoramic Cameras Using 3D Scene Information Fay Huang, Shou Kang Wei, and Reinhard Klette CITR, Computer Science Department, The University of Auckland Tamaki Campus, Auckland, New Zealand

Abstract. This chapter proposes a novel approach for the calibration of a panoramic camera using geometric information available in real scenes. Panoramic cameras are of increasing importance for various applications in computer vision, computer graphics or robotics. Previously developed camera calibration methods (for ‘standard’ camera architectures following the pinhole camera model) are not applicable due to the non-linearity of the panoramic camera, defined by the existence of multiple (nonlinear) optical centers and a cylindrical image manifold. This article addresses the calibration subject of a more general yet flexible panoramic camera model for the first time. The chapter focuses on the calibration of two dominant parameters that characterize the camera model and provide flexibility in selecting different types of panoramas such as single-center (e.g. as assumed for QTVR), symmetric stereo, concentric or polycentric panoramas. We elaborate selected geometric constraints (for increasing numerical stability) with the corresponding solutions; summarize the experimental results with captured image data, and discuss the performance of different geometric constraints via error-sensitivity simulation and analysis.

1 Introduction An example of a prototype of a panoramic camera is depicted in Fig. 1. The architecture of the camera allows the acquisition of different panoramas for various applications, such as single-center panoramas for QTVR, route-planning, or robot localization [1, 12]; symmetric panoramas for stereo reconstruction and/or visualization [5, 9, 11]; and concentric or polycentric panoramas for image-based rendering, novel view synthesis, or walk-through animations [4, 7, 8]. Unfortunately such a widely applicable panoramic camera cannot be calibrated using traditional camera calibration methods due to its non-linearity: it possesses multiple (nonlinear) optical centers and a cylindrical image surface, see Fig. 2. This camera requires a new methodology for the design of a calibration method. Technically, it is possible to calibrate the camera using production-site facilities. But they are not available for the end user. This chapter addresses an on-site camera calibration method utilizing geometric properties available in real scenes. 1.1 Basic Camera Model A panoramic camera can geometrically be described as follows: let O be the rotation center and C the optical center, see Fig. 2(a). The camera-viewing angle is defined by T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 335–345, 2003. c Springer-Verlag Berlin Heidelberg 2003

336

Fay Huang et al.

Fig. 1. A panoramic camera build at the Space Sensory Institute of DLR (German Aerospace Center) at Berlin-Adlershof

the angle ω between optical axis and the line passing through rotation center O and optical center C. A line sensor is rotated with respect to O at the distance R away from the rotation axis (i.e. distance between O and C). It captures one slit (line) image for subsequent constant-size angular increments during rotation. (Note: each slit image contributes to one column of a panoramic image.) Ideally the parameters remain constant throughout one acquisition process (we neglect minor deviations during one rotation in this chapter). Different types of panoramas are acquired depending on the specification of the camera parameters R and ω. Failure in measuring these two parameters results in unsatisfactory/unstable outcomes in applications. The chapter focuses on calibration of these two dominant parameters. 1.2 Traditional Scenario and Motivation Traditionally, if a camera model is available, a straightforward way in camera calibration is to minimize the difference between ideal projections and actual projections of known 3D points (such as calibration objects or localized 3D scene points). See, for example, DLT and Tsai method in [6]. In the context of panorama camera calibration, given a set of known 3D points, say n pairs, (Xwi , Ywi , Zwi ) (in world coordinates) and their projections (ui , vi ) (in image coordinates), this means the following minimization: min

2 2 n 2ui π f Yoi Xoi A + Zoi R sin ω sin + v − , (1) +ω − +v i c 2 + Z2 W Xoi A − R cos ω oi i=1

Calibration of Panoramic Cameras Using 3D Scene Information

where A =

2 + Z 2 − R2 sin2 ω and Xoi oi 



337



 Xwi  Ywi   − Rwo Two ]   Zwi  1

Xoi  Yoi  = [Rwo Zoi   Xwi t11 + Ywi t12 + Zwi t13 + t14 =  Xwi t21 + Ywi t22 + Zwi t23 + t24  . Xwi t31 + Ywi t32 + Zwi t33 + t34 Note that the parameter f is the effective camera focal length measured in pixels and vc is the principle row of the panoramic image. Both these two parameters are assumed to be pre-calibrated using a traditional method [10, 2]. The parameter W is the width of the panoramic image in pixels. The matrix Rwo and the vector Two describe the rotation and translation of the panoramic camera coordinate system with respect to the world coordinate system. Therefore, there are totally 14 parameters to be estimated using a nonlinear least square optimization method. These 14 parameters consist of the targeted parameters R, ω, and also of twelve unknowns tjk , j = 1 . . . 3 and k = 1 . . . 4. The objective function in Eqn. 1 is non-linear. The parameters to be estimated are enclosed in sine functions and square roots involved in both numerator and denominator of the fractions. The dimensionality (i.e. 14) is high due to the fact that an estimation of extrinsic parameters in Rwo and Two is unavoidable in this approach. Hence, a large set of 3D points is needed for a reasonably accurate estimation (say, using least-square error optimization). The quality of a calibration result following this approach highly depends on given initial values in the parameter estimation procedure. Our error sensitivity analysis (see later in the chapter) shows exponential growth. All this motivated us to search for linear geometric relations allowing numerically more stable calibration results. 1.3 Problem Statement The problem is to estimate the values of camera parameters R and ω from a given panoramic image based on some knowledge available from 3D scenes (such as distances, lengths, or orthogonalities) under the following assumptions: We assume there are at least three straight line segments in the captured real scene (e.g. a special object with straight edges), which are all parallel to the rotation axis. For each straight segment, we assume that there are at least two points on this line which are visible and identifiable in the panoramic image, and that the distance between these two points and the length of the projected line segment on the image are measurable, i.e. these are assumed to be available input data. Furthermore, for each straight segment we assume either there exists another parallel straight segment where the distance between these two segments is known, or there exist two other parallel straight lines such that these three lines form an orthogonal triple. (The definition of an orthogonal triple is given in Section 3.) The main intention is to find a single linear equation that links 3D geometric scene features to the camera model such that by providing sufficient scene measurements we

338

Fay Huang et al.

Orthographic View L1

Object A

L3 P1

H1

Top View

L2

P2

H2

P3

H3

P4

H4

D34

D12

h3

h2

h1

R

O

L5

Object B

L4

P5

Rotation Axis

P2 P1

h4

q34

C1

Ci

C3 C 4

C2

S1

h5

P3

Object B

S3

S2

D45

q12

S4

C5

P5

P4 S5

q45

O

w

Panoramic Image

Object A

H5

R

Slit Image

(A)

(B)

Fig. 2. Geometrical relation among parameters

are able to calibrate accurately the values of R and ω. We explore the related geometric properties such as distances, lengths, orthogonalities of the straight lines, and formulate them as constraints for estimating the camera parameters.

2 Distance Constraint All straight segments measured in the 3D scene are denoted as L and indexed, wherever is necessary, by a subscript for the distinction of multiple lines. The distance of two visible points on a line L is denoted as H which can also be interpreted as the length of a line segment defined by those two points. The length of a projection of a line segment on an image column u can be determined from the input image, denoted as h in pixels. Examples of Hi and its corresponding hi values are depicted in Fig. 2(A) where i ∈ [1, . . . , 5]. The distance between two parallel lines Li and Lj is the length of a line segment that connects both lines Li and Lj and which is perpendicular to these lines. The distance is denoted as Dij . If the distance between two parallel straight lines is measured, then we say that both lines form a line pair. One line may be paired up with more than one other line. Figure 2(A) shows examples of three line pairs, namely (L1 , L2 ), (L3 , L4 ), and (L4 , L5 ). Consider two straight lines Li and Lj in 3D space and the image columns of their projections, denoted as ui and uj respectively, on a panoramic image. The camera optical centers associated to image columns ui and uj , respectively, are denoted as Ci and Cj . Let the distance of the two associated image columns dij = |ui − uj | in pixels. The angular distance of two associated image columns of lines Li and Lj is the angle defined by line segments Ci O and Cj O. We denote the angular distance of a line pair


339

Fig. 3. The coordinate system of a line pair

(Li , Lj ) as θij . Examples of angular distances for some line pairs are given in Fig. 2(B). The angular distance θij can be calculated in terms of dij , that is θij =

2πdij , W

where W is the width of a panorama in pixels. The distance between a line L and the associated camera optical center (which ‘sees’ the line L) is defined by the length of a line segment starting from the optical center and ending at one point on L such that the line segment is perpendicular to the line L. The distance is denoted as S. We can infer the distance S by S=

fH , h

where f is the pre-calibrated effective focal length of the camera. Geometric Relation Now we are ready to formulate the geometric relation of the distance constraint by linking together all the previously described geometric information. A 2D coordinate system is defined on the base plane1 for every line pair, i.e. (Li , Lj ), depicted in Fig. 3. Note that even though all the measurements are defined in 3D space, the geometrical relation can be described on a plane, i.e. in 2D space, since all the straight lines are assumed to be parallel to the rotation axis. The coordinate system is originated at O, and the z-axis passes through the camera focal point Ci while the x-axis is orthogonal to the z-axis and lies on the base plane. 1

Base plane is the plane that is perpendicular to the rotation axis and passes through the camera’s optical center.

340

Fay Huang et al.

The position of Ci can then be described by coordinates (0, R) and the position Cj can be described by coordinates (R sin θij , R cos θij ). The intersection point of line −−→ Li and the base plane, denoted as Pi , can be expressed by a sum vector of OCi and −−→ Ci Pi . Thus, we have Si sin ω Pi = . R + Si cos ω Similarly, the intersection point of line Lj and the base plane, denoted as Pj , can be −−→ −−−→ described by a sum vector of OCj and Cj Pj . We have R sin θij + Sj sin(θij + ω) Pj = . R cos θij + Sj cos(θij + ω) As the distance between points Pi and Pj is pre-measured, denoted by Dij , thus we have the following equation 2 Dij = (Si sin ω − R sin θij − Sj sin(ω + θij ))2

+ (R + Si cos ω − R cos θij − Sj cos(ω + θij ))2 . Finally, we obtain 0 = (1 − cos θij )R2 + (Si + Sj )(1 − cos θij )R cos ω − (Si − Sj ) sin θij R sin ω +

2 Si2 + Sj2 − Dij − Si Sj cos θij . 2

(2)

Objective Function In Eqn. 2, the values of Si , Sj , Dij , and θij are known. Thus Eqn. 2 can be arranged into the following linear form K1 X1 + K2 X2 + K3 X3 + K4 = 0, where Ki , i = 1, 2, 3, 4, are coefficients. The three linearly independent variables are X1 = R 2 X2 = R cos ω and X3 = R sin ω. Because of the dependency among the variables X1 , X2 , and X3 , there are multiple solutions of R and ω, if we solve it by a linear least-square technique. To tackle this multiple-solutions problem, we may constrain the parameter estimation further by the inter-relation among X1 , X2 , and X3 , which is X12 = X22 + X32


because of

341

R2 = (R cos ω)2 + (R sin ω)2 .

Assume that n equations are given. We want to minimize the following: min

n

2

(K1n X1 + K2n X2 + K3n X3 + K4n ) ,

(3)

i=1

subject to the equality constraint X1 = X22 +X32 , where the values of Kin , i = 1, 2, 3, 4, are calculated based on the measurements from real scenes and the image, and X1 = R2 , X2 = R cos ω, and X3 = R sin ω. Now, the values of R and ω can be found uniquely by R = X1

and ω = arccos

X2 √ X1

.

Note that even though the additional constraint forces that a non-linear optimization method to be used, the expected linear parameter estimation quality remains.

3 Orthogonality Constraint We say that an ordered triple (Li , Lj , Lk ) of three parallel lines is orthogonal iff the plane defined by lines Li and Lj and the plane defined by lines Lj and Lk are orthogonal. It follows that the line Lj is the intersection of these two planes. For example, in Fig. 2(A), (L3 , L4 , L5 ) is an orthogonal triple. Consider the orthogonal triple (Li , Lj , Lk ) in 3D space. The measures of Si , Sj , Sk , θij , and θjk are defined and obtained in the same way as in the case of the distance constraint. A 2D coordinate system is defined for each orthogonal triple in a similar way as in the distance constraint case. Figure 4 illustrates the 2D coordinate system for the orthogonal triple (Li , Lj , Lk ). The position of Cj can be described by coordinates (0, R), the position of Ci by coordinates (−R sin θij , R cos θij ), and the position of Ck by coordinates (R sin θjk , R cos θjk ). The intersection points of lines Li , Lj , and Lk with the base-plane are denoted as Pi , Pj , and Pk , respectively. We have −R sin θij + Si sin(ω − θij ) , Pi = R cos θij + Sj cos(ω − θij ) Pj =

and Pk =

Sj sin ω R + Sj cos ω

,

R sin θjk + Sk sin(θjk + ω) R cos θjk + Sk cos(θjk + ω)

.

342

Fay Huang et al.

Pi Si Cj

Z

Pk

Pj Sj

w

Sk

Ci q q Ck jk ij

R

O

X

Fig. 4. The coordinate system of an orthogonal triple −−→ −−−→ Since the vector Pi Pj and vector Pj Pk are orthogonal, thus we have the following equation 0 = (−R sin θij + Si sin(ω − θij ) − Sj sin ω) × (R sin θjk + Sk sin(ω + θjk ) − Sj sin ω) + (R cos θij + Sj cos(ω − θij ) − R − Sj cos ω) × (R cos θjk + Sk cos(ω + θjk ) − R − Sj cos ω). This equation can be rearranged to as follows: 0 = (1 − cos θij − cos θjk + cos(θij + θjk ))R2 + (2Sj − (Sj + Sk ) cos θij − (Si + Sj ) cos θjk +(Si + Sk ) cos(θij + θjk ))R cos ω + ((Sk − Sj ) sin θij + (Sj − Si ) sin θjk +(Si − Sk ) sin(θij + θjk ))R sin ω + Sj2 + Si Sk cos(θij + θjk ) −Si Sj cos θij − Sj Sk cos θjk .

(4)

In Eqn. 4, the values of Si , Sj , Sk , θij , and θjk are known. It can be described by the following linear form K1 X1 + K2 X2 + K3 X3 + K4 = 0, where Ki , i = 1, 2, 3, 4, are coefficients and Xi , i = 1, 2, 3, are three linearly independent variables. We have X1 = R2 , X2 = R cos ω, and X3 = R sin ω. Note that this linear form is the same as in the distance-based approach except that the values of the coefficients are different. Hence, the minimization Eqn. 3 and the calculations of R and ω in the distance-based approach can be used in this approach.


3

343

7 8

2 4

1

5

6

Fig. 5. A test panorama image (a seminar room at DLR in Berlin) with indexed linepairs

4 Experiments The line camera WAAC, see Fig. 1, was used for experiments with real panoramic images. The camera specifications applied are given as follows: each image line has 5184 pixels and the effective focal length of the camera is equal to 3420 pixels. The camera mounts on a turntable supporting an extension arm up to 1.0m. Figure 5 shows one of the taken panoramic images in a seminar room of the Institute of Space Sensor Technology and Planetary Exploration, German Aerospace Center (DLR) in Berlin. The size of the seminar room is about 120m2. The image has a resolution of 5, 184 × 21, 388 pixels. The line-pairs (eight pairs in total) are highlighted and indexed. They are used for estimating R and ω, but in this case, only the distance constraint is applied. The value of R is manually set to be 10cm. The principal angle ω is equal to 155◦ according to our definition. The lengths of those lines are also manually measured, with an expected error of no more than 0.5% of their readings. The data of these sample lines used for the camera calibration are summarized in Tab. 1. We use the optimization method of sequential quadratic programming [3] for estimating R and ω. We minimize Eqn. 3. The results are summarized as follows: when all pairs are used, we obtain R = 10.32cm and ω = 161.68◦. If we select pairs {2,3,4,7,8}, we have R = 10.87cm and ω = 151.88◦. If we only use the pairs {2,4,8}, then R =

Table 1. Parallel-line-based panoramic camera calibration measurements associate with the panorama shown in Fig. 5 Index 1 2 3 4 5 6 7 8

Hi = Hj (m) 0.0690 0.6320 0.5725 1.0860 0.2180 0.0690 0.5725 1.3300

hi (pixel) 91.2 600.8 351.4 1269.0 273.0 81.8 318.0 831.2

hj (pixel) 133.8 683.0 367.4 1337.6 273.6 104.2 292.0 859.4

Dij (m) 1.4000 1.0000 1.5500 0.6000 0.2870 1.4000 1.5500 1.3400

dij (pixel) 1003.1 447.3 490.5 360.9 180.1 910.5 398.2 422.5

344

Fay Huang et al. 20

Estimated R and w errors in %

18 16

Distance R error

14 12

Distance w error

10 8

Orth. R error

6 4

Orth. w error

2 0 0

1 2 3 4 Input data error in %

5

Fig. 6. Error sensitivity results of parallel-line-based approach

10.83cm and ω = 157.21◦. From our experiments, we conclude that sample selections and the quality of sample data are influential to the calibration results. We tested error sensitivity for both constraints, the distance between two parallel lines and the orthogonality of three lines with respect to each of the estimated parameters. Ground-truth data in correspondence to values of the real case (i.e. R= 10cm and ω = 155◦ ) are simulated. The errors of values of Si , Dij , and θij , are introduced to the ground-truth data independently with a maximum of 5% additive random noise in normal distribution. The range of Si is from 1m to 8m, and the range of θij is from 4◦ to 35◦ . The sample size is eight. The average results of 100 trials are shown in Fig. 6. The results suggest that estimated parameters using the orthogonality constraint are more sensitive to errors than the case of using the distance constraint. The errors of the estimated parameters increase approximately linearly with respect to the input errors for both cases.

5 Conclusion and Future Work For each of geometric properties of parallel lines, i.e. distance and orthogonality, there is a single linear equation that links those 3D geometric scene features of parallel lines to the camera model. Simplified formulations of both constraints lead to identical algebraic forms allowing a mixed optimization in accordance with available ‘scene structure’. Experimental results on error sensitivity analysis suggest that the calibration performance using the orthogonality constraint is more sensitive to input-errors than using the distance constraint. A comparison between our approach and a traditional projection-difference approach is summarized in Tab. 2. Overall, the reduction of dimensionality, the simplification of computational complexity, and being less sensitive to errors are attributes of the linear geometric feature approach. The result shows the importance of the exploration of geometric properties/constraints for the calibration of a panoramic camera.


345

Table 2. Comparisons between approaches Dimensionality Projection -difference approach

[R|T] unavoidable

Our approach distance & orthogonality

[R|T]* avoidable under our assumption

*

Complexity

Initial Value Dependence

sine fun. and square root

high

linear form

ignorable

Error Sensitivity exponential growing

linear growing

* A matrix of the camera extrinsic parameters consists of a rotation matrix R and translation vector T.

Our approach is based on the assumption that reference lines are parallel to the camera’s rotation axis. Although the results suggest that our approach is tolerable to a certain degree of errors, it is necessary in future studies to quantify the tolerance and evaluating the impact of violations of our assumptions.

References [1] S. E. Chen. QuickTimeVR - an image-based approach to virtual environment navigation. In Proc. SIGGRAPH’95, pages 29–38, Los Angeles, California, USA, August 1995. 335 [2] O. Faugeras. Three-Dimensional Computer Vision: A Geometric Viewpoint. The MIT Press, London, England, 1993. 337 [3] P. E. Gill, W. Murray, and M. H. Wright. Practical Optimization. Academic Press, London, 1981. 343 [4] F. Huang, S.-K. Wei, and R. Klette. Geometrical fundamentals of polycentric panoramas. In Proc. ICCV’01, pages I:560–565, Vancouver, Canada, July 2001. 335 [5] H. Ishiguro, M. Yamamoto, and S. Tsuji. Omni-directional stereo. PAMI, 14(2):257–262, 1992. 335 [6] R. Klette, K. Schlüns, and A. Koschan. Computer Vision - Three-Dimensional Data from Images. Springer, Singapore, 1998. 336 [7] L. McMillan and G. Bishop. Plenoptic modeling: an image-based rendering system. In Proc. SIGGRAPH’95, pages 39–46, Los Angeles, California, USA, August 1995. 335 [8] H.-Y. Shum and L.-W. He. Rendering with concentric mosaics. In Proc. SIGGRAPH’99, pages 299–306, Los Angeles, California, USA, August 1999. 335 [9] H.-Y. Shum and R. Szeliski. Stereo reconstruction from multiperspective panoramas. In Proc. ICCV’99, pages 14–21, Korfu, Greece, September 1999. 335 [10] R. Y. Tsai. A versatile camera calibration technique for high-accuracy 3d machine vision metrology using off-the-shelf tv cameras and lenses. IEEE Journal of Robotics and Automation, 3(4):323–344, 1987. 337 [11] S.-K. Wei, F. Huang, and R. Klette. Visibility determination in depth-layered cylinder scene representations. In Proc. IVCNZ’99, pages 175–180, Christchurch, New Zealand, August 1999. 335 [12] J.-Y. Zheng and S. Tsuji. Panoramic representation for route recognition by a mobile robot. IJCV, 9(1):55–76, 1992. 335

Quantitative Depth Recovery from Time-Varying Optical Flow in a Kalman Filter Framework John Barron1 , Wang Kay Jacky Ngai1 , and Hagen Spies2 1

Department of Computer Science, University of Western Ontario London, Ontario, Canada [email protected],[email protected] 2 ICG-III: Phytosphere, Research Center J¨ ulich 52425 J¨ ulich, Germany [email protected]

Abstract. We present a Kalman filter framework for recovering depth from the time-varying optical flow fields generated by a camera translating over a scene by a known amount. Synthetic data made from ray traced cubical, cylinderal and spherical primitives are used in the optical flow calculation and allow a quantitative error analysis of the recovered depth. Keywords: Depth Map, Depth from Optical Flow, Kalman Filter, 3D Camera Motion, Quantitative Error Analysis

1

Introduction

We consider the problem of depth recovery from monocular image sequences when the 3D camera motion is either known [10] or recovered from a Motion and Structure algorithm [2]. Using the assumption of local planarity, inverse depth can first be computed from the measured optical flow, then integrated into surface normals in small neighbourhoods and tracked over time in a Kalman filter framework [7, 6]. There are a number of approaches for computing depth from optical flow and/or image derivatives in the literature. Piecewise planarity has been used by Faugeras and Lustman [5] to solve for motion and structure from point displacements using recursive least squares. Barron et al. [1] also used the planarity assumption to recover motion and structure from time varying optical flow. More recent work has used Kalman filtering as a way to recover depth values from time varying flow where the camera motion is known. Kalman filtering seems especially appropriate here as, from empirical observations by the 1st author, optical flow error (as computed from Lucas and Kanade’s algorithm [8]) has Gaussian mean-zero error. This was conjectured after examining plots of optical flow error for a number of images that looked Gaussian. Also, use of the Lorentzian robust T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 346–355, 2003. c Springer-Verlag Berlin Heidelberg 2003

Quantitative Depth Recovery from Time-Varying Optical Flow

347

estimator [4] with the Lucas and Kanade calculation shows that the best results approached the standard least squares Lucas and Kanade results: least squares is optimal for Gaussian mean-zero noise.

2

Literature Survey

Some of the Kalman filter approaches include those by Matthies et al. [12], Heel [9] Xiong and Shafer [15] and Hung and Ho [10]. Matthies et al.’s approach [12] is a Kalman filter-based algorithm for recovering the depth map of a 3D scene from 2 frame image displacements, using the sum of squared differences (SSD) method to integrate the squared intensity difference between two shifted images over a small area to obtain an estimate of the optical flow, (v1 , v2 )T , and its covariance matrix, Pm , for each pixel of the image. A Kalman filter tracks the inverse depth using predictions about how the depth changes. Heel [9] computed depth directly from image intensity derivatives in a Kalman filter framework by assuming local constant depth. If the camera motion is unknown it can also be estimated in a least squares framework. Given (X(t), Y (t), Z(t)) (X(t + 1), Y (t + 1), Z(t + 1)) can be estimated and bilinear interpolation is then used to find the Z value at pixel locations. Further interpolation and extrapolation are used to compute dense fields. Heel has tested his algorithm on planar images with unknown translation with good results. Xiong and Shafer [15] use an extended Kalman filter to track the sensor’s motion and the depth of each image pixel. The Levenberg-Marquardt method is used here to perform the nonlinear minimization to get an initial estimate of the motion and depth parameters. Their experiments include estimating camera motion and surface depth in a 51 image sequence generated by a moving camera over a scene of a straw hat, with realistic subjective surface results. Hung and Ho [10] use image intensity derivatives in a Kalman filter framework for known camera translation. At each time, the depth map is warped using the known camera motion and used to compute depth at pixel points. An experiment with a scene consisting of a coke can, a small box and a planar poster as background, show that with smoothing, good subjective depth can be recovered.

3

A New Approach

We also assume that the direction of sensor translation, u ˆ, and its rotation, ω, are known or can be computed by some other means [2]. In the case where the true sensor translation U is known, we can compute the absolute 3D depth map; otherwise we can compute relative depth. Interpolation of surface orientation values at non-pixel image locations is avoided by assuming local planarity, a seemingly reasonable assumption everywhere in the image except at depth discontinuities. Our use of planarity is different than Faugeras and Lustman’s [5]: we use it to avoid having to compute non-pixel correspondences. Given image velocity v(Y , t) we believe it valid most of the time to assume the correspondence is at round(Y + vδt) at time t + δt.

348

John Barron et al.

We consider a setup consisting of a single camera taking the images while it is moving through a static 3D scene. The standard image velocity equations [11] relate a velocity vector measured at image location Y = (y1 , y2 , f ) = f P /X3 , [i.e. the perspective projection of a 3D point P = (X1 , X2 , X3 )], to the 3D sensor translation U and 3D sensor rotation ω. We can rewrite the standard equation for the image velocity v = (v1 , v2 ) as v(Y , t) = v T (Y , t) + v R (Y , t) where v T and v R are the translational and rotational components of image velocity: v T (Y , t) = A1 (Y )

U X3

and

v R (Y , t) = A2 (Y )ω(t)

(1)

and A1 (Y ) =

−f 0 y1 0 −f y2

and A2 (Y ) =

y1 y2 f y2 (f + f2 )

−(f +

y12 f )

− y1fy2

y2 −y1

.

(2)

We define the depth scaled camera translation as u(Y , t) =

U (t) = u ˆµ(Y , t), ||P (t)||2

(3)

ˆ = (u1 , u2 , u3 ) is the normalized direction of translation and where u ˆ = U ||U ||2 ||2 µ(Y , t) = ||P ||2 = |Xf3||U |||Y ||2 is the depth scaled sensor speed at Y at time t. We refer to µ as scaled speed. Note that translational image velocity, v T , is bilinear in u ˆ and µ, making the image velocity equations non-linear. The focal length f is assumed to be known via some camera calibration scheme. If we define 2 vectors: r(Y ) = (r1 , r2 ) = |v − A2 (Y )ω| and ||Y ||2 d(Y ) = (d1 , d2 ) = |A1 (Y )ˆ , u| f

(4) (5)

where |A| means each element in the vector is replaced by its absolute value. Then we can solve for µ from the image velocity equation, which can now be written as: r − dµ = (r1 , r2 ) − (d1 , d2 )µ = 0. (6) µ has solutions

r1 d1

or

r2 d2

or (best) a weighted average: µ=

r1 |v1 | d1

+

r2 |v2 | d2

|v1 | + |v2 |

.

(7)

We use the magnitudes of v1 and v2 to weight the calculation: we expect µ values computed from the larger velocity component magnitude to be more reliable.


4

349

Planar Orientation from Relative Depth

We are interested in computing the local surface orientation as a unit normal vector, α ˆ = (α1 , α2 , α3 ) from µ values. Consider two 3D points, P1 =

(X11 , X12 , X13 ) and P2 = (X21 , X22 , X23 ), with images Y 1 = fXX1311 , fXX1312 , f and Y 2 = fXX2321 , fXX2322 , f . If they lie on the same 3D plane then: α ˆ · Y1 X23 = . α ˆ · Y2 X13

(8)

This equation gives the ratio of the 3rd coordinates (X3 ) of two 3D points in terms of their image locations and their planar surface orientation (assuming ||2 f ||U||2 they lie on a common 3D plane). From the definition of µ = ||U ||P ||2 = |X3 |||Y ||2 we can write f ||U ||2 X3 = . (9) µ||Y ||2 yielding:

α ˆ · Y1 µ1 ||Y 1 ||2 = . α ˆ · Y2 µ2 ||Y 2 ||2

(10)

From the planar equation α ˆ ·P = α ˆ · Xf3 Y = c and using equation (9) we obtain: α ˆ·Y =

cµ||Y ||2 ||U ||2

(11)

We can solve for αˆc by setting up a linear system of equations, one for each pixel in a n × n neighbourhood where planarity has been assumed and using a standard least squares solution method [14].

5

The Overall Calculation

If we assume that u ˆ (or indeed U ) and ω are knowns, we need only concern ourselves with the surface orientation step of the calculation. At the initial time, t = 1: 1. Given u ˆ and ω, we compute all the µ’s as described above (see [2] for one way of computing u ˆ and ω). 2. In each n × n neighbourhood centered at a pixel (i, j) we compute ( αˆc )(i,j) at that pixel using equations (7) and (11). We call these computed αˆc ’s the measurements and denote them as g M(i,j) . Given these measurements, g M(i,j) , we can recompute the µ(i,j) ’s as: µ(i, j) =

(g M(i,j) · Y (i,j) )||U ||2 ||Y (i,j) ||2

(12)

350

John Barron et al.

The recomputed µ values are more “smoothed” than the actual measurements and better represent the scene shape. These µ(i, j) values are currently the best estimate of the scene’s dense shape. Note that we can obtain µ values for pixels with no optical flow, these are computed from the image velocities in its neighbourhood (which are assumed to result from the same local planar patch). At time t = 2: 1. Given the measurements or best estimates of u ˆ and ω, we compute µ at each pixel location and then compute all g M(i,j) ’s in the same way described above for the new optical flow field. Using the image velocity measurements at time t = 2, we use the best estimate of surface orientation at time t = 1 at location Y − v (∆t = 1) plus the measurement at Y and its covariance matrix to obtain a new best estimate at Y at time t = 2. We do this at all Y locations (where possible), recompute the µ values via equation (12) and output these as the 3D shape of the scene. At time t = i we proceed as for time t = 2, except we use the best µ estimates from time t = i − 1 instead of time t = 1 in the Kalman filter updating. Note that if the true sensor translation U is known, the absolute 3D depth X3 can be computed everywhere from the filtered µ values: X3 =

6

f ||U ||2 . µ||Y ||2

(13)

The Kalman Filter Equations

We note here that the components of αˆc in equation (11) are not independent, thus we have a covariance matrix with non-zero off diagonal elements in the Kalman filter equations. [In [2], the components of all 2D and 3D vectors, u ˆ and ω respectively were treated as 1D variables in the Kalman filter framework.] If we assume u ˆ and ω are known, we can compute αˆc at each pixel (i, j) as outlined above, assuming local planarity. For the purposes of understanding the equations below we will subscript symbols with a M to indicate measured quantities (computed from the image velocities), P to indicate predicted quantities and C to indicate the computed quantities (the current best estimates). We use g to denote αˆc . It is a 3D quantity, so we need an initial predicted value and an initial covariance matrix at time t = 0: g P(i,j) = 0,   ∞ 0 0 CP(i,j) =  0 ∞ 0  . 0 0 ∞

(14) (15)

This definition of CP says that initially the coefficients of g are independent and we have no confidence in their estimates.


351

For each time t = 1, 2, 3, ... at all pixels (i, j), we use equations (7) and (11) to make a measurement of g M(i,j) and an estimate of its covariance matrix CM(i,j) , respectively. Then using the previous best surface orientation estimate at time t−∆t at image location Y −v∆t, denoted by (i− , j − ), the Kalman filter equations are computed [12] as follows: −1 , (16) K(i,j) = CP(i− ,j− ) CP(i− ,j− ) + CM(i,j) and (17) g C(i,j) = g P(i− ,j− ) + K(i,j) g M(i,j) − g P(i− ,j− ) CC(i,j) = CP(i− ,j− ) − K(i,j) CP(i− ,j− ) = K(i,j) CM(i,j) .

(18)

Because g, i.e. αˆc , can be rotated by sensor rotation, the predicted values g P(i,j) must take this into account in their update: g P(i,j) = R(ω, t + ∆t)RT (ω, t)g C(i,j)

and

CP(i,j) = CC(i,j) .

7

(19) (20)

Abnormal Situations and their Resolution

While tracking individual surface orientations, a number of situations may arise. It is possible that: 1. When tracking a surface orientation at a moving pixel, no surface orientation can be computed at the latest time, in which case tracking stops. 2. A new, untracked surface orientation can be computed, in which case tracking starts. 3. A surface orientation is tracked to a wrong pixel, in which case the tracking continues from that wrong pixel as no error recovery is possible or detectable. 4. The surface orientations at two different pixels with (perhaps) different surface orientations track to the same pixel, in which case the surface orientations are combined and then tracked as a single surface orientation (a weighted average using the covariance matrices as weighting matrices).

8

Generation of Synthetic Test Image Sequences

For the experiments, we use synthetic images because that allows for a quantitative error analysis on the estimated depth values. We generate a 30 image sequence of 512 × 512 images using each of three different 3D objects: a sphere, a cube and a cylinder (Figures 1a, 1b and 1c). For each experiment, a sequence of images of one of these three objects is generated while the synthetic camera is moving in a known way and the object is rendered by perspective projection onto the image plane. In all the experiments we performed, all the image sequences were generated using a camera translation of (1, 1, 0) with a focal length of 1000. All the objects have a marble texture to facilitate optical flow computation. Although the synthetic images themselves are error free, the optical flow computed using them is not, as can be seen in Figure 2a through 2c.

352

John Barron et al.

Fig. 1. Synthetic test data: (a) A marble-texture cube with sides of length 300 with its center located at (0,0,1500), (b) A marble-texture cylinder with two end spheres of radius 200 and a wall of length 400 with its center located at (0,0,1500) and (c) A marble-texture sphere of radius 200 with its center located at (0,0,1300)

Fig. 2. The optical flow of the (a) cube, (b) cylinder and (c) sphere

9

Experimental Results and Discussion

Error was measured for relative depth µ using the exact µ values: hence the use of the term absolute relative error. Optical flow was computed using (see Figures 2a, 2b and 2c) Lucas and Kanade’s algorithm [8] with differentiation performed as proposed by Simoncelli [13]. Figures 3a to 3c shows the histograms of the error distribution for the cube data for the 1st , 22nd and 25th frames. Figures 4a to 4c shows the histograms of the error distribution for the cylinder data for the 1st , 22nd and 25th frames. Figures 5a to 5b shows the histograms of the error distribution for the cylinder data for the 1st , 17th and 25th frames. Note that at depth discontinuities, where the local planarity assumption is definitely violated, no depth is recovered as the Kalman filter rejects those values as unreliable. This also supposes image velocities can be computed at a depth

12

12

10

10

10

8

6

4

8

6

4

2

2

0

0 0

15

30 45 60 75 Absolute Relative Error (%)

Percentage of Depth Values (%)

12




90

353

8

6

4

2

0 0

15

(a)


90

0

15

(b)


90

(c)

12

12

10

10

10

8

6

4

2


12



Fig. 3. The histogram of the absolute relative errors of the estimated depth values at the (a) 1st , (b) 22nd and (c) 24th images of the cube sequence using our approach

8

6

4

2

0 15


(a)

90

6

4

2

0 0

8

0 0

15


(b)

90

0

15


90

(c)

Fig. 4. The histogram of the absolute relative errors of the estimated depth values at the (a) 1st , (b) 22nd and 24th images of the cylinder sequence using our approach

discontinuities. Normally Lucas and Kanade optical flow does not yield velocity values at discontinuities. Table 1 shows the average absolute relative error for the 3 objects for a number of ranges. The quantitative error results were best for the cube (the most planar) and worst for the sphere (the most curved). Obviously, the planarity assumption is not always valid and we plan on replacing this with a higher order parametric model. We have already seen that higher order parametric models to account for surface non-planarity were sometimes necessary to get better optical flow [3].

10

Conclusions

We have presented a new algorithm to compute dense accurate depth using a Kalman filter framework. We need to test our algorithm for camera motions including rotation and to use real data (preferably with ground truth). Lastly, we plan to integrate this algorithm into the Kalman filter based motion and structure algorithm designed earlier [2].

John Barron et al. 12

12

10

10

10

8

6

4

8

6

4

2

2

0

0 0

15


(a)

90


12



354

8

6

4

2

0 0

15


90

0

(b)

15


90

(c)

Fig. 5. The histogram of the absolute relative errors of the estimated depth values at the 1st , (b) 17th and (c) 24th images of the sphere sequence using our approach Table 1. The percentage of the estimated depth values that have certain absolute relative errors in the experiments for the cube, cylinder and sphere Percent Relative Error 1st depth map (Cube) 22nd depth map (Cube) 25th depth map (Cube) 1st depth map (Cylinder) 22nd depth map (Cylinder) 24th depth map (Cylinder) 1st depth map (Sphere) 17th depth map (Sphere) 24th depth map (Sphere)

>15% 42.92% 21.47% 20.47% 49.85% 37.38% 36.65% 65.66% 61.25% 61.29%

5% - 15% 34.70% 41.80% 42.35% 29.90% 33.81% 34.33% 20.79% 23.40% 23.01%

4. Then in the corresponding graph, there are p3 vertices that represent all the possible triplets with those p planes. As p > 4, there exist two vertices v1 and v2 which labels do not have two planes in common. Therefore, the edge (v1 , v2 ) does not exist in the graph. Consequently, this graph cannot be a clique. As a consequence, a clique search in the graph will never find correct cliques with more than 4 vertices. Then, with such an algorithm, we never reconstruct some vertices which are adjacent to more than 4 planes. We have to characterize those vertices adjacent to more than 4 planes in the graph in order to find an algorithm to detect them.

From Digital Plane Segmentation to Polyhedral Representation

361

1,2,4

1,2,3

1,2,3 2,3,4

1,3,4

2,3,4

1,3,4

(a)

(b)

Fig. 2. Illustration of Proposition 1: (a) a non valid clique with 4 different planes but only 3 = 43 vertices ; (b) a valid clique with 4 planes and 4 = 43 vertices

Proposition 3. Let v be a polyhedron’s vertex adjacent to p planes, with p > 4. Then the adjacency graph related to this vertex contains p cliques of size 4 that are not disjoint and such that if we define a new adjacency relation as “having one vertex in common”, those p cliques form a cycle. Proof. Let v be such a vertex, and note (0, 1, 2, 3, 4, 5, . . . p) the planes it belongs to. In the adjacency graph, all the quadruples (n, n + 1 (mod p), n + 2 (mod p), n + 3 (mod p)), 0 ≤ n ≤ p are cliques of size 4 with vertices (n, n + 1 (mod p), n + 2 (mod p)), (n + 1 (mod p), n + 2 (mod p), n + 3 (mod p)), (n + 2 (mod p), n + 3 (mod p), n) and (n + 3 (mod p), n, n + 1 (mod p)) . Two consecutive quadruples have three planes in common, so the corresponding cliques have one vertex in common in the graph. If we introduce the adjacency relation exposed in the proposition, and reduce each clique into one point, we then get a cycle of size p. The cycles containing all the quadruples given above are called valid cycles. This proposition gives a characterization of the vertices that are adjacent to more than 4 planes. With this characterization, we are now able to propose an algorithm to determine vertices and edges. Algorithm 1 extracts from the adjacency graph the structures that correspond to the polyhedron’s vertices. Let us see each step in details. The first step extracts from the graph the basic element of any candidate vertex, i.e. the valid cliques of size 4. The result of this operation is uniquely determined by the adjacency graph, and so by the DP segmentation: Proposition 4. The result of the clique extraction done in the first step of the algorithm minimum vertices is uniquely determined by the initial DP segmentation. Proof. Suppose that there exists two different cliques covers. This means that a choice between two distinct cliques C1 and C2 is done during the clique covering algorithm. Such choice implies that the clique C1 must invalidate the clique C2 ,

362

Isabelle Sivignon and David Coeurjolly

Algorithm 1 Minimum number of vertices Polyhedral representation \EXTERNNAME \INTERNNAME{minimum vertices(G)} \STATE Extract the cliques of size 4 in the adjacency graph, keeping only the valid cliques. Two cliques can have a common vertex, but no common edge. \STATE Reduce the graph a first time: – vertices are the cliques ; – two types of edges that correspond to two different adjacency relations: • the former edges remain in the reduced graph after contraction of the cliques (edges of type 1). • a new adjacency relation is introduced: two vertices are linked by an edge when the corresponding cliques have a common vertex (edges of type 2). 1: In this graph, extract the minimum number of valid cycles using only edges of type 2. 2: Reduce the graph a second time: – vertices are the cycles ; – edges of type 1 remain.

or conversely. In other words, since two cliques of the decomposition can share a vertex, C1 and C2 must share at least an edge. However, according to the definition of valid cliques, two valid cliques cannot share an edge. Therefore, C1 or C2 is not valid and the clique covering algorithm has no choice. Hence we prove the proposition. During the second step of the algorithm, a first reduction is done: each clique found during step 1 is reduced to one new vertex of the graph. During this reduction, the labels of all the vertices of each clique are given to the representing vertex. An edge of type 1 is drawn between two vertices when the cliques corresponding to those vertices were linked by at least one edge. Moreover, new edges are added: edges of type 2 are the ones that take part in the cycles introduced in Proposition 3: they represent the fact that two cliques that share three planes should be part of the same vertex. After the reduction of step 2, the graph with edges of type 2 should be disconnected, and each connected component should contain a unique cycle (so a vertex, cf Proposition 3). We note that if one connected component does not contain a cycle or contains many cycles, after reduction we get many different vertices that can share 3 planes. That is in contradiction with the definition of a vertex. This case is due to the discrete structure of the objects and an example of such a case is shown in Figure 6. The cycle covering extracts disjoint cycles and the decomposition we get is not unique, once more because of the discrete structure of the initial object (see Figure 3 for an example). In the last step, a second reduction is done to get to the final graph representation of the polyhedron’s structure. Edges of type 1 are the ones that represent


363

3,4,5,1

4,5,1,2

1,2,3,4,5

2,3,4,5

5,1,2,3

3,4,5,1

4,5,1,2

1,2,3,4

2,3,4,6 6,1,2,3

2,3,4,6 6,1,2,3

2,3,4,5

5,1,2,3

3,4,6,1

3,4,6,1

1,2,3,4,6

4,6,1,2

(a)

(b)

4,6,1,2

(c)

Fig. 3. The cycle extraction is not unique: (a) graph before cycle extraction; (b) and (c) graph after cycle extraction and reduction with two different strategies

the fact that two vertices share two planes. So after the second reduction done in step 4, the remaining edges correspond to the polyhedron’s edges. The final graph obtained after this algorithm is a description of the corresponding polyhedron: the graph’s vertices are the polyhedron’s vertices, and the graph’s edges are the polyhedron’s edges. Moreover, the vertices labels provide the planes adjacent to each vertex. If we apply this algorithm to an Euclidean polyhedron adjacency graph, after the second step we get many connected components, each one corresponding to one polyhedron’s vertex, and containing exactly one valid cycle. Hence, from a Euclidean adjacency graph, after reduction of the cycles, there is a bijection between graph and polyhedron vertices and edges. If we consider now a discrete surface, since all reduction steps consider the minimum number of cliques and cycles, no adjacency graph with less vertices than the final graph can be built. Hence, the obtained polyhedron is optimal, according to the number of vertices. In a computational cost point of view, both the minimum clique covering of step 1 and the minimum vertex disjoint cycle cover of step 3 are NP-complete in the general case [17, 13]. In other words, no exact solution can be found in a polynomial time in the number of vertices in G. However, these two steps are equivalent to classical graph coloring problems [17] and thus, many approximated solutions can be found using efficient algorithms. For example, polynomial in time algorithm exists that approximates the solution of the minimum clique covering problem at a factor 2 of the optimal solution [6].

4

Examples

In this section, we present two examples of the algorithm different steps. The two objects studied are synthesized objects. The first object represented on Figure 4 is composed of two pyramids with square basis that have a common basis. The DP segmentation finds 8 pieces of discrete planes, which fit with the object’s 8 faces. Figure 4 (b) represents the

364


4,0,7

4,0,7

8,7,0

1,8,7

7,0,1

0,1,8

4,0,1

5,4,0

0,1,5

1,5,4

8,1,5

0,7,12 13,8,1

8,13,12

(a)

12,4,0

1,5,13

12,13,5

4,12,13

5,13,8

13,5,4

5,4,12

12,4,0

7,12,4

7,8,13

12,7,8

13,12,7

0,7,12

7,12,4

(b)

Fig. 4. Example of the construction of the graph: (a) DP segmentation of the object ; (b) corresponding adjacency graph

adjacency graph corresponding to this DP segmentation. In the Figure 5 (a), the first step of the algorithm has been processed, and only the valid cliques found are drawn. The last graph (Figure 5 (b)) is the one obtained after step 2, 3 and 4. Indeed, as the cliques have no vertex in common, there is no edge of type 2 in the reduced graph. Then step 3 and 4 are useless, and the reduced graph is the final graph. This graph is composed of 6 vertices and 12 edges which also are the polyhedron’s vertices and edges. The graph vertices’s labels contain the information about the planes adjacent to each polyhedron’s vertex. The second example is a synthesized pyramid with five lateral faces. The DP segmentation retrieve those 5 faces and the basis as shown on the Figure (one color for each plane). The first graph (Figure 6 (b)) is the adjacency graph. Step 1 determines 7 cliques on this graph: 2 cliques of size 4 and 5 isolated points. With the reduction done in step 2, we get the graph drawn on Figure 6 (c): there is only one edge of type 2 between the two vertices obtained after reduction of the two cliques of size 4. This edge is not a valid cycle, so step 3 doesn’t modify the graph. Finally, we get a graph with 7 vertices and 11 edges. As we already noticed, this graph highlights some problems that are tightly linked to the discrete structure of the object: indeed, in this graph, we get 2 vertices that have three planes in common in their labels. This type of problem will have to be solved for the embedding in the Euclidean space.


4,0,7

365

12,4,0

4,0,1

5,4,0

0,1,5

1,5,4

8,7,0

1,8,7

8,1,5

1,5,13

12,13,5

4,12,13

7,0,1

0,1,8

13,8,1

5,13,8

13,5,4

5,4,12

12,4,0

4,0,7

0,7,12

7,12,4 0,4,7,12 0,1,4,5

8,13,12

7,8,13

12,7,8

13,12,7

1,5,8,13 0,1,7,8

4,5,12,13

7,12,4

0,7,12

7,8,12,13

(a)

(b)

Fig. 5. Vertices and edges extraction: (a) after the minimum clique covering and sorting out ; (b) reduction of the cliques to detect edges

4,5,1

4,5,1

4,7,1

4,5,7

4,7,1

4,5,10

4,5,7,10 5,7,10 7,10,13

7,10,4

5,7,10,13 13,5,7

10,13,5

13,7,1

1,5,10 13,10,1

(a)

(b)

13,7,1

1,5,10 13,10,1

(c)

Fig. 6. Example of the construction of the graph: (a) DP segmentation of the object ; (b) corresponding adjacency graph and (c) final polyhedral representation

366

5


Conclusion

In this paper, we have presented both theoretical and practical aspects of the discrete volume polyhedrization problem. The objective was to define and locate vertices of the optimal discrete object polyhedral representation. The optimality criterion we have defined is based on the minimum number of vertices. Hence, given a digital plane segmentation of a discrete volume surface, we have presented an algorithm based on classical graph theory tools in order to extract from this segmentation, an optimal polyhedral representation, i.e. vertices and faces adjacencies. The next step of this work consists in the embedding of this polyhedral representation into the Euclidean space in order to give a complete reversible polyhedron with minimal number of vertices associated to the discrete object.

References [1] Ph. Borianne and J. Fran¸con. Reversible polyhedrization of discrete volumes. In DGCI’94, pages 157–168, Grenoble, France, sept. 1994. 357, 358 [2] L. Buzer. An incremental linear time algorithm for digital line and plane recognition using a linear incremental feasibility problem. In A. Braquelaire, J.-O. Lachaud, and A. Vialard, editors, DGCI’02, volume 2301 of Lect. Notes of Comp. Sci., pages 372–381. Springer Verlag, 2002. 356, 358 [3] J. M. Chassery, F. Dupont, I. Sivignon, and J. Vittone. Recognition of digital naive planes. In ICIAP’01 11th International Conference on Image Analysis and Processing, pages 662–636, September 2001. 357, 358 [4] I. Debled-Rennesson and J.-P. Reveillès. An incremental algorithm for digital plane recognition. In DGCI’94, pages 207–222, September 1994. 356, 358 [5] J. Fran¸con, J. M. Schramm, and M. Tajine. Recognizing arithmetic straight lines and planes. In S. Miguet A. Montanvert and S. Ubéda, editors, DGCI’96, volume 1176 of Lect. Notes of Comp. Sci., pages 141–150. Springer Verlag, 1996. 356, 358 [6] D. S. Hochbaum, editor. Approximation algorithms for NP-hard problems. PWS Publishing Company, 1997. 363 [7] C. E. Kim and A. Rosenfeld. Convex digital solids. IEEE Trans. on Pattern Anal. Machine Intell., PAMI-4(6):612–618, 1982. 356, 358 [8] C. E. Kim and I. Stojmenović. On the recognition of digital planes in three dimensionnal space. Pattern Recognition Letters, 32:612–618, 1991. 356, 358 [9] R. Klette. Digital Geometry - The birth of a new discipline, chapter 1. 2001. Retirement of A. Rosenfeld. 357 [10] R. Klette and H. J. Sun. Digital planar segment based polyhedrization for surface area estimation. In C. Arcelli, L. P. Cordella, and G. Sanniti di Baja, editors, International Workshop on Visual Form 4, volume 2059 of Lect. Notes Comput. Sci., pages 356–366. Springer-Verlag, 2001. 357, 358 [11] L. Papier and J. Fran¸con. Polyhedrization of the boundary of a voxel object. In M. Couprie G. Bertrand and L. Perroton, editors, DGCI’99, volume 1568 of Lect. Notes of Comp. Sci., pages 425–434. Springer Verlag, 1999. 357, 358 [12] J.-P. Reveillès. Géométrie discrète, calcul en nombres entiers et algorithmique. PhD thesis, Université Louis Pasteur, 1991. 357


367

[13] Carsten Thomassen. On the complexity of finding a minimum cycle cover of a graph. SIAM Journal on Computing, 26(3):675–677, June 1997. 363 [14] P. Veelaert. Digital planarity of rectangular surface segments. IEEE Trans. Pattern Anal. Machine Intell., PAMI-16:647–653, 1994. 356, 358 [15] P. Veelaert. Concurrency of line segments in uncertain geometry. In A. Braquelaire, J.-O. Lachaud, and A. Vialard, editors, DGCI’02, volume 2301 of Lect. Notes of Comp. Sci., pages 289–300. Springer Verlag, April 2002. 357 [16] Joëlle Vittone. Caractérisation et reconnaissance de droites et de plans en géométrie discrète. PhD thesis, Université Joseph Fourier, Grenoble, France, 1999. 356, 358 [17] D. B. West. Introduction to Graph Theory. Prentice Hall, 2 edition, 2001. 360, 363

Lines as the Fundamental Unit of Vision Patrick Baker University of Maryland, College Park MD 20742, USA [email protected]

Abstract. We explore the consequences of using only lines as our object in computer vision. We show a new constraint, which we call the prismatic line constraint. This constraint is based on the reconstruction of local shape using line measurements and rotation only, which is a new reconstruction in computer vision. We show that the point trilinear constraint can be broken down into the epipolar constraint and constraints on lines, which are thus the only constraints which need to be considered in computer vision.

1

Introduction

In this paper, we show how lines rather than points can be used as the basic atom of computer vision. This new view allows us to derive a new constraint, which we call the prismatic line constraint, and also provides a new perspective on the relationships between the epipolar and trilinear constraints. The prismatic line constraint is a new constraint on only local shape and rotation, in a sense that allows us to find local shape without knowing camera positions, or even correspondence (in a sense). In this paper we explain the derivation and intuition behind this constraint. We also show a new direct proof that the point trilinear constraint is composed exactly of the epipolar equations and the line trilinear constraint, assuming there are at least three world points. The reason that we take lines as our measurement atom rather than points is because that is what our signal processing operators are best at. For instance, when we take a gradient measurement, we are measuring the perpendicular to an isophote, or a line. To find points we often intersect two lines, so points are usually a secondary measurement, and by intersecting the lines, we lose valuable statistics about their angle of intersection and orientation. Most feature detectors have an orientation at which they match, and this property is much better represented by two lines rather than a point, which destroys the orientation information. This paper shows that we get a definite advantage by considering lines, in that rotation and local shape can be separated out from the line correspondence, translation, and depth. In order to derive these relationships, we step back from the tensor formalism in which much of the constraint work is done, and use only geometry and vector algebra. To that end, we base all our constraints on the reconstruction and intersection of lines in space. We reconstruct the Pl¨ ucker coordinates of a line in space and then use these equations to form all our constraints in simple proofs. T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 368–385, 2003. c Springer-Verlag Berlin Heidelberg 2003

Lines as the Fundamental Unit of Vision

369

Fig. 1. The Prismatic Line Constraint operates on parallel lines

The prismatic line constraint is a constraint on the rotation of three cameras based on images of lines parallel in space, as in figure 1. The equation was first noted in the context of computer vision in [5], but it was not stated that these lines do not have to be identical. They could be images of parallel lines from, say, the edges of a prism. Thus while we use the same equation as in [5], we operate on different objects, and thus this is a new constraint. It is important to note that the position of the cameras is irrelevant to the constraint. The constraint has an accompanying reconstruction, which says that we can find the direction of a line (or set of parallel lines) in space from the images of the lines in two cameras, and we do not need to know the position of the cameras. If we have a planar patch containing two lines in nonparallel directions, and we have corresponded these lines in two cameras, we can then find the surface normal, again with no knowledge of camera position, as in figure 2. A new result of this paper is that local shape is independent of camera position. While this is a trivial result in this formalism, it has not been to our knowledge stated before, and it is an important observation. We see that local shape is an inherently rotational property. If we have calibrated our cameras rotationally and not translationally, it is still possible to find local shape without having to take derivatives of depth. If in addition, we have sets of parallel lines embedded in a plane (or just a texture with bidirectionality), then we do not need to correspond particular lines to each other, but only which directions correspond in both cameras, as in figure 3. This is why we say that local shape is independent of translation and correspondence. The prismatic line constraint is contained in the standard point trilinear constraint, as we show in this paper, but because of this separation between translation and rotation it is advantageous to consider it separately. We show that the line trilinear constraint is composed of the point trilinear constraint

370

Patrick Baker

Fig. 2. A surface normal can be found from projections of two intersecting lines

Fig. 3. A surface normal can also be found from projections of two pairs of differing but parallel lines

together with a depth consistency constraint, so that we can consider all our constraints to be spanned by: 1. The epipolar constraint 2. The prismatic line constraint 3. The depth consistency constraint The first one ensures that image points are aligned properly. The second one ensure that lines lie in the same direction. The third ensures that the depth of a line calculated from pairs of cameras is consistent.


2

371

Projecting Points and Lines

We define world points in three dimensions as one would expect. Definition 1. A world point is an element P ∈ R3 . We may represent such a point in a particular coordinate system as:   X P = Y  Z Our image points exist in the projective plane P2 . This space has the advantage of duality between points and lines, which allows us to consider both and transform formulas which consider one into formulas which consider the other. Definition 2. An image point is an element p ∈ P2 . We may represent such a point in a particular coordinate system as:   x p = y  z These coordinates are homogeneous. Note that the coordinates of the world and image points are both 3-vectors, so our representation for both coordinates is identical. In fact, we can think of a camera as as device for considering the coordinates of a point in R3 to be coordinates in P2 . However, our world points exist in one fiducial coordinate system, while the image points exist in the particular camera coordinate systems. Therefore our camera is defined in relation to this fiducial coordinate system as follows. Definition 3. A camera C is a map C : R3 → P2 from world points to image points. Given a fiducial coordinate system, we may represent this map with a pair (B, T ), where B : R3 → R3 is a linear function (represented by a 3×3 matrix), and T is a 3-vector representing the camera center. The action of the map on a world point coordinate P is: C(P ) = B · (P − T ) where C(P ) is considered as a member of P2 . Here we have defined a camera as taking a world point and mapping it to an image point through a general linear transformation on the world coordinates. Note that we have defined the transformation as first a translation and then a matrix multiplication on the world point. This allows us to easily undo the matrix multiplication on the image point by applying B −1 . Each camera will then be only a translation away from the fiducial coordinate system, which allows easier derivation of our constraints. This technique was used in [9]. Also

372

Patrick Baker

note that the matrix B is applied to the 3-vector resulting from the translations of the point P . B may be considered to be a transformation on the world points R3 or of the image points P2 . In fact, B is usually split apart into an orthogonal matrix representing a rotation of the camera (a transformation on R3 ) and the residual upper triangular matrix representing a linear transformation of the image (a transformation on P2 ). Since the coordinates are the same, we ignore such distinctions for this paper. 2.1

Lines and Cameras

Now that we see how world points are projected to image points, we can derive similar formulas for lines. First, we define a convenient coordinate system for world lines, the Pl¨ ucker coordinate system. Definition 4. A world line L is the set of all the points P ∈ R3 such that P = (1 − λ)Q1 + λQ2 for two points Qi , and some scalar λ. The Pl¨ ucker coordinates of this line are L = LLmd , where: Ld = Q2 − Q1

direction of L

Lm = Ld ×P

moment of L

Lm has many easily derived alternative definitions: Theorem 1. Given the variables defined in definition 4, the following definitions for Lm are equivalent. Lm = Ld ×P = Ld ×((1 − λ)Q1 + λQ2 ) = Ld ×Q1 = Ld ×Q2 = Q2 ×Q1 The meaning of the direction Ld of the line is clear. The moment of the line has magnitude equal to the minimum distance of the line from the origin, and is perpendicular to the plane containing the origin and the line. The coordinates of L are homogeneous, and we can consider the lines to be members of the space P5 . However, note that not every element of P5 is a line, since by our construction we must have that Ld and Lm are perpendicular, or LTm Ld = 0. We thus have parameterized the four dimensional space of 3D lines as quadric (called the Klein quadric) in P5 . The condition for incidence of lines on lines, which will form the basis for our constraint proof, can be found in [2]. Theorem 2. If we have two lines L1 and L2 , they intersect if and only if LTd,1 Lm,2 + LTd,2 Lm,1 = 0


373

We wish to define image lines, so that we may project our world lines onto the image. It is well known that in the space P2 , the lines are dual to the points, and can also be represented as homogeneous 3-vectors. One may think of the image line coordinate as a normal to the plane containing the line and the origin. The magnitude of our coordinates has no meaning in P2 . One may also think of the image line as a great circle on a sphere, with the coordinates representing the axis perpendicular to all the points of the great circle. Definition 5. An image line is a line ∈ P2 . The line may be given coordinates   1 = 2  3 A point p is incident on a line if and only if pT = 0 Since a line and point are incident if and only if their coordinates are perpendicular to each other, this provides a simple means for finding the line incident on two points Theorem 3. The line which connects two points p1 and p2 has coordinates = p1 ×p2 and the point at the intersection of two lines Theorem 4. The point p which is on both lines 1 and 2 has coordinates p = 1 ×2 The linear transformation on image lines given a linear transformation on image points is as follows: ˆ = Bp, then the image lines Theorem 5. If we transform the image points p are transformed by ˆ = B −T . Theorem 6. If we have a line L and a camera (B, T ), then the image line associated with L is ˆ = B −T (Lm − T ×Ld ) where ˆ is considered as an element of P2 . Proof. If we have two points on the world line P1 and P2 , the coordinates of their image points on a camera (B, T ) are just B(P 1 − T ) and B(P 2 − T ), considered as members of P2 . Then the image line containing them must be just ˆ = (B(P 1 − T ))×(B(P 2 − T )) = B −T (P 1 ×P 2 − (T ×(P 2 − P 1 ))) = B −T (Lm − T ×Ld )

374

Patrick Baker

BLd Since a translated and linearly transformed line is just B −T (Lm −T×L , d) we see an interesting parallel between the projection of points and lines. The coordinates of the projection of a linearly transformed world point are the same as P , just considered in P2 . The coordinates of the projection of a linearly transformed world line are equal to Lm , just considered in P2 . The projection operation for points and lines is similar, except that in the case of lines, we lose both depth and direction information, rather than just depth information, as with the points.

3

Reconstruction of Plucker Lines

This reconstruction is the heart of this paper, and all the constraints flow from it with a couple of lines of algebra. The reason that line reconstruction is made the heart of the paper is because it is well-defined, since image lines in any two distinct cameras always can be created by one, and only one, world line, though that line may be at infinity. Theorem 7. If we have a line L in space which projects to two image lines ˆ1 and ˆ2 in distinct cameras (B1 , T 1 ), and (B2 , T 2 ), then we can calculate the coordinates for L if |1 ×2 | = 0 as: 1 ×2 L= (1) 1 T T2 2 − 2 T T1 1 Proof. Using theorem 6, we know that our projected lines are i Lm − T i ×Ld

i ∈ {1, 2}

(2)

where indicates equality up to a scale factor. We assume that 1 and 2 are not parallel. Since Lm is perpendicular to Ld , then we know that the i are both perpendicular to Ld . Using this, and the fact that the i are distinct, we can calculate the Ld as: Ld = 1 ×2

(3)

Note that we have used an equals sign here, which sets a scale factor for the homogeneous coordinates of L. Since the i are linearly independent and perpendicular to Ld , and since Lm is perpendicular to Ld , we may write: Lm = γ1 + δ2

(4)

We can substitute this into equations (2), and multiply by scale factors α and β to make the equations equalities, and obtain: α1 = γ1 + δ2 − T 1 ×(1 ×2 ) = γ1 + δ2 + 2 T T1 1 − 1 T T1 2 β2 = γ1 + δ2 − T 2 ×(1 ×2 ) = γ1 + δ2 + 2 T T2 1 − 1 T T2 2


375

By equating the coefficients of the i in these equations, we obtain δ = T T1 1

γ = −T T2 2

(5)

From equations (4) and (5) we obtain Lm = 1 T T2 2 − 2 T T1 1 We have made the condition that the i not point in the same direction. The condition is equivalent to: Lm − Ld ×T 1 Lm − Ld ×T 2 which is equivalent to Lm ×(Ld ×T 1 ) = Lm ×(Ld ×T 2 ) Ld LTm T 1 − T 1 LTm Ld = Ld LTm T 2 − T 2 LTm Ld using the fact that LTm Ld = 0 LTm T 1 = LTm T 2 Lm (T 2 − T 1 ) = 0 T

which means that the translation between the cameras cannot be perpendicular to the moment vector of the world line. Translating in this plane leaves the image line the same in both cameras, so that there is no depth information in the images, and no reconstruction is possible. We may calculate the line reconstruction even if the i are parallel, and the result will just be the zero vector, which indicates that we have zero confidence in the location, which is just what we would expect. The arbitrary magnitude of our homogeneous Pl¨ ucker line can be used as a confidence measure when we actually make calculations. For instance, if we were to try to construct a model based on many views of a scene, and we want to integrate our line reconstructions, it would be helpful to know in which lines we are most confident. The magnitude of our Pl¨ ucker line could tell us which lines to use in our reconstruction.

4

Derivation of Quadrilinear, Trilinear, and Bilinear Constraints

We derive here the three major multilinear constraints used in computer vision with just a few lines of proof. We do this by using the intersection property on reconstructed Pl¨ ucker lines. To make the formulas easier to read, we consider that our actual measurements of lines are the î , while the i = Bi −T î are used in the equations. It is a simple matter to multiply the missing Bi back in so that the constraints on translation and rotation are apparent.

376

Patrick Baker

We start with the quadrilinear constraint [4], so named because it constrains the parameters of four cameras from images of a common world point. Our quadrilinear constraint is based on the reconstruction and intersection of lines in space. If we have two lines L1 and L2 , then the condition for them to intersect is shown in theorem 2. Theorem 8. If we have a world point P which projects to cameras 1 through 4 with parameters (Bi , T i ), and the image points are intersected by image lines î , then if we set i = Bi −T î , the following holds: |4 3 2 |T1 T 1 + |3 4 1 |T2 T 2 + |2 1 4 |T3 T 3 + |1 2 3 |T4 T 4 = 0

(6)

This is known as the quadrilinear constraint. Proof. Let us reconstruct the world lines from the images lines in the camera pairs (1, 2) and (3, 4). These are virtual world lines, not necessarily having any reality, but even so they can be expressed as: 1 ×2 L1,2 = 1 T T2 2 − 2 T T1 1 3 ×4 L3,4 = T T 3 T 4 4 − 4 T 3 3 Both these lines must contain the world point P . Therefore the lines must intersect. We may use the line intersection property from theorem 2 to obtain: (1 ×2 )T (3 T T4 4 − 4 T T3 3 ) + (3 ×4 )T (1 T T2 2 − 2 T T1 1 ) = 0 If we distribute the cross products into the parenthesized sums and convert (·×·)T · to | · · · |, then we obtain our result immediately. We may derive both the point trilinear [8] [7] and the epipolar equations from this equation by consider one or two pairs of cameras to be identical, meaning that we set T i = T j . We start with the trilinear constraint. Theorem 9. If we have three cameras with parameters (Bi , T i ), and a world ˆ i , then if we have image lines ˆ1 and ˆ3 which intersect point P which projects to p their respective image points, then if we set i = Bi −T î : T T1 1 T3 p2 − T T3 3 T1 p2 − (T 2 ×p2 )T (1 ×3 ) = 0 This is known as the point trilinear constraint. Proof. If we set T 4 = T 2 in (6), and let two distinct image lines which go through p2 be 2 and 4 , then we may write: T T1 1 T2 (4 ×3 ) + T T2 2 T1 (3 ×4 ) + T T3 3 T4 (2 ×1 ) + T T2 4 T3 (1 ×2 ) = 0 or by using some cross product identities: T T1 1 T3 (2 ×4 ) − T T3 3 T1 (2 ×4 ) + (T 2 ×(4 ×2 ))T (1 ×3 ) = 0 Note that whatever 2 and 4 we choose, we will end up with the same p2 = 2 ×4 . Our result follows immediately.


377

If we go one step further and identify camera 3 with camera 1, to get the epipolar equation [6]: Theorem 10. If we have a world point P projected onto two cameras with paˆ i , then if we make the usual calibration asrameters (Bi , T i ) at image points p sumption, we have the epipolar constraint: (T 1 ×p1 )T p2 + (T 2 ×p2 )T p1 = 0 Proof. If we set T 3 = T 1 in equation (9), and let two distinct image lines through p1 be 1 and 3 , we obtain: T T1 1 T3 p2 − T T1 3 T1 p2 − (T 2 ×p2 )T (1 ×3 ) = 0 or by again using cross product identities −(T 1 ×(1 ×3 ))T p2 − (T 2 ×pT2 (1 ×3 )) = 0 Again, it does not matter which 1 and 3 we choose, because we will end up with the same p1 = 1 ×3 . All these constraints are merely a consequence of intersecting two Pl¨ ucker lines. This is an intuitive derivation which corresponds to the geometry of the quadrilinear and related constraints. Also, note that no camera is singled out as special in the case of the quadrilinear and epipolar constraints. This makes these equations easier to use in a many camera system. In the trilinear case the camera with the projected point is singled out. We show later why we need not use this asymmetric trilinear constraint.

5

Derivation of Line Trilinear Constraint

The above constraints were a result of considering lines which intersect in space. Now let us consider lines which coincide in space. That is, if we have reconstructions of lines based on image lines, we want to see what constraint results if these images lines come from the same line. For this, we need three cameras, so here is another trilinear constraint: Theorem 11. If we have three cameras with parameters (Bi , T i ), and a world ˆ 2 which is on ˆ2 line L which projects to î , then if we have an image point p −1 −T ˆ ˆ i make the usual calibration/derotation: and we set i = Bi i and pi = Bi p T T1 1 T3 p2 − T T3 3 T1 p2 − (T 2 ×p2 )T (1 ×3 ) = 0 which is the same equation as in theorem 9, but with different measurements, and is known as the line trilinear constraint.

378

Patrick Baker

Proof. Using theorem 7, we may reconstruct our line as: Ld = 3 ×1 Lm = 3 T T1 1 − 1 T T3 3 If we project this line into the second camera, we obtain, from theorem 6 2 = 3 T T1 1 − 1 T T3 3 − T 2 ×(3 ×1 ) Therefore, if we consider any point p2 on 2 , we must have the constraint that: pT2 3 T T1 1 − pT2 1 T T3 3 − (p2 ×T 2 )T (3 ×1 ) = 0 Note that the while point trilinear constraint operates on image points, the line trilinear constraint operates on image lines. By varying our choices of lines, we can get four equations from the point trilinear constraint while we can only obtain two equations from the line trilinear constraint. We show how the line trilinear constraint is contained within the point trilinear constraint. 5.1

Prismatic Line Constraint

There is, however, another constraint which does not depend on the intersection or coincidence of lines. This is the prismatic line constraint. Although this equation is a consequence of the line trilinear constraint, its full meaning is better realized when based on equation (1). Instead of trying to reconstruct the whole line, we just reconstruct Ld . Given two image lines 1 and 2 , we may find Ld using equation (1). Note that we have made no reference to the position of the cameras. The constraint operates independently of any translation, whether of the cameras or of the line. In particular, the cameras may be separate or identical. Also, the image lines can have been created from a single line or two parallel lines. In the latter case, whether there is one or two cameras, we may reconstruct the direction. In the case of one camera, this is called shape from texture [10, 1] in the computer vision literature. So while the full reconstruction equation can be used to find particular lines, equation (3) can be used to find the direction of a set of parallel world lines, regardless of whether exact correspondence is known, as in figure 1. In a similar fashion to the quadrilinear constraint, we may use the shape from texture equation (3) to constrain the motion. In particular, if we have three image lines, each of which are images of one of a set of parallel world lines, then we may reconstruct the direction Ld of the world lines with (3). From the construction of the Pl¨ ucker lines, we know that any line with direction Ld must have moment vector perpendicular to Ld . Putting these two facts together, we obtain Theorem 12. If we have one, two, or three parallel world lines, and three cameras with rotation/calibration matrices Bi , then if these three cameras view images of one of our world lines as î , with the lines not necessarily the same in


379

all cameras, then if we make the derotation/calibration as above, we obtain the prismatic line constraint. T2 (1 ×3 ) = 0 or in the uncalibrated sense we obtain a constraint on our rotation and calibration as: ˆT B −1 (B1 −T ˆ1 ×B3 −T ˆ3 ) = 0 2 2 We may use the same trick as we used earlier in making B3 = B1 , which allows the case where both 1 and 2 are taken from the same camera. If these are different parallel lines, then we obtain the vanishing point constraint (ˆ2 )T B3−1 B1 (ˆ1 × ˆ3 ) = 0 The quantity ˆ1×ˆ3 is called a vanishing point, and it is the point through which all images of world lines of direction Ld will pass. The constraint says that if we have a vanishing point in one image and a line in another image which we know is parallel to the lines in the first camera, then we have a constraint on the Bi . Note that for the prismatic line constraint to constrain, all the i must be linearly independent, which means that the cameras may not all lie in one plane perpendicular to Ld . Thus for linear projection equations, the prismatic line constraint has no meaning for just one camera.

6

Bilinear + Line Trilinear = Point Trilinear

While these derivations are geometrically correct, let us step back for a moment and look at the measurements of the images which we are able to obtain. From this, we discover that instead of considering the epipolar, trilinear, quadrilinear, and prismatic constraints, it is sufficient to consider the epipolar, prismatic, and a modified 2D trilinear constraint which only considers translation. As we have formulated it, the line trilinear constraint operates on three image lines i in three cameras. If we choose any image point p which is incident on 2 (pT 2 = 0), the equations hold. Note that although we may choose any incident point, we can only obtain two linearly independent equations, since the equation (9) is linear in all the lines and the space of all points incident on 2 is of rank 2. Further, if we choose two points which are linearly independent, then this accounts for all the constraints possible on these three lines. The point trilinear constraint operates on three corresponding image points pi . The equation will still hold if we choose any two lines 1 and 3 which are incident on p1 and p3 , respectively, so that pTi i = 0, for i ∈ {1, 3}. We see here that we can create at most four linearly independent equations, since we may choose two lines for each of two points. Again, if we choose two sets of two linearly independent lines, then these four equations account for all possible constraints on three points.

380

Patrick Baker

Fig. 4. With three world points, an equivalence of points and lines is obtained

The corresponding point formulations of the trilinear constraint is equivalent to the corresponding line formulation plus the epipolar constraint. The line correspondence formulation has the advantage of being able to be split into the prismatic line constraint plus a 2D trilinear constraint. Therefore it is desirable to use the line trilinear constraint plus the epipolar constraint rather than using the point trilinear constraint. Let us show that where there are three image points, the point trilinear constraint is equivalent to the line trilinear constraint plus the epipolar constraint. See figure 4 for a diagram. Theorem 13. Given three world points P i projected into three cameras j, with ˆ j,i . The constraints on the positions of the cameras usparameters (Bj , T j ), at p ing the point trilinear constraint are equivalent to the constraints on the cameras using the line trilinear constraint plus the epipolar constraint. Proof. We work with the calibrated coordinates pj,i . Note that we may define image lines: j,1 = pj,2 ×pj,3 j,2 = pj,3 ×pj,1 j,3 = pj,1 ×pj,2


381

A consequence of this is that we also have j,2 ×j,3 |pj,1 pj,2 pj,3 | j,3 ×j,1 = |pj,1 pj,2 pj,3 | j,1 ×j,2 = |pj,1 pj,2 pj,3 |

pj,1 = pj,2 pj,3

We assume without loss of generality that T 2 = 0 We may form our twelve trilinear equations by writing four for each point. For point 1 we get (by choosing lines 2 and 3 through point 1): T T1 1,2 T3,2 (2,2 ×2,3 ) − T T3 3,2 T1,2 (2,2 ×2,3 ) = 0

(7)

T 1 1,3 3,2 (2,2 ×2,3 ) − T 3 3,2 1,3 (2,2 ×2,3 ) = 0

(8)

T 1 1,2 3,3 (2,2 ×2,3 ) − T 3 3,3 1,2 (2,2 ×2,3 ) = 0

(9)

T 1 1,3 3,3 (2,2 ×2,3 ) − T 3 3,3 1,3 (2,2 ×2,3 ) = 0

(10)

T T T

T T T

T T T

T T T

for point 2 (choosing lines 3 and 1), we get similarly T T1 1,3 T3,3 (2,3 ×2,1 ) − T T3 3,3 T1,3 (2,3 ×2,1 ) = 0

(11)

T 1 1,1 3,3 (2,3 ×2,1 ) − T 3 3,3 1,1 (2,3 ×2,1 ) = 0

(12)

T 1 1,3 3,1 (2,3 ×2,1 ) − T 3 3,1 1,3 (2,3 ×2,1 ) = 0

(13)

T 1 1,1 3,1 (2,3 ×2,1 ) − T 3 3,1 1,1 (2,3 ×2,1 ) = 0

(14)

T T T

T T T

T T T

T T T

for point 3 (choosing lines 1 and 2), we get T T1 1,1 T3,1 (2,1 ×2,2 ) − T T3 3,1 T1,1 (2,1 ×2,2 ) = 0

(15)

T 1 1,2 3,1 (2,1 ×2,2 ) − T 3 3,1 1,2 (2,1 ×2,2 ) = 0

(16)

T 1 1,1 3,2 (2,1 ×2,2 ) − T 3 3,2 1,1 (2,1 ×2,2 ) = 0

(17)

T 1 1,2 3,2 (2,1 ×2,2 ) − T 3 3,2 1,2 (2,1 ×2,2 ) = 0

(18)

T T T

T T T

T T T

T T T

Let us look closely at equations (14) and (15). We see that (2,3 × 2,1 ) and (2,1 × 2,2 ) are just points on line 1 in camera 2. So these two equations are equivalent to the line trilinear constraint using the j,1 . Similarly, equations (7) and (18) can be derived using j,2 and equations (10) and (11) can be derived using j,3 . The remaining equations are equivalent to the epipolar constraints between pairs of points, in the following manner. We may equate the T 3 terms in equations (8) and (17) to obtain (switching some triple product orders also): T T1 1,3 T2,3 (3,2 ×2,2 ) T T1 1,1 T2,1 (2,2 ×3,2 ) = T1,3 p2,1 T1,1 p2,3

382

Patrick Baker

similarly, using equations (9) and (12): T T1 1,1 T2,1 (3,3 ×2,3 ) T T1 1,2 T2,2 (2,3 ×3,3 ) = T 1,2 p2,1 T1,1 p2,2 finally, using equations (13) and (16): T T1 1,2 T2,2 (3,1 ×2,1 ) T T1 1,3 T2,3 (2,1 ×3,1 ) = T1,3 p2,2 T1,2 p2,3 We now note that since 3,i × 2,i gives the direction of line i, and so does 1,i ×2,i , we know that these vectors have the same direction, but have different magnitudes. We may therefore substitute and divide out the magnitudes. We want equations in only cameras 1 and 2, so we thus may derive the following from the above equations: T T1 1,1 T2,1 (2,2 ×1,2 ) T T1 1,3 T2,3 (1,2 ×2,2 ) = T1,3 p2,1 T1,1 p2,3

(19)

T T1 1,2 T2,2 (2,3 ×1,3 ) T T1 1,1 T2,1 (1,3 ×2,3 ) = T 1,2 p2,1 T3,3 p2,2

(20)

T T1 1,2 T2,2 (1,1 ×2,1 ) T T1 1,3 T2,3 (2,1 ×1,1 ) = T1,3 p2,2 T1,2 p2,3

(21)

By substituting points for cross products of lines, we may change equations (19) and (20) to: T T1 1,3 T1,2 p2,1 T T1 1,1 T1,2 p2,3 = T1,3 p2,1 T1,1 p2,3 T T1 1,1 T1,3 p2,2 T T1 1,2 T1,3 p2,1 = T 1,2 p2,1 T3,3 p2,2 By multiplying through the left denominators, subtracting the above equations, and using cross product identities we may obtain: (T 1 ×(1,3 ×1,2 ))T p2,1 = =

T T1 1,1 T1,2 p2,3 T1,3 p2,1 T T1 1,1 T1,3 p2,2 T1,2 p2,1 − (22) T 1,1 p2,3 T3,3 p2,2 T T1 1,1 (T p T p T p T1,1 p2,3 T3,3 p2,2 1,2 2,3 1,3 2,1 3,3 2,2

− T1,3 p2,2 T1,2 p2,1 T1,1 p2,3 ) Now let us do a simple derivation. First, we know that we may write 1,1 = λ1 ((P 2 − T 1 )×(P 3 − T 1 )) = λ1 (P 2 ×P 3 + T 1 ×P 2 + P 3 ×T 1 )

(23)


383

similarly, we get 1,2 = λ2 (P 3 ×P 1 + T 1 ×P 3 + P 1 ×T 1 ) and 1,3 = λ3 (P 1 ×P 2 + T 1 ×P 1 + P 2 ×T 1 ) where the λi are scale factors. Since T 2 = 0, we may also write p2,i = γi P i Using these substitutions, we see that the parenthesized term in the RHS of equation (23) is: λ1 λ2 λ3 γ1 γ2 γ3 (|P 1 T 1 P 3 ||P 2 T 1 P 1 ||P 3 T 1 P 2 | − |T 1 P 1 P 2 ||T 1 P 3 P 1 ||T 1 P 2 P 3 |) and this is zero. So that we have derived that (T 1 ×(1,3 ×1,2 ))T p2,1 = 0 which is just the epipolar constraint for point 1 with cameras 1 and 2. If we subtract other pairs of equations (19) through (21), we may obtain the epipolar constraints for points 2 and 3. We find the epipolar equations between cameras 2 and 3 by equating the T 1 terms instead of the T 3 terms.

7

Prismatic + Depth Consistency = Line Trilinear

We now split the line trilinear constraint into the prismatic line constraint and a new 2D trilinear constraint. Theorem 14. If we have three cameras with parameters (Bi , T i ), and a world line which projects to î , then the prismatic line constraint holds. There is only one other independent constraint, and it is: T Ti1 i1 |i2 ×i3 | 0= [i1 ..i3 ]∈P+ [1..3]

where | · | is the signed magnitude. We call this the depth consistency constraint. Proof. Recall the line trilinear constraint from theorem 11: T T1 1 T3 p2 − T T3 3 T1 p2 − (T 2 ×p2 )T (1 ×3 ) = 0 Let us introduce the notation Q = 1 × 3 . We know that the prismatic line constraint holds, that is |1 2 3 | = 0. This is the same as QT 2 = 0. Since Q

384

Patrick Baker

and 2 are perpendicular, we know that we may choose p2 = Q×2 . Using this definition, we may derive from equation (11) that T T1 1 |3 Q 2 | − T T3 3 |1 Q 2 | − (T 2 ×(Q×2 ))T Q = 0 T T1 1 |3 Q 2 | − T T3 3 |1 Q 2 | − (QT T2 2 )T Q + (2 T T2 Q)T Q = 0 but since T2 Q = 0, the last term is zero and we can derive T T1 1 |2 3 Q| + T T2 2 |3 1 Q| + T T3 3 |1 3 Q| Since the prismatic line constraint holds, it is clear that Q is arbitrary, as long as it does not lie in the same plane as the i . We therefore remove it and replace it with the signed magnitude | · | to obtain our result. It has already been proved in [3] that the quadrilinear constraint is redundant with the trilinear constraints and the epipolar constraint. What we are left with are three constraints: 1. the epipolar constraint 2. the prismatic line constraint 3. the depth consistency constraint The constraints all have different properties which can be used for different portions of structure from motion. The first constraint ensures that two points correspond. The second constraint is only on rotation and ensures that lines are properly aligned. The third constraint ensures that depths are consistent when calculated from different pairs of cameras.

8

Discussion

We have shown that all our geometric constraints can be put in terms of the three geometric constraints shown above. These constraints are symmetric over all the cameras, an important point in multiple view geometry. The prismatic line constraint is independent of translation and correspondence (for sets of parallel lines). We started this paper stating that lines could provide the basis for all our geometry, yet we include the epipolar constraint. This is not a contradiction because the epipolar constraint can be understood as constraining two lines in each of two images, which are constrained to lie in a single plane. This is an important distinction, because there may be no such actual point of intersection. If we do indeed extract points as our primitive object, it is a simple matter to create lines from images of points using a cross product. If we have at least three corresponding lines, then there is no difference between using points and lines with three images.


385

References [1] J. Aloimonos. Shape from texture. Biological Cybernetics, 58:345–360, 1988. 378 [2] O. Bottema and B. Roth. Theoretical Kinematics. Dover, 1990. 372 [3] O. D. Faugeras and B. Mourrain. On the geometry and algebra of the point and line correspondences between n images. In Proc. 5th International Conference on Computer Vision, pages 951–956, Cambridge, MA, 1995. 384 [4] A. Heyden. A common framework for multiple view tensors. In ECCV98, pages 3–19, 1998. 376 [5] Y. Liu and T. S. Huang. Estimation of rigid body motion using straight line correspondences. Computer Vision, Graphics, and Image Processing, 43:37–52, 1988. 369 [6] H. C. Longuet-Higgins. A computer algorithm for reconstructing a scene from two projections. Nature, 293:133–135, 1981. 377 [7] A. Shashua. Algebraic functions for recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(8):779–789, 1995. 376 [8] M. E. Spetsakis and J. Aloimonos. Structure from motion using line correspondences. International Journal of Computer Vision, 4:171–183, 1990. 376 [9] M. E. Spetsakis and J. Aloimonos. A multi-frame approach to visual motion perception. International Journal of Computer Vision, 6:245–255, 1991. 371 [10] A. P. Witkin. Recovering surface shape and orientation from texture. Artificial Intelligence, 17:17–45, 1981. 378

Topological Analysis and Characterization of Discrete Scalar Fields Emanuele Danovaro, Leila De Floriani, and Mohammed Mostefa Mesmoudi Department of Computer Science and Information Science (DISI) University of Genova, Via Dodecaneso, 35 -16146 Genova (Italy) {danovaro,deflo,mesmoudi}@disi.unige.it

Abstract. In this paper, we address the problem of analyzing the topology of discrete scalar fields defined on triangulated domains. To this aim, we introduce the notions of discrete gradient vector field and of Smalelike decomposition for the domain of a d-dimensional scalar field. We use such notions to extract the most relevant features representing the topology of the field. We describe a decomposition algorithm, which is independent of the dimension of the scalar field, and, based on it, methods for extracting the critical net of a scalar field. A complete classification of the critical points of a 2-dimensional field that corresponds to a piecewise differentiable field is also presented.

1

Introduction

Morse theory [15] has relevant applications in geometric modeling, since it provides a powerful tool to study the geometry and the topology of a C 2 -differentiable scalar field defined on a manifold [18, 19]. Usually, the values of a scalar field given at the vertices of a mesh are interpolated by a C 2 -differentiable function. Morse theory can then be used to extract the most relevant features representing the geometry of the triangulated domain and associated with the scalar field. (see, for instance, [2, 3, 5, 7, 20]). Such features must be taken into account for visualizing or simplifying data describing a scalar field [2, 3]. Morse theory can also be used to give a CW -representation of the domain (see [11]). This result is due originally to Thom [19] and Smale [18]. Smale has shown that trajectories integrating the gradient field of a Morse function can describe submanifolds (cells) decomposing the manifold domain of the function. Some of such trajectories, called separatrices, connect critical points having an increasing index order. Therefore, a critical net can be built to represent the topology of the scalar field (see [12, 16]). Each node of the net represents a critical point, while each edge corresponds to a separatrix connecting two nodes. This representation can help, for instance, in understanding the topology of the level sets (isolines, isosurfaces) of the field which propagate along the net [10]. Other authors (see, for instance, [5]) apply Morse theory to represent the topology of the scalar field through a Reeb graph, or through some of its generalizations. More recently, some authors [2, 3, 7, 9, 21] have investigated scalar fields using piecewise linear functions interpolating the discrete values of the field. T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 386–402, 2003. c Springer-Verlag Berlin Heidelberg 2003

Topological Analysis and Characterization of Discrete Scalar Fields

387

In [7], a notion of quasi-Morse complexes has been introduced for piecewise linear functions to decompose compact 2-manifolds without boundary. A hierarchical representation of the domain is built using a notion of persistency for critical points. Other authors have been interested in handling discrete data directly. In [13, 17], important features of a scalar field representing 2-dimensional gray-scale images have been extracted by using a local study of the values of the field around each point. The classification of critical points provided compares with the classical classification known for the C 2 -differentiable case and does not extend to higher dimensions. Recently, Forman [8] introduced a discrete equivalent of Morse functions for cell complexes, and has proven similar results as in the differentiable Morse theory. Forman’s theory is very interesting, it allows handling discrete data on a generic simplicial complex (i.e, not only manifolds), and has the same properties as in the differentiable case. Forman functions are defined on all cells of the complex, but scalar fields are only known at the vertices (i.e., 0-cells) of a simplicial complex. A scalar field can be easily extended to a non-unique Forman function defined on all cells, but, in this case, the simplification process proposed by Forman stops after few steps and all remaining cells in the complex become critical. This fact prevented us from applying Forman theory in a multiresolution context. In this paper, we propose to modify the notion of discrete gradient vector field, introduced by Forman to handle discrete scalar fields. This leads to a constructive definition of a Smale-like decomposition for scalar fields defined on simplicial complexes in arbitrary dimension, in which each component is characterized by an extremum (which is a local maximum or a local a minimum). Such decomposition generalizes (to high dimensions) the watershed transform used for morphological image segmentation [14]. Comparing our discrete gradient vector field to the differentiable case, we can define other critical points associated with the discrete scalar field. When the domain is a triangulated manifold, these critical points are located on the boundary of the components in the decomposition and are defined inductively using successive dimensionally-increasing decompositions of the boundaries of the components. A local study of the behavior of the discrete gradient vector field around boundary points allow us to identify critical points and classify them. We present here an analysis and a classification of the critical points the 2-dimensional discrete case that compares to the differentiable case and that generalizes it in a certain sense. This classification is a necessary step for a further classification of critical points in higher dimensions. The notion of a critical net, i.e. of a net connecting critical points, come naturally from our decomposition. A critical net can be constructed from this decomposition to represent the topological structure of the scalar field. Here, we describe an approach for extracting a critical net in any dimension. The remaining of this paper is organized as follows. In Section 2, we briefly review the basic notions from Morse theory and combinatorics needed for the

388

Emanuele Danovaro et al.

remaining material. In Section 3, we introduce the notion of discrete scalar field associated with a Smale-like decomposition of its domain, and we describe an algorithm for constructing such decomposition. In Section 4, we give an inductive definition of critical points in any dimension and we apply our decomposition algorithm inductively to extract the critical net that contains critical points and that represents the topological structure of the scalar field. In Section 5, we present a classification of critical points in the two-dimensional discrete case. Finally, we present some concluding remarks.

2

Background Notions

In this Section, we summarize some results from Morse theory necessary to understand the topological structure of the domain associated with a scalar field as well as some combinatorial notions we use throughout the paper. A Morse function on Rd is a differentiable (at least C 2 ) real-valued function f defined on Rd such that its critical points are isolated [15]. This means that the Hessian matrix HessP f of the second derivatives of f at any point P ∈ Rd , on which the gradient of f vanishes (GradP f = 0), is non-degenerate. Since these properties are local, a Morse function can be defined on a manifold M (that is a topological space where the neighborhood of any point is homeomorphic to Rn or R+ × Rn−1 ). Morse [15] has proven that there exists a local coordinate system (y 1 , ..., y n ) in a neighborhood U of any critical point P , with y j (P ) = 0, for all j = 1, . . . , n, such that the identity f = f (P ) − (y 1 )2 − ... − (y ı )2 + (y ı+1 )2 + ... + (y n )2 holds on U , where ı is the number of negative eigenvalues of HessP f , and is called the index of f at P . The above formula implies that the critical points of a Morse function are isolated. This allows studying the behavior of f around them, and classifying their nature according to the signs of the eigenvalues of the Hessian matrix of f . If the eigenvalues are all positives, then the point P is a strict local minimum (a pit). If the eigenvalues are all negatives, then P is a strict local maximum (a peak). If the index ı of f at point P is different from 0 and n, then the point P is neither a minimum nor a maximum, and, thus, it is called an ı-saddle point (a pass). The decomposition of the manifold domain, on which function f is defined, introduced by Thom [19] and followed by Smale [18], is based on the study of the growth of f along its integral curves. An integral curve is a curve that is everywhere tangent to the gradient vector field Gradf . The classical Taylor formula shows that integral curves follow the (gradient) directions in which the function has the maximum increasing growth. Hence, integral curves cannot be closed, nor infinite (in a compact manifold), and they do not self-intersect. They are emanating from critical points, or from boundary components of the domain, and reach other critical points, or boundary components. The integral curves originating from a critical point of index ı form a ı-cell C s , called a stable


389

Maximum

Saddle

Minimum

(a)

(b)

Fig. 1. (In (a), decomposition of a domain into four stable 2-manifolds. In (b), the critical net corresponding to decomposition in (a)

manifold. In the same way, integral curves converging to a critical point of index ı form a dual (n−ı)-cell C u , called an unstable manifold. Because of the properties of integral curves, the stable manifolds are pairwise disjoint and decompose the field domain M into open cells (see Figure 1(a)), which form a complex, since the boundary of every stable manifold is the union of lower dimensional cells. Similarly, the unstable manifolds decompose M into a complex dual to the complex of stable manifolds. Figure 1(a) gives an example of a stable decomposition of a two-dimensional domain. The function is assumed to be a Morse function. It has three minima (shown by •), two maxima (shown by ), and five saddle points (shown by ). Integral curves originate from each minimum in all directions and from the right side of the boundary. Each integral curve converges either to a saddle, to a maximum, or to a boundary component. Two integral curves originate from each saddle point. Integral curves originating from a minimum (or from the right side boundary) sweep a 2D cell, while integral curves emanating from a saddle point form a segment containing the saddle point in its interior. Integral curves connecting saddles to other critical points are called separatrices. The graph formed by all separatrices with their endpoints is called a critical net [12]. Figure 1(b) illustrates the critical net corresponding to the stable decomposition in Figure 1(a). Let k be a non-negative integer. A k-simplex (or a k-dimensional simplex) is the convex hull of (k + 1) affinely independent points in Rd (with k ≤ d) called vertices. A face σ of a k-simplex γ, σ ⊆ γ, is a j-simplex (0 ≤ j ≤ k) generated by (j + 1) vertices of γ. Conversely, γ is said to be a coface of σ. A simplicial complex K is a collection of simplexes such that, if γ is a simplex in K, then each face σ ⊆ γ is in K, and the intersection of two simplexes is either empty or a common face. The dimension of a simplicial complex K is the maximum of the dimensions of its simplexes. A maximal simplex in K is any simplex which has the same dimension as K. A top simplex in K is a simplex which is not a proper face of any other simplex in K. When all top simplexes have the same dimension as K, we say that complex K is regular. The carrier

390


| K | of a simplicial complex K is the space of all points belonging to simplexes of K. In this case, K is called a triangulation of | K |. Let K be a simplicial complex and Γ be a set of simplexes in K. The closure of Γ is the smallest sub-complex Γ of K containing Γ . Clearly, Γ consists of all simplexes of Γ plus all their faces. Let p a 0-simplex (i.e. a vertex) of K. The open star of p, denoted St◦ (p), is the set of all cofaces of p. Thus, St◦ (p) = {σ ∈ K : p ⊆ σ}. We call star of p, that we denote as St(p) the closure of St◦ (p) in K. The star of p describes the closed neighborhood of p in the complex (see Figure 2(a)). The link of p is the sub-complex Lk(p) of K defined by Lk(p) = St(p)−St◦ (p). The link describes the boundary of St(p) (see Figure 2(a)). A cone from a vertex w to a simplex γ is the convex combination of all vertices of γ with w. We denote it by (γ, w). If w is affinely independent of the vertices of γ, then the cone from w to γ is a simplex of dimension dim(γ) + 1, where dim(γ) denotes the dimension of γ. A regular k-complex is said to be a combinatorial manifold if and only if the link of each vertex is equivalent either to a (k − 1)sphere or to (k − 1)-ball. A pseudo-manifold [1] is a simplicial complex K of dimension k such that 1. Every simplex in K is a face of a k-simplex of K and every (k − 1)-simplex is a face of at most two k-simplexes. 2. If σ and σ are two distinct k-simplexes in K, then there exists a sequence of simplexes (σi )ni=1 in K such that σ = σ1 , σ = σn , and, for all 1 ≤ i < n, σi and σi+1 are incident at a (k − 1)-face. The (k − 1)-simplexes satisfying condition (1) are called manifold cells. In Figure 2(b), we give an example of a pseudo-manifold which is not a manifold. Pseudo-manifolds are interesting especially in dimensions greater than three, since these spaces support algorithms of decomposition and recognition [6].

z w v

(a)

(b)

Fig. 2. In (a), shaded region is the star of v, the graph in bold is the link of the vertex w. In (b), an example of a pseudo-manifold that is not a manifold: the link of z is formed by to disjoint segments (in bold) and is not homeomorphic to circle nor to a segment


3

391

Discrete Gradient Vector Field and Smale-Like Decomposition

Let M be a triangulated domain of dimension d and f be a scalar field defined at the vertices of the triangulation of M . In this Section, we introduce the notion of discrete gradient vector field (Gradf ) associated with f . The discrete gradient vector indicates the directions in which the scalar field is increasing. It starts from local minima and ends at other critical points. As in the differentiable case, the discrete gradient vector field characterizes intrinsic components, called stable and unstable components, each of which corresponds to a local extremum. Stable (resp. unstable) components subdivide the domain M into an assembly of complexes. We call such a decomposition a Smale-like decomposition of the domain M associated with the scalar field f . In differentiable case, the negative gradient vector field, (−Gradf ), is usually considered instead of the gradient field itself. In the remainder of this paper, we will stick to this convention. Our discrete (negative) gradient vector field, thus, indicates the main directions in which the field f is decreasing. Let K be the simplicial complex associated with the given triangulation of M . We assume that f (u)

= f (v) for all vertices u

= v in K. This can be obtained through small perturbations of the scalar field f . This condition ensures the uniqueness of the decomposition when M is a manifold or a pseudo-manifold. We note here that the same condition is considered in [7] to build quasi-Morse complexes corresponding to interpolation by piecewise linear functions. We can also replace this condition by requiring that f (u)

= f (v) only for all vertices u

= v of the same simplex, but in this case the uniqueness of the decomposition is not guaranteed. To perform the decomposition we keep a current complex K which is initialized to be equal to K. We consider a vertex v in K corresponding to the global maximum of the scalar field f . The values of f at the vertices of St(v) are, thus, less than f (v). We define the component C(v), corresponding to v, to be the same as the star St(v) of v, and we initialize the boundary ∂C(v) of C(v) to be equal to the link Lk(v) of v. Then, for each top simplex γ in ∂C(v) that is incident to another simplex (γ, w) in K − C(v), we compare the values of f at vertices of γ with f (w). If f (w) is less than all of them, we extend C(v) to be C(v) ∪ (γ, w), where (γ, w) is the cone defined by γ and w, and we replace γ in ∂C(v) with those faces of cone (γ, w) that contain w. We continue extending C(v) in such a way that C(v) is a region on which f decreases from v. This process stops when C(v) cannot be extended any further while maintaining the above property. Then C(v) is deleted from K , and we repeat the process. In the case in which v belongs to the boundary of another component C(w), component C(v) is merged into C(w). The result is, thus, a decomposition D of K into components that we call unstable Smale complexes Ci ≡ C(vi ), each of which corresponds to a maximum of the scalar field f . In Figure 3(a), we illustrate the construction of the first component of a two-dimensional scalar field, while all the components before

392


merging are depicted in Figure 3(b). The resulting decomposition, after merging, is shown in Figure 3(c). The decomposition algorithm described above allows us to define a discrete form of the gradient vector field for a scalar field f . A discrete (negative) gradient vector field is defined by two functions: (i) a multi-valued function φ associates each local maximum v, which corresponds to a component C(v) of K, with the top cells γ in St(v), i.e., φ(v) = {γ : γ is a top cell in St(v)}; (ii) a multi-valued function ψ associates with each cell γ in C(v) − {∂C(v) ∪ St(v)}, that has been used in the extension process, the cones (γ, wi ) added to γ. Equivalently, vertices wi are sufficient to characterize function ψ. Functions φ and ψ define the discrete (negative) gradient vector field of f . As in the differentiable case, the gradient field denotes the directions in which the function decreases, and characterizes the critical cells, i.e., those cells on which ψ cannot be extended. To illustrate functions φ and ψ, we draw vectors, from the initial vertex v, to all top cells in St(v), and a vector from each cell γ to the cones (γ, wi ) used in the decomposition process as shown in Figure 3. Vectors are originating from a local maximum in all possible directions. A local minimum receive vectors from all possible directions, while a saddle point has vectors originating and entering in alternating groups of directions. When there are only two groups, the point will be called a slope point. We call such points critical points. We will see below that slope points can be considered non-singular since the gradient vector field behaves around a regular point inside a component in the same manner as around a slope point. In our implementation the discrete (negative) gradient vector field is a function g associated with top simplices. It is stored as a function diverging from a vertex (to encode φ) or converging to a vertex (this simulate ψ). Figure 4 shows the results produced by the decomposition algorithm on a terrain data set representing Mount Marcy (courtesy of USGS), which contains 69718 triangles. The algorithm produces only 119 components.

5

3 6

8

8

7 1

2

1

3

3

1 6

(a)

8

7 1

2

1

1 7

5

6

7

1 3

6

4 5

6

7

7 2

1

5

3 4

5

7

5

3

4

1 6

7

(b)

7

(c)

Fig. 3. In (a), the decomposition process: component corresponding to the point with f = 8 with its discrete gradient vector field. In (b), the components decomposing K and their discrete gradient vector field. In (c), the final decomposition after merging is shown


393

Fig. 4. 119 unstable components produced by our decomposition algorithm applied to Mount Marcy with 69718 triangles

Our implementation of the decomposition algorithm is based on an indexed data structure that stores the vertices, the d-simplexes and their adjacency relations. For each vertex v, we store its coordinates, the field value and a simplex in the star of v. For each simplex, we store its vertices, the simplexes adjacent to it along (d−1) simplexes, and the discrete gradient. As an auxiliary data structure, we use in a priority queue, implemented as a heap, encoding the vertices according to their field values. The decomposition algorithm visits each (d − 1)-simplex exactly twice and this is linear in the number of (d − 1)-simplexes. In the 2D case, where the number of edges is in O(n), the algorithm has a O(nlogn) worstcase time complexity, which is dominated by the cost of handling the priority queue In the d-dimensional case, the worst-case time complexity is linear in the number of d-simplexes and, for instance, it is O(n2 ) in the 3D case. In practical applications, the domain M of a scalar field is a triangulated manifold. In this case, the following results hold: (i) when K is a pseudo-manifold or a combinatorial manifold, a Smale-like decomposition is independent of the order in which vertices are selected (ii) when the domain K is a combinatorial manifold, then each Smale complex Ci in the decomposition of K is a pseudo-manifold Figure 5 shows a component of a decomposition which is a pseudo-manifold, but not a combinatorial manifold.

4

Critical Points and Critical Nets

To study the nature of critical points of a discrete scalar field and classify them, we need to perform a local study of the behavior of the gradient vector field on the boundary of the components in the decomposition. A point p is a critical point for a discrete scalar field f defined on the vertices of a d-dimensional combinatorial manifold K (with d ≥ 2) when one of the following properties holds:

394


(i) p a local maximum: f (p) is larger than all values of f at the vertices of its star St(p); (ii) p a local minimum: f (p) is smaller than all values of f at the vertices of St(p); (iii) p belongs to the boundary of a component (stable or unstable) C and is critical for the restriction of f to the boundary ∂C of such component. This inductive definition is coherent with the differentiable case and gives us an effective way to recognize and classify critical points in two and higher dimensions. A representation of the topological structure of a scalar field is provided by the critical net. In the differentiable case, a Smale decomposition gives a CW representation of the domain M associated with a Morse function f defined on it, (see [11]). This cellular representation can be understood by studying the integral lines that define the structure of each cell. Smale [18] has shown that among all integral lines incident in a k-saddle p (where 0 ≤ k ≤ d = dim(M )), there exists a finite number of such lines that connect p to some neighboring (k + 1)-saddles. Thus, a d-cell C(m) corresponding to a minimum m, in the stable decomposition, can be described with all its faces just by integral lines. Vertex m is in the interior of C(m) and attached directly, through some integral lines, to some 1-saddle lying on the (d − 1)-boundary ∂C(m) of cell C(m). Now, by restricting the study just to the boundary ∂C(m) of C(m), 1-saddles play the role of local minima in ∂C(m) and each of them generates a (d − 1)-cell, denoted as C d−1 ∈ ∂C(m). By applying the same argument again as for m, a 1-saddle is connected directly, through some integral lines, to some neighboring 2-saddles lying on the boundary ∂C d−1 of the C d−1 . Iteratively, this provides all integral lines connecting k-saddles (2 ≤ k ≤ d) to some neighboring (k + 1)-saddles, each of which lies on the boundary of a (d − k)-cell. The set of all these integral lines describe d-cell C(m) completely with all its faces. In Figure 6(a), we illustrate such representation for a minimum of a 3D scalar field.

−2

0

−3 −3

0

−2 8 4

5

−3 2

7

3

6 7 6

10 3

5

8 6

Fig. 5. The shaded region is the component Ci associated to the maximum 10. It is a pseudo-manifold, but not a combinatorial manifold


395

Minimum 1−saddles 2−Saddles Maximum

(a)

(b)

Fig. 6. In (a), the description a 3-cell corresponding to a minimum by its corresponding integral lines. In (b), the critical net of N acl molecule corresponding to electron location density function

The set of all such integral lines is sufficient to represent faithfully the topological structure of the scalar field f , and forms the critical net of f . In Figure 6(b), the critical net associated with the N acl molecule of an electron position density function is shown [12] . The corresponding critical net has a cubic shape. There is only one minimum located at the center of the cube, and eight maxima located on the corners of the cube. 1-saddles, represented in Figure 6(b) with small discs, are located in the center of each face and directly connected to the minimum and to other 2-saddles located on the cube edges and represented in Figure 6(b) with small thick segments. This model has been considered in [10] to study the propagation of isosurfaces from maxima. A critical net can be iteratively constructed by considering a sequence of cells of decreasing dimension, each of which lies on the boundary of a previous cell. This fact allows us to built, from a Smale-like decomposition, a discrete equivalent critical net in any dimension. It also provides a practical way to extend the notions of ridges and valleys (ravines) to higher dimensions. The basic idea behind the extraction of critical nets is that the boundaries of stable and unstable components intersect transversally (i.e, the direct sum of their tangent spaces at an intersection point has the dimension of K [18]). In the case of two-dimensional scalar fields, the boundary lines of the unstable components give the separatrices connecting saddles to minima, while the boundary lines of the stable components give the separatrices connecting saddles to maxima. Thus, the critical net is just the union of the boundaries of the components of the stable and unstable decomposition of K, denoted as D+ and D− , respectively. Figures 7(a) and 8(a) describe the stable and unstable decomposition of two analytic functions, whose values have been sampled at the vertices of a regular grid. The corresponding critical nets are depicted in Figures7(b) and 8(b), respectively. In Figure 9(a), we present the stable and unstable decomposition

396


applied to the Mount Marcy data set (see Figure 4). A perspective view of the corresponding critical net is depicted in Figure 9(b). In the three-dimensional case, the boundaries of the components in the decompositions are surfaces. The scalar field has higher values in the normal directions at the points of the boundary surfaces obtained form the unstable decomposition. We call such surfaces 2-valleys (or 2-ravines). The scalar field has lower values in the normal directions at the points of such surfaces when obtained from the stable decomposition. We call such surfaces 2-ridges. The critical net of a 3D discrete scalar field can be computed by extracting 2D critical nets on each boundary surface coming from the stable and unstable decomposition. The union of all curves obtained by this process forms the critical net. We note here that local minima and maxima are reached by this process since they belong to the boundary components of unstable and stable decompositions, respectively. Let K denote the given complex, f the scalar field, and N the resulting critical net. We summarize this process in the following steps:

(a)

(b)

Fig. 7. In (a), Stable and unstable decompositions of a synthetic function f (x, y) = sinx + siny representing the eggs plateau surface. In (b) the corresponding critical net

(a)

(b)

Fig. 8. In (a), Stable and unstable decompositions of a synthetic function 2 2 f (x, y) = xye−(x +y ) . In (b) the corresponding critical net


397

1. Construct the stable decomposition D+ of K, with respect to f , and extract the union ∂D+ of the boundaries of the components in D+ 2. Construct the unstable decomposition D− of K with respect to f , and extract the union ∂D− of the boundaries of the components of D− 3. Extract the 2D critical net N + for f on ∂D+ 4. Extract the 2D critical net N − for f on ∂D− 5. N ← N + ∪ N − In a similar way, we can extract the critical net in higher dimensions. Suppose that for dimension d ≥ 3 we have an algorithm that extracts the critical net in dimension (d − 1). Let K be a d-dimensional combinatorial manifold endowed with a discrete scalar field f defined at its vertices. The Smale-like decomposition algorithm provides stable and unstable components. The boundary of each component is a hyper-surface(of dimension d − 1). As in 3D case, hyper-surfaces obtained from unstable decomposition can be associated with (d − 1)-valleys (or (d − 1)-ravines), while hyper-surfaces obtained from stable decomposition can be associated with (d − 1)-ridges. The (d − 1)-dimensional extraction algorithm of critical net applied to each hyper-surface Si provides a sub-net Ni of the global critical net N of K. The union i Ni of all these sub-nets gives the final net N .

5

Classification of Critical Points in the 2D Case

As we have seen in the previous section, critical points are defined by induction. Therefore, the nature of critical points in any dimension depends heavily on the nature of low-dimensional critical points and, in the particular, on the nature of critical points in the 2-dimensional case. Critical points for a 2D scalar field are well known in the differentiable case. They are classified into maxima (peaks), minima (pits) and saddles (passes). In Morse theory saddle points are simple (i.e, there are four separatrices at each saddle). When the function is not a Morse function other kind of saddles

(a)

(b)

Fig. 9. In (a), Stable and unstable decompositions of the image of Mount Marcy already represented in Figure 4. In (b), a perspective view of the corresponding critical net

398


appear, that are called generalized saddles (i.e., an even number larger than four of separatrices meet at the point). When the number of separatrices is equal to six, the point is called a Monkey saddle. We note that this kind of saddles are degenerate critical points: for the classification of such points we need to go beyond the Hessian in Taylor development. As an example, consider functions x4 + y 4 , x4 − y 4 and x4 + y 3 . They are all degenerate at the origin (i.e., their Hessian matrices are null at (0, 0)), but the nature of (0, 0) for each function is quite different. For x4 + y 4 , (0, 0) is a minimum, it is a saddle point for x4 − y 4 , and it is an inflection point for x4 + y 3 . We expect that the nature of the critical points in the discrete case will be similar to that discussed above. It is exactly what we obtain when we consider a piecewise differentiable function. Here, the degree of differentiability is not important since we are discussing a discrete case. Compared with the differentiable case, our approach is more flexible since it can recognize and classify critical points either inside the domain, or on the boundary. Moreover, in case of saddle points, it can recognize their nature: standard saddle, monkey saddle etc. The idea is to count the number of ridges and valleys incident into each critical point. This goes in the same direction as Banchoff classification [4]. Let K be a triangular mesh endowed with a scalar field f defined at its vertices. Let C(v) be a component of the unstable Smale-like decomposition associated with a local maximum v. Since vertex v has been selected to be a maximum in K, then v is a critical point and the gradient vector field is emanating from v towards all triangles in the star of v. Now, the local behavior of the scalar field around a point w on the boundary of component C(v) gives rises to several possible cases, which are discussed below. Local Minima. Point w is a minimum when f (w) < f (w ), for each w ∈ Lk(w). Thus, w is a minimum if the gradient vector field of all triangles in the star St(w) of w converges to w. If v ∈ Lk(w), the gradient of the triangles incident into w and into v is diverging from w, but, in this case, we still classify w as a local minimum (see Figure 10).

wL

w wR

C(v) w5

w

w4

1

w

2

w

w

5

w1 v=w2

w3

w3

(a)

wR

w wL

(b)

w4

C(v)

(c)

Fig. 10. Possible situations in (a) and (b) near a relative minimum w on the boundary of one component or of the mesh. In (c), the general situation around a relative minimum w


399

Saddles and Generalized Saddles. Let us consider the star St(w) of w and study its geometric shape. The idea is to group triangles in St(w) so that a group is a triangle fan with coherent gradient vector field: only converging to w or only diverging from it. We start from a triangle t in St(w), and we initialize a first group of triangles with t. Then, we iteratively check the coherence of the gradient on the current triangle (at the beginning initialized with t) and on its clockwise neighbor. We keep on adding triangles to the current group until we find a triangle having a gradient with opposite orientation with respect to its immediate predecessor in counter-clockwise order. In this case, we start a new group. The traversal of St(w) continues until we either go back to the initial triangle, t or we reach the boundary of the mesh. This latter situation occurs only when w is on the boundary of the mesh. In this case, we need to proceed from t in C counterclockwise order around w and continue the process. If v ∈ Lk(w), the gradient of the triangles incident into w and v is diverging from w, but, in this case, we still consider that gradient as converging to w. While traversing St(w), we count the number q of groups of triangles with coherent gradient orientation that we obtain. If w is inside the mesh (i.e., not on the boundary), the number of groups q is obviously even (since we have to count alternatively coherent and not coherent groups). If q is equal to 4, w is a saddle point and, if it is greater than 4, we say that w is a generalized saddle with q separatrices (see Figure 11). If w is on the mesh boundary, we have a generalized saddle when q is greater or equal to 3. Note here that, if the gradient field converges to w on a large fan of triangles, integral curves in the differentiable situation corresponding to this case may converge to w on a branch (i.e., a thick sector), that we call a bundle, and diverge from w in the neighboring branches (see Figure 11). This situation corresponds to a piecewise differentiable function, for which each differentiable part is defined on one of the previous branches. Point w can be considered here as a generalized saddle and, in this sense our classification generalizes the classical classification of critical points.

w

Fig. 11. A generalized saddle point: Integral curves converges to w in three thick bundles (the shaded regions). Here the number of coherent and not coherent groups is 6

400

Emanuele Danovaro et al. wL

w

w

wR w5 w

w4

1

w

2

C(v)

w3 (a)

(b)

Fig. 12. Slope situations: in (a) , point w is on the boundary of the mesh. There is two groups of triangle fans: the gradient field is directed toward w on triangles ∆1 = (wR ww1 ), ∆2 = (w1 ww2 ) and ∆3 = (w2 ww3 ), and directed from w in triangles ∆4 = (w3 ww4 ), ∆5 = (w4 w5 ) and ∆6 = (w5 ww6 ). In (b), point w is an internal point and we have a similar situation as in (a)

Slopes. We call a slope a point w satisfying the following property: the region around w can be subdivided into two regions such that the values of the scalar field in one region is greater than the value at w, and values of the scalar field in the second region are smaller than the value at w. To identify slope points we can apply the same technique used for saddle points: if the number of groups q is exactly equal to 2, w is a slope. Note that, if q = 1, w can be classified as a local minimum. Since the behavior of the gradient field inside each component of the decomposition is the same at a slope point, we can consider slope points as regular points for the scalar field. The equivalent differentiable case corresponding to these points is illustrated in Figure 13 with smooth integral lines. The extraction algorithm has been developed and implemented by considering an unstable decomposition of the mesh K and labels the vertices v generating the decomposition as local maxima. For each other point w, it computes the number q of regions with coherent gradient, and classify w as a saddle, a generalized saddle or a regular point, based on the value of q, as discussed above. The time complexity of the extraction algorithm is linear in the the number of triangles of the input mesh, and, thus, in the number n of its vertices.

w

Fig. 13. The differentiable equivalent case corresponding to a slope point


6

401

Concluding Remarks

We have introduced a discrete form of gradient vector field that allowed us to define and compute a so-called Smale-like decomposition of the domain of the field and to extract a critical net representing the topology of the field in any dimension. The notion of discrete critical points in any dimension has been introduced in an inductive way based on our decomposition. A complete classification of critical points has been performed for the 2D case. Such a classification generalizes the classical one, since it corresponds to approximating the scalar field with a piecewise differentiable function. There are several applications of our work that we are currently studying. Critical points and critical nets can be used to provide a user with a navigation tool in addition to an iso-surface viewer when visualizing and analyzing volume data sets. Critical points are also relevant in designing an automatic transfer function in volume rendering applications [21]. In our ongoing work, we are using critical nets to guide the simplification of meshes describing 2D and 3D scalar fields. In this way, we can generate a discrete description of the field based on a mesh of smaller size which has a critical net with the same connectivity structure as the critical net of the original mesh. This is the basis for constructing accurate multi-resolution models of scalar fields in both 2D and 3D case (terrain modeling, medical images, volume data analysis).

Acknowledgments The authors wish to thank Davide Sobrero for his useful comments on an earlier version of this paper. This work has been performed while Leila De Floriani has been visiting the Department of Computer Science of the University of Maryland, College Park, MD (USA). This work has been partially supported by the European Research Training Network MINGLE under reference HPRN-CT-199900117 and by the project funded by the Italian Ministry of Education (MIUR) on Representation and Processing of Spatial Data in Geographic Information Systems.

References [1] M.K. Agoston. Algebraic Topology, A First Course,. Pure and Applied Mathematics, Marcel Dekker, 1976. 390 [2] C. L. Bajaj, V. Pascucci, and D. R. Shikore. Visualization of scalar topology for structural enhacement. In Proceedings of the IEEE Conference on Visualization’98 1998, pages 51–58, 1998. 386 [3] C. L. Bajaj and D. R. Shikore. Topology preserving data simplification with error bounds. Journal on Computers and Graphics, 22(1):3–12, 1998. 386 [4] T. F. Banchoff. Critical Points and Curvature for Embedded Polyhedral Surfaces. Amer. Math. Monthly, 77(1):475–485, 1977. 398

402


[5] S. Biasotti, B. Falcidieno, and M. Spagnuolo. Extended reeb graphs for surface understanding and description. In Proc. 9th DGCI’2000, Upsala, LNCS 1953, Springer-Verlag, pages 185–197, 2000. 386 [6] L. DeFloriani, M. M. Mesmoudi, F. Morando, and Enrico Puppo. Non-manifold decomposition in arbitrary dimension. In Proc. DGCI’2002, LNCS 2301, pages 96–80, 2002. 390 [7] H. Edelsbrunner, J. Harer, and A. Zomorodian. Hierarchical Morse complexes for piecewise linear 2-manifolds. In Proc 17th Sympos. Comput. Geom., pages 70–79, 2001. 386, 387, 391 [8] R. Forman. Morse theory for cell complexes. Advances in Mathematics, 134:90– 145, 1998. 387 [9] T. Gerstner and R. Pajarola. Topology preserving and controlled topology simplifying multiresolution isosurface extraction. In Proceedings IEEE Visualization 2000, pages 259–266. IEEE Computer Society, 2000. 386 [10] J.C Hart. Morse theory for implicit surface modeling. In H.C. Hege and K. Poltihier (Eds), Mathematical Visualization, Springer-Verlag, pages 256–268, 1998. 386, 395 [11] J.C Hart. Using the CW-complex to represent topological structure of implicit surfaces and solids. In Proc. Implicit Surfaces 1999, Eurographics/SIGGRAPH, pages 107–112, 1999. 386, 394 [12] C Johnson, M. Burnett, and W. Dumbar. Crystallographic topology and its applications. In Crystallographic Computing 7: Macromolecular Crystallography data, P. E. Bourne, K. D. Watenpaugh, eds., IUCr Crystallographic Symposia, Oxford University Press,, 2001. 386, 389, 395 [13] J.Toriwaki and T. Fukumura. Extraction of structural information from grey pictures. Computer Graphics and Image Processing, 7:30–51, 1975. 387 [14] F. Meyer. Topographic distance and watershed lines. Signal Processing, 38(1):113–125, 1994. 387 [15] J. Milnor. Morse Theory. Princeton University Press, 1963. 386, 388 [16] L. R. Nackman. Two-dimensional critical point configuration graph. IEEE Transactions on Pattern Analysisand Machine Intelligence, PAMI-6(4):442–450, 1984. 386 [17] T. K. Peucker and E. G. Douglas. Detection of surface-specific points by local paprallel processing of discrete terrain elevation data. Graphics Image Processing, 4:475–387, 1975. 387 [18] S. Smale. Morse inequalities for a dynamical system. Bulletin of American Mathematical Society, 66:43–49, 1960. 386, 388, 394, 395 [19] R. Thom. Sur une partition en cellule associées a une fonction sur une variété. C.R.A.S., 228:973–975, 1949. 386, 388 [20] L. T. Watson, T. J. Laffey, and R.M. Haralick. Topographic classification of digital image intensity surfaces using generalised splines and the discrete cosine transformation. Computer Vision, Graphics and Image Processing, 29:143–167, 1985. 386 [21] G.H. Weber, G Scheuermann, H. Hagen, and B. Hamann. Exploring scalar fields usign criticla isovalues. In Proceedings IEEE Visualization 2002, pages 171–178. IEEE Computer Society Press, 2002. 386, 401

External versus Internal Parameterizations for Lengths of Curves with Nonuniform Samplings Ryszard Kozera1 , Lyle Noakes2, and Reinhard Klette3 1 The University of Western Australia School of Computer Science and Software Engineering 35 Stirling Highway, Crawley WA 6009, Australia 2 The University of Western Australia School of Mathematics and Statistics 35 Stirling Highway, Crawley WA 6009, Australia 3 The University of Auckland Centre for Images Technology and Robotics Tamaki Campus, Building 731, Auckland, New Zealand

Abstract. This paper studies differences in estimating length (and also trajectory) of an unknown parametric curve γ : [0, 1] → IRn from an ordered collection of data points qi = γ(ti ), with either the ti ’s known or unknown. For the ti ’s uniform (known or unknown) piecewise Lagrange interpolation provides efficient length estimates, but in other cases it may fail. In this paper, we apply this classical algorithm when the ti ’s are sampled according to first α-order and then when sampling is εuniform. The latter was introduced in [20] for the case where the ti ’s are unknown. In the present paper we establish new results for the case when the ti ’s are known for both types of samplings. For curves sampled ε-uniformly, comparison is also made between the cases, where the tabular parameters ti ’s are known and unknown. Numerical experiments are carried out to investigate sharpness of our theoretical results. The work may be of interest in computer vision and graphics, approximation and complexity theory, digital and computational geometry, and digital image analysis.

1

Introduction

For k ≥ 1, consider the problem of estimating the length d(γ) of a C k regular parametric curve γ : [0, 1] → IRn from ordered (m + 1)-tuples Qm = (q0 , q1 , . . . , qm ) of points qi = γ(ti ) on the curve γ. In this paper the tabular parameters ti ’s are assumed to be either known or at least distributed in some specific manner. The problem is easiest when the ti ’s are chosen uniformly, namely ti = mi (see [15] or [26]). In such a case it seems natural to approximate γ by a curve γ r

This research was supported by an Australian Research Council Small Grant1,2 and by an Alexander von Humboldt Research Fellowship1 .

T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 403–418, 2003. c Springer-Verlag Berlin Heidelberg 2003

404

Ryszard Kozera et al.

that is piecewise polynomial of degree r ≥ 1. The following result can be proved (see [20]): Theorem 1. Let γ be C r+2 , with the ti ’s be sampled uniformly. Then a piecewise-r-degree Lagrange polynomial γr determined by Qm satisfies 1 O( mr+1 ) if r ≥ 1 is odd , d( γr ) − d(γ) = (1) 1 O( mr+2 ) if r ≥ 1 is even , and γ − γ r ∞ = O(

1 ). mr+1

(2)

As usual, O(am ), means a quantity whose absolute value is bounded above by some constant multiple of am as m → ∞. Both asymptotic estimates appearing in (1) and (2) are sharp (see [20]), namely there exist C r+2 regular curves γ which, when sampled uniformly, yield lower bounds of convergence rates such as specified in the upper bounds (1) or (2). Consider samplings of the following type. Definition 1. We say that sampling {ti }m i=0 is of α-order, for some 0 < α ≤ 1, if ti < ti+1 and the following holds ti+1 − ti = O(

1 ). mα

(3)

The second part of the present paper is mainly concerned with the case, where α = 1. We may ask whether Theorem 1 extends, either to an arbitrary sampling (3), or to some subclasses of (3) for both the ti ’s known or unknown. More specifically, we examine the existence of some β1 , β2 > 0 yielding d( γr ) − d(γ) = O(

1 1 ) and γ − γ r ∞ = O( β2 ) . β 1 m m

(4)

Subsequently, the comparison and the analysis of underlying difference between internal and external parameterizations (the ti ’s known versus unknown) will follow. Those two issues are treated in this paper in detail and some new results for internal parameterization are established. Evidently, the knowledge of explicit distribution of the tabular points ti ’s, provides extra information to the problem in question (including the order of the points in Q). Thus, as expected and proved later in this paper a nonuniform case (3) together with internal parameterization yields a better result than its external parameterization counterpart. The latter is in contrast with the uniform case where the corresponding convergence rates coincide - see Theorem 1. Note that if the ti ’s are unknown, the order of points in Qm is also assumed to be given. This work is relevant to some computer vision problems: tracking an object or its center of mass from satellite or video images, finding the boundary of

External versus Internal Parameterizations

405

planar objects (for example in medical image analysis or automated production line) or handling any data (such as a sequence of video images) parametrized by one parameter in decompressing, interpolation, or noise rectification processes. There is another context of possible applications outside the scope of approximation theory. Recent research in digital and computational geometry and digital image analysis concerns analogous work for estimating lengths of digitized curves. Depending on the digitization model [11], γ is mapped onto a digital curve and approximated by a polygonal curve γˆm whose length is an estimator for d(γ). Approximating polygons γˆm based on local configurations of digital curves do not ensure multigrid length convergence, but global approximation techniques 1 yield linearly convergent estimates, namely d(γ) − d(ˆ γm ) = O( m ) [1], [13], [14] or [25]. Recently, experimentally based results reported in [4], [5], [6], and [12] confirm a similar rate of convergence for γ ⊂ IR3 . In the special case of discrete straight line segment in IR2 a stronger result is proved, for example, [8], where O( m11.5 ) errors for asymptotic length estimates are established. Our paper focuses on curve interpolation and asymptotical analysis is based on the number of interpolation points. On the other hand digital models assume curve approximation and the corresponding asymptotics is based on the size of image resolution. So strict comparisons cannot be made yet. However, as a special case we provide upper bounds for optimal rates of convergence when piecewise polynomials are applied to the digitized curves. Related work can also be found in [2], [3], [9], [10], [22], and [24]. There is also some interesting work on complexity [7], [23], and [27]. The layout of the present paper is as follows. The first part is mainly expository with some extension of standard result for 1-order case to α-order one (see Theorems 1 and 2). The second part discusses essential differences in estimating length and trajectory of γ between both cases with the interpolation times ti ’s either known or unknown. In particular the above difference for ε-uniform sampling (constituting a special case of 1-order sampling) is empahsized in Theorem 3 and Theorem 4. Finally, as Theorem 4 also indicates, if the ti ’s are known, the results in Theorem 2 covering also ε-uniform case (as a special 1-order one) can be strengthened.

2

Preliminaries

Let · be the Euclidean norm in IRn , where n ≥ 1, with < ·, · > the corresponding inner product. The length d(γ) of a C k parametric curve (k ≥ 1) γ : [0, 1] → IRn is defined as 1 d(γ) = γ(t)dt ˙ , 0

where γ(t) ˙ ∈ IR is the derivative of γ at t ∈ [0, 1]. The curve γ is said to be regular when γ(t) ˙ = 0, for all t ∈ [0, 1]. A reparameterization of γ is a parametric

curve of the form γ ◦ ψ : [0, 1] → IRn , where ψ : [0, 1] → [0, 1] is a C k diffeomorphism. The reparameterization γ ◦ ψ has the same image and length as γ. n

406


For simplicity we assume here that ψ is C ∞ . Let γ be regular: then so is any reparameterization γ ◦ψ. Recall that a regular curve γ is said to be parameterized proportionally to arc-length when γ(t) ˙ is constant for t ∈ [0, 1]. We want to estimate d(γ) from ordered (m + 1)-tuples Qm = (q0 , q1 , q2 , . . . , qm ) ∈ (IRn )m+1 , where qi = γ(ti ), whose parameter values ti ∈ [0, 1] are either known or unknown and sampled in some reasonably regular way. We are going to discuss different ways of forming ordered samples 0 = t0 < t1 < t2 < . . . < tm = 1 of variable size m + 1 from the interval1 [0, 1]. The simplest procedure is unii form sampling, where ti = m (where 0 ≤ i ≤ m). Uniform sampling is not invariant with respect to reparameterizations, namely order-preserving C ∞ diffeomorphisms φ : [0, 1] → [0, 1]. A small perturbation of uniform sampling is no longer uniform, but may approach uniformity in some asymptotic sense, at least after some suitable reparameterization. We define now a special subclass of (3) (see also [20]), namely a special type of 1-order sampling: Definition 2. For 0 ≤ ε ≤ 1, the ti ’s are said to be ε-uniformly sampled when there is an order-preserving C ∞ reparameterization φ : [0, 1] → [0, 1] such that ti = φ(

1 i ) + O( 1+ε ) . m m

Note that ε-uniform sampling arises from two types of perturbations of uniform sampling: first via a diffeomorphism φ : [0, 1] → [0, 1] combined subsequently 1 ). In particular, for φ the identity, and with added extra distortion term O( m1+ε ε = 0 (ε = 1) the perturbation is linear (quadratic), which constitutes asymptotically a big (small) distortion of a uniform partition of [0, 1]. The extension of Definition 2 to ε > 1 could also be considered. This case represents, however, a very small perturbation of uniform sampling (up to a φ-shift) which seems to be of less interest in applications. As mentioned the perturbation of uniform sampling via φ has no effect on both d(γ) and geometrical representation of γ. 1 ). The only potential nuisance stems from the second perturbation term O( m1+ε ∞ Finally, note that ε-uniform sampling is invariant with respect to C order preserving reparameterizations ψ : [0, 1] → [0, 1]. So suppose in all the following, without loss of generality, that γ is parameterized proportionally to arc-length. We shall need later the following lemma (see [16]; Lemma 2.1): Lemma 1. Let f : [a, b] → IRn be C l , where l ≥ 1 and assume that f (t0 ) = 0, for some t0 ∈ (a, b). Then there exists a C l−1 function g : [a, b] → IRn such that f (t) = (t − t0 )g(t). 1

In the present context there is no real gain in generality from considering other intervals [0, T ].


407

Proof. For each i-th component of f = (f1 , f2 , . . . , fn ) consider Fi : [0, 1] → IR Fi (u) = fi (tu + (1 − u)t0 ). By the Fundamental Theorem of Calculus 1 fi (t) = Fi (1) − Fi (0) = (t − t0 ) fi (tu + (1 − u)t0 ) du . 0

Take g = (g1 , g2 , . . . , gn ), where gi (t) = 0

1

fi (tu + (1 − u)t0 ) du .

This proves Lemma 1. The proof of Lemma 1 shows also that g = O( df ), namely the uniform norm dt of g is bounded by a constant multiple of the uniform norm of df dt . Here f may depend on some other parameter m → ∞. If f has multiple zeros t0 < t1 < . . . < tk then k + 1 applications of Lemma 1 give f (t) = (t − t0 )(t − t1 )(t − t2 ) . . . (t − tk )h(t) ,

(5)

k+1

where h is C l−(k+1) and h = O( ddtk+1f ).

3

Internal and External Clocks for α-Order Samplings

We begin with some results for estimating d(γ) and γ when piecewise-r-degree Lagrange interpolants are used with internal parameterization applied to arbitrary sampling of α-order (for proof see Appendix 1). When α = 1 formula (6) is well-known. Theorem 2. Let γ be C r+2 and let the ti ’s be given explicitly and sampled according to α-order. Then a piecewise-r-degree Lagrange polynomial γ r , determined by Qm yields d( γr ) − d(γ) = O(

1 1 ) and γ − γ r ∞ = O( α(r+1) ) . mα(r+2)−1 m

(6)

1 Remark 1: Note that, if α ≤ r+2 then formula (6) does not guarantee convergence for d(γ) estimation. On the other hand, the most interesting case when α = 1 renders convergence for arbitrary r > 0 integer. For the general case when the ti ’s are unknown and sampling is of α-order, Lagrange interpolation for length estimation can behave badly. For example, consider the most interesting case when α = 1. From now we shall call the derivation of γ 2 as a QS-Algorithm (Quadratic Sampler). The next example shows that for the ti ’s unknown with α = 1 in (3) and r = 2, the formula (4) may not hold even if γ is well approximated.

Example 1. Consider the following two families of the ti ’s distributions:  i if i even , m 1 if i odd & i = 4k + 1 , ti = mi + 2m  i 1 − m 2m if i odd & i = 4k + 3 ,

(7)

408


-1

1

1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

-0.5

0.5

1

-1

-0.5

-0.2

(a)

1

(b)

1.2 1 0.8 0.6 0.4 0.2 -0.2

0.5 -0.2

1.2 1 0.8 0.6 0.4 0.2 1

2

3

(c)

4

-0.2

1

2

3

4

(d)

Fig. 1. (a) γ 2 for a semicircle γs and (7). (b) γ 2 for a semicircle γs and (8). (c) γ2 for a cubic curve γc and (8) γ 2 for a cubic curve γc and (7). (d)

(−1)i+1 i + , (8) m 3m with t0 = 0 and tm = 1. In order to generate synthetically sampling points Qm assume temporarily that the ti ’s distributions from (7) and (8) are known and that the analytic formulae for regular curves semicircle and cubic curve γs , γc : [0, 1] → IR2 ti =

γs (t) = (cos(π(1 − t)), sin(π(1 − t))) and γc (t) = (πt, (

πt + 1 3 ) ) π+1

(9)

are given. Consequently, upon deriving initial data Qm , the QS-Algorithm is used merely with Qm . As it turns out with uniform estimate tî = i/m of the ti ’s, QSAlgorithm yields a good trajectory estimation in either cases (see Figure 1). Note also that for synthetic generation of curve samplings proportional to arc-length parameterization is not needed. Only the existence of the latter (assured by the regularity of γ) is used to prove both Theorems 2 and 4. On the other hand the length estimation by QS-Algorithm (used with tî = i/m) for d(γs ) = π and d(γc ) = 3.3452 yields a dual result (see Table 1), where 200 γ2 )| and βd(γ) defines an estimate of β1 (see (4)) found by linear ρm d(γ) = |d(γ)−d( regression applied to the pairs of points (log(m), − log(ρm d(γ) )), with m running from 6 to 200. In contrast, if the ti ’s for both samplings (7) and (8) are known, then QSAlgorithm yields a better result for (4) (see Table 2). In the next section we discuss a similar problem of estimating d(γ) with either internal or external parameterizations used and applied to the special subclass of 1-order samplings, namely the so-called ε-uniform ones.


409

Table 1. d(γ) estimation by QS-Algorithm with the ti ’s unknown curves:

a

semicircle γs

cubic curve γc

samplings:

(7)

(8)

200 βd(γ) :

1.44

n/aa

(7)

(8) n/aa

1.99

ρ200 3.45x10−4 0.1288 6.36x10−8 0.1364 d(γ) : not applicable: limm→∞ d( γr ) exists but is not equal to d(γ).

Table 2. d(γ) estimation by QS-Algorithm with the ti ’s known curves:

semicircle γs

samplings:

(7)

200 βd(γ) :

3.99

ρ200 d(γ) :

cubic curve γc

(8)

(7)

4.02 −9

4.52x10

(8)

3.99 −11

2.26x10

2.99 −9

5.54x10

1.39x10−8

Table 3. d(γ) estimation: r = 2 and the ti ’s are known α-order samplings curves:

semicircle γs

cubic curve γc

samplings:

α = 1/2

α = 1/3

200 βd(γ) :

2.46

1.61

ρ200 d(γ) :

−7

7.32x10

α = 1/2 2.09

−4

0.71x10

α = 1/3 1.43

−7

1.10x10

6.03x10−6

In the last example, among all, the sharpness (6) for length estimation was confirmed when α = 1, r = 2 with internal clock available. The validity of (6) can in fact be similarly verified for all r integer and α = 1. The next example tests the case for some 0 < α < 1, r = 2, 3 and the ti ’s known. Example 2. Consider the following α-order samplings ti = (i/m)α , for 0 < α < 1. For γc and γs defined in Example 1, the QS-Algorithm yields: Similarly, for r = 3 (here γ 3 forms a piecewise cubic spline) and for γs and for 4 a quartic curve γq4 (t) = (πt, ( πt+1 π+1 ) ) (where t ∈ [0, 1]) for which d(γq4 ) = 3.3909, the results are shown in Table 4. Note that γc was replaced here by γq4 as otherwise piecewise cubic spline γ3 coincides with γc thus yielding error equal zero.

410


Table 4. d(γ) estimation: r = 3 and the ti ’s are known α-order samplings curves:

semicircle γs

quartic curve γq4

samplings:

α = 1/2

α = 1/3

200 βd(γ) :

2.46

1.60

ρ200 d(γ) :

−6

4.65x10

α = 1/2 2.64

−3

2.41x10

α = 1/3 1.81

−8

1.74x10

1.18x10−6

The convergence rates in Table 3 (or in Table 4) are faster than the corα=1/2 responding β1 from Theorem 2 for r = 2 (or r = 3), namely: β1 = 1 (or α=1/2 α=1/3 α=1/3 = 1.5) and β1 = 1/3 (or β1 = 2/3), respectively. As it stands β1 now it remains an open problem whether for 0 < α < 1 and arbitrary r Theorem 2 indeed provides sharp estimates. In the next section we will establish sharp estimates for the special subclass of 1-order sampling, namely for ε-uniform with internal (when r > 0) and external parameterizations (when r = 2) used.

4

Internal & External Clocks for ε-Uniform Samplings

In this section we shall discuss the performance of QS-Algorithm (r = 2) for εuniformly sampled C r+2 curves. Note that both examples of 1-order samplings (7) and (8) are also 0-uniform samplings. As shown in Example 1 Lagrange interpolants for length estimation can behave badly for 0-uniform sampling and external parameterizations (where tî = i/m is used to approximate ti ). The more elaborate algorithms of [17], [18] or [19] are needed for this case to correctly in parallel estimate both γ and the ti ’s distribution. However, for ε > 0 and QSAlgorithm the following can be proved (see [20]): Theorem 3. Let the ti ’s be unknown and sampled ε-uniformly, where ε > 0, and suppose that γ is C 4 . Then QS-Algorithm used with tî = i/m yields d( γ2 ) = d(γ) + O(

1 m4 min{1,ε}

) , γ − γ 2 ∞ = O(

1 m1+2 min{1,ε}

).

(10)

The estimates from Theorem 3 are sharp (see [20] and [21]). Note that for ε = 0 the proof of Theorem 3 fails and in fact as shown in Example 1 dual outcomes are possible. Whereas Theorems 1, 2 permit length estimates of arbitrary accuracy (for r arbitrary large or r > α1 − 2, respectively) Theorem 3 refers only to piecewisequadratic estimates, and accuracy is limited accordingly. The proof of Theorem 3 shows that if r > 2 and the ti ’s are unknown, then any convergence result for γ r and tî = i/m requires ε to be large. The latter would force the sampling to be almost uniform which does not constitute the most interesting case. Note


411

also that if r = 1 a piecewise linear interpolation provides the same quadratic convergence rates (see proof of Theorem 3) independently whether the ti ’s are known or unknown. Equal convergence rates result from the existence of exactly one (and the same for the ti ’s known and unknown) linear interpolant passing through two points in IRn . Note that if the ti ’s are known for ε-uniform sampling (for which α = 1) by sharpness of Theorem 1 and 2 the following hold r + 1 ≤ β1 ≤ r + 2 (if r is even) and r + 1 ≤ β1 ≤ r + 1 i.e. β1 = r + 1 (if r is odd). It turns out that for ε-uniform samplings (a subclass of 1-order sampling (3)) a tighter result than claimed by Theorem 2 can be proved at least for r even (for a proof which constitutes a new result see Appendix 2). Theorem 4. If sampling is ε-uniform, ε ≥ 0 and γ ∈ C r+2 then with the ti ’s known explicitly piecewise-r-degree Lagrange interpolation yields 1 O( mr+1 ) if r ≥ 1 is odd , d( γr ) − d(γ) = (11) 1 O( mr+1+min{1,ε} ) if r ≥ 1 is even , and γr − γ∞ = O(

1 mr+1

).

(12)

Remark 2: Note that Theorem 4 can be applied to the extended definition of ε-uniform samplings namely: −1 < ε < 0, for which in fact ti = O( m1α ) satisfying (3) with 0 < α < 1 and α = 1 + ε. Then formula (32) is replaceable by 1 1 ) and as α(r + 2) ≤ r + 2 we would have (33) of order O( mα(r+2) ). This O( mα(r+2) consequently yields the same length estimates as Theorem 2 with 0 < α < 1. There is still, however need for Theorem 1 as not all order preserving samplings (3) are of the form ti = O( m1α ). Next we test the sharpness of the theoretical results in Theorem 4 with some numerical experiments which assume the ti ’s to be known. Example 3. Experiments as in the previous section were performed with Mathematica on a 700 MHZ Pentium III with 384 MB RAM. We show first the sharpness of (11) for r = 2 and γc sampled according to ε-uniform sampling: ti =

(−1)i+1 i + , m 3m1+ε

(13)

with d(γc ) = 3.3452. We use a similar notation in Table 5 as in Example 1. 200 Note that computed rates βd(γ) nearly coincide with those asserted by the Theorem 4, namely: for ε = 2, 1, 1/2, 1/3, 1/10, 5/100, 1/100, and 0 we have ε=1/2 ε=1/3 ε=1/10 ε=5/100 β1ε=2 = 4, β1ε=1 = 4, β1 = 3.5, β1 = 10/3, β1 = 3.1, β1 = ε=1/100 = 3.01, and β10 = 3, respectively. Similar sharp results can be 3.05, β1 obtained for r = 4 and (13) with ε = 0, 0.5, 2 yielding β1 = 4.91, 5.31, 5.88, respectively. Here the cubic curve (9) is replaced by a quintic curve γq5 (t) = 5 (πt, ( πt+1 π+1 ) ), with t ∈ [0, 1] and d(γq5 ) = 3.4319. Otherwise a piecewise quartic

412


Table 5. d(γ) estimation: r = 2 and the ti ’s known from (13) computed β1 for γc ε

2

1

1/2

1/3

1/10

5/100

1/100

0

200 βd(γ)

4.00

4.01

3.48

3.32

3.09

3.04

3.00

3.00

spline γ4 coincides with γc thus yielding error equal zero. The computed estimates are slightly less than (11) with r = 4 (they should be at least 5, 5.5, and 6, respectively) as only a small number of interpolation points was considered before reaching machine precision during integration. Of course, the asymptotical nature of Theorem 4 requires m to be sufficiently large. Finally, for r = 3 and γs we have for ε = 1, 0.5, 0 the following values β1 = 3.99, 4.02, and 3.92, respectively. The latter coincides with α = 4 claimed by Theorem 4 which strongly confirms the sharpness of the last theorem also for r odd.

5

Conclusions

We examined here a class of α-order and ε-uniform samplings for piecewise Lagrange interpolation to give length (and trajectory) estimates converging to d(γ), including investigation of convergence rates for both internal (with the ti ’s known) and external (with tî = i/m taken as estimates of ti ) parameterizations. Our results are confirmed to be sharp or nearly sharp for both classes of samplings.

Acknowledgment The authors thank the referees for valuable comments.

References [1] Asano T., Kawamura Y., Klette R., Obokkata K. (2000) A new approximation scheme for digital objects and curve length estimation. In: Cree M. J., SteynRoss A. (eds) Proc. Int. Conf. Image and Vision Computing New Zealand, Hamilton, New Zealand, Nov. 27-29, 2000. Dep. of Physics and Electronic Engineering, Univ. of Waikato Press, 26–31. 405 [2] Barsky B. A., DeRose T. D. (1989) Geometric continuity of parametric curves: three equivalent characterizations. IEEE. Comp. Graph. Appl. 9:60–68. 405 [3] Boehm W., Farin G., Kahmann J. (1984) A survey of curve and surface methods in CAGD. Comput. Aid. Geom. Des., 1:1–60. 405


413

[4] B¨ ulow T., Klette R. (2000) Rubber band algorithm for estimating the length of digitized space-curves. In: Sneliu A., Villanva V. V., Vanrell M., Alquézar R., Crowley J., Shirai Y. (eds) Proc. 15th Int. IEEE Conf. Pattern Recognition, Barcelona, Spain, Sep. 3-8, 2000, Vol. III, 551–555. 405 [5] B¨ ulow T., Klette R. (2001) Approximations of 3D shortest polygons in simple cube curves. In: Bertrand G., Imiya A., Klette R. (eds) Digital and Image Geometry, Springer, LNCS 2243, 285–295. 405 [6] Coeurjolly D., Debled-Rennesson I., Teytaud O. (2001) Segmentation and length estimation of 3D discrete curves. In: Bertrand G., Imiya A., Klette R. (eds) Digital and Image Geometry, Springer, LNCS 2243, 299–317. 405 [7] D¸abrowska D., Kowalski M. A. (1998) Approximating band- and energy-limited signals in the presence of noise. J. Complexity 14:557–570. 405 [8] Dorst L., Smeulders A. W. M. (1991) Discrete straight line segments: parameters, primitives and properties. In: Melter R., Bhattacharya P., Rosenfeld A. (eds) Ser. Contemp. Maths., Amer. Math. Soc., 119:45–62. 405 [9] Epstein M. P. (1976) On the influence of parametrization in parametric interpolation. SIAM. J. Numer. Anal., 13:261–268. 405 [10] Hoschek J. (1988) Intrinsic parametrization for approximation. Comput. Aid. Geom. Des., 5:27–31. 405 [11] Klette R. (1998) Approximation and representation of 3D objects. In: Klette R., Rosenfeld A., Sloboda F. (eds) Advances in Digital and Computational Geometry. Springer, Singapore, 161–194. 405 [12] Klette R., B¨ ulow T. (2000) Critical edges in simple cube-curves. In: Borgefors G., Nystr¨ om I., Sanniti di Baja G. (eds) Proc. 9th Int. Conf. Discrete Geometry for Computer Imagery, Uppsala, Sweden, Dec. 13-15, 2000, Springer, LNCS 1953, 467–478. 405 [13] Klette R., Kovalevsky V., Yip B. (1999) On the length estimation of digital curves. In: Latecki L. J., Melter R. A., Mount D. M., Wu A. Y. (eds) Proc. SPIEConf. Vision Geometry VIII, Denver, USA, July 19-20, 1999, 3811:52–63. 405 [14] Klette R., Yip B. (2000) The length of digital curves. Machine Graphics and Vision, 9:673–703. 405 [15] Moran P. (1966) Measuring the length of a curve. Biometrika, 53:359–364. 403 [16] Milnor J. (1963) Morse Theory. Princeton Uni. Press, Princeton, New Jersey. 406 [17] Noakes L., Kozera R. (2002) More-or-less-uniform sampling and lengths of curves. Quart. Appl. Maths., in press. 410 [18] Noakes L., Kozera R. (2002) Interpolating sporadic data. In: Heyden A., Sparr G., Nielsen M., Johansen P. (eds) Proc. 7th European Conf. Comp. Vision, Copenhagen, Denmark, May 28-31, 2002, Springer, LNCS 2351, 613–625. 410 [19] Noakes L., Kozera R. (2002) Cumulative chord piecewise quadratics. In: Wojciechowski K. (ed.) Proc. Int. Conf. Computer Vision and Graphics, Zakopane, Poland, Sep. 25-29, 2002, Association of Image Processing of Poland, Silesian Univ. of Technology and Institute of Theoretical and Applied Informatics, PAS, Gliwice Poland, Vol. 2, 589–595. 410 [20] Noakes L., Kozera R., Klette R. (2001) Length estimation for curves with different samplings. In: Bertrand G., Imiya A., Klette R. (eds) Digital and Image Geometry, Springer, LNCS 2243, 339–351. 403, 404, 406, 410 [21] Noakes L., Kozera R., Klette R. (2001) Length estimation for curves with εuniform sampling. In: Skarbek W. (ed.) Proc. 9th Int. Conf. Computer Analysis

414

[22] [23] [24]

[25]

[26] [27]

Ryszard Kozera et al. of Images and Patterns, Warsaw, Poland, Sep. 5-7, 2001, Springer, LNCS 2124, 518–526. 410 Piegl L., Tiller W. (1997) The NURBS Book. Springer, Berlin. 405 Plaskota L. (1996) Noisy Information and Computational Complexity. Cambridge Uni. Press, Cambridge. 405 Sederberg T. W., Zhao J., Zundel A. K. (1989) Approximate parametrization of algebraic curves. In: Strasser W., Seidel H. P. (eds) Theory and Practice in Geometric Modelling. Springer, Berlin, 33–54. 405 Sloboda F., Zaˇtko B., St¨ or J. (1998) On approximation of planar one-dimensional continua. In: Klette R., Rosenfeld A., Sloboda F. (eds) Advances in Digital and Computational Geometry. Springer, Singapore, 113–160. 405 Steinhaus H. (1930) Praxis der Rektifikation und zur L¨ angenbegriff (In German). Akad. Wiss. Leipzig, Berlin 82:120–130. 403 Traub J. F., Werschulz A. G. (1998) Complexity and Information. Cambridge Uni. Press, Cambridge. 405

Appendix 1 In this Appendix we shall prove Theorem 2. Part of the proof from this section shall be used also in Appendix 2 to justify Theorem 4. Proof. Suppose that γ is C k , where k = r + 2 with r ≥ 1, and (without loss of generality) that m is a multiple of r. Then Qm gives m r (r + 1)-tuples of the form (q0 , q1 , . . . , qr ), (qr , qr+1 , . . . , q2r ), . . . , (qm−r , qm−r+1 , . . . , qm ) . The j-th (r + 1)-tuple is interpolated by the r-degree Lagrange polynomial Prj : [t(j−1)r , tjr ] → IRn , here 1 ≤ j ≤ m r : Prj (t(j−1)r ) = q(j−1)r , . . . , Prj (tjr ) = qjr . Clearly, each Prj is defined in terms of a global parameterization t ∈ [t(j−1)r , tjr ]. A simple inspection shows that f = Prj − γ : [t(j−1)r , tjr ] → IRn is C r+2 and that it satisfies f (t(j−1)r ) = f (t(j−1)r+1 ) = . . . = f (tjr ) = 0 . Note also that Prj depends implicitly on m and thus f (and later h) should be understood as a sequence of fm , while m varies. By Lemma 1 and (5) we have f (t) = (t − t(j−1)r )(t − t(j−1)r+1 ) . . . (t − tjr )h(t) ,

(14)

where h : [t(j−1)r , tjr ] → IRn is C 1 . Still by proof of Lemma 1 h(t) = O(

dr+1 γ dr+1 f ) = O( ) = O(1) , dtr+1 dtr+1

(15)


because deg(Prj ) ≤ r and

dr+1 γ dtr+1

415

is O(1). Thus by (3), (14), and (15) we have

f (t) = O(

1 ), mα(r+1)

for t ∈ [t(r−1)j , trj ]. This completes the proof of the second formula in (6). Furthermore, differentiating function h (defined as a (r + 1)-multiple integral of f (r+1) over the compact cube [0, 1]r+1 ; see proof of Lemma 1) yields dr+2 f dr+2 γ ˙ h(t) = O( r+2 ) = O( r+2 ) = O(1) , dt dt

(16)

as deg(Prj ) ≤ r. Thus by (3), (14), and (16) f˙ = O( m1αr ) and hence for t ∈ [t(j−1)r , tjr ] 1 γ(t) ˙ − P˙rj (t) = f˙(t) = O( αr ) . (17) m ˙ Since Let Vγ˙⊥ (t) be the orthogonal complement of the line spanned by γ(t). γ(t) ˙ = d(γ) (as γ can be parameterized proportionally to arc-length) ˙ > < P˙rj (t), γ(t) P˙rj (t) = γ(t) ˙ + v(t) , 2 d(γ)

(18)

where v(t) is the orthogonal projection of P˙rj (t) onto Vγ˙⊥ (t). As P˙rj (t) = f˙(t) + γ(t) ˙ and γ(t) ˙ = d(γ), by (18) we have < f˙(t), γ(t) ˙ > v(t) = f˙(t) − γ(t) ˙ . 2 d(γ) The latter combined with (17) yields v = O( m1αr ). Hence as by (17) and (18) < f˙(t), γ(t) ˙ > P˙rj = (1 + )γ(t) ˙ + v(t) d(γ)2 and as < γ(t), ˙ v(t) >= 0, the Binomial Theorem yields < f˙(t), γ(t) ˙ > 1 P˙rj (t) = γ(t) ˙ 1+2 + O( 2αr ) 2 d(γ) m ˙ < f (t), γ(t) ˙ > 1 = γ(t)(1 ˙ + ) + O( 2αr ) . d(γ)2 m ˙

(19)

γ(t)> ˙ 1 Note that by (17) |2 j ˙ dt + O( α(2r+1) ) (Pr (t) − γ(t)) ˙ dt = d(γ) m t(j−1)r t(j−1)r tjr 1 < f (t), γ¨ (t) > dt + O( α(2r+1) ) . (20) =− d(γ) m t(j−1)r

416


Since γ is compact and at least C 3 by (15), (16), and h = O(1) we have < h(t), γ¨ (t) >= O(1) ,

˙ γ¨ (t) >= O(1) . < h(t), γ (3) (t) >= O(1) and < h(t),

Hence, by (14) and Taylor’s Theorem applied to r(t) =< h(t), γ¨ (t) > at t = t(j−1)r , we get < f (t), γ¨ (t) >= (t − t(j−1)r ) . . . (t − tjr )(a + O(

1 )) , mα

(21)

where a is constant in t and O(1). Note that it is important that a is of order O(1) as it varies with m changed. Thus by (20) and (21) we arrive at tjr 1 (P˙ rj (t) − γ(t)) ˙ dt = O( α(r+2) ) . m t(j−1)r As already defined take γr to be a track-sum of the Prj , i.e. m

−1

r d( γr ) = Σj=0 d(Prj ) = d(γ) + O(

1 ). mα(r+2)−1

This proves the Theorem 1.

Appendix 2 In this Appendix we justify Theorem 4. Proof. The second formula (12) results directly from Theorem 2 by setting α = 1 (as each ε-uniform sampling with ε ≥ 0 is also a 1-order sampling). Furthermore, upon repeating the argument from Theorem 2 up to (21) we obtain < f (t), γ¨ (t) >= (t − t(j−1)r ) . . . (t − tjr )(a + O(

1 )) , m

(22)

where a is constant in t and O(1). Upon substitution (t(j−1)r , t(j−1)r+1 , . . . , tjr ) = (t0 , t1 , . . . , tr ) let χi : IRr+1 → IR be defined as tr (t − t0 ) . . . (t − tr ) dt , (23) χi (h) = t0

0 ≤ k ≤ r) with h = (h0 , h1 , . . . , hr ) ∈ 0 ≤ k ≤ r. By Taylor’s Theorem and ¯ δ) ⊂ IRr+1 ε-uniformity there exists δ > 0 such that for each h ∈ B(0, where i = (j − 1)r, tk = φ( i+k m ) + hk (for 1 IRr+1 satisfying hk = O( m1+ε ), for each

χi (h) = χi (0) + Dh χi (ξ(h))(h) ,

(24)

¯ δ) positioned on the line between with ξ(h) = (ξ0 (h), ξ1 (h), . . . , ξr (h)) ∈ B(0, 1 1 ) (and thus here δ = O( m1+ε )). Furthermore, the 0 ∈ IRr+1 and h = O( m1+ε integral (23) at h = 0 upon integration by substitution reads i+r m i+r ˙ i ))φ(s) ds . (25) (φ(s) − φ( )) . . . (φ(s) − φ( χi (0) = i m m m


417

Again, Taylor’s Theorem applied to each factor of the integrand of (25) combined with compactness of [0, 1] and φ being a diffeomorphism yields χi (0) = b

i+r m

(s −

i m

i 1 ˜ 0 ) . . . (s − i + r + ˜hr )(φ(0) ˙ +h + O( )) ds , m m m

r ˙ i+k ) is constant in s and O(1) and ˜hk = O( 12 ) (for 0 ≤ k ≤ where b = k=0 φ( m m r). Furthermore, χi (0) = c

i+r m i m

(s −

i+r 1 i ) . . . (s − ) ds + O( r+3 ) , m m m

(26)

˙ where c = bφ(0) is constant in s and O(1). Again, as previously, it is vital that both b and c are of order O(1), since they vary with m. A simple verification shows that the integral in (26) either vanishes for r even or otherwise is of order 1 ). Hence O( mr+2 1 O( mr+2 ) if r ≥ 1 is odd , χi (0) = (27) 1 O( mr+3 ) if r ≥ 1 is even . In order to determine the asymptotics of the second term in (24) let i i+r ) − hr ) . fi (t, h0 , . . . , hr ) = (t − φ( ) − h0 ) . . . (t − φ( m m

(28)

i 1 As [φ( m ) + h0 , φ(( i+r) m ) + hr ] is compact and fi (t, h) is C we have

∂χi (h) = ∂hk

φ( i+r m )+hr

i φ( m )+h0

∂ fi (t, h) dt , for 1 ≤ k ≤ r − 1 . ∂hk

(29)

Similarly, ∂χi (h) = ∂h0

φ( i+r m )+hr

i φ( m )+h0

∂ fi i (t, h) dt − fi (φ( ) + h0 , h) . ∂h0 m

(30)

Note that by (28) the second term in (30) vanishes. Thus formulae (29) extend to k = 0 and similarly to k = r. Hence by the Mean Value Theorem the second term in (24) satisfies Dh χi (ξ(h))(h) =

r k=0

=

r k=0

O(hk )O(φ(

hk

φ( i+r m )+ξr (h)

i φ( m )+ξ0 (h)

∂ fi (t, ξ(h)) dt ∂hk

i i+r ∂ fi ) − φ( ) + ξr (h) − ξ0 (h))O( (t, ξ(h))) , m m ∂hk (31)

418


¯ with t ∈ Iξ = [φ( mi ) + ξ0 (h), φ( i+r m ) + ξr (h)] and, where as in (24) h ∈ B(0, δ) r+1 ¯ and ξ(h) ∈ B(0, δ) is positioned on the line between 0, h ∈ IR . By Taylor’s i 1 Theorem φ( i+r m ) − φ( m ) = O( m ) and |ξr (h) − ξ0 (h)| ≤ 2h = O(

1 m1+ε

).

1 Similarly, for each 0 ≤ l ≤ r we have t − φ( i+l m ) − ξl (h) = O( m ) and thus as ∂ fi (t, ξ(h)) = O( m1r ). Hence the asymptotics in (31) t ∈ Iξ by (28) we have ∂h k coincides with

Dh χi (ξ(h))(h) =

r

O(

k=0

1 1 1 1 )O( )O( r ) = O( r+2+ε ) . m1+ε m m m

Coupling (27) and (32) with (24) renders 1 O( mr+2 ) χi (h) = 1 O( mr+2+min{1,ε} )

if r ≥ 1 is odd , if r ≥ 2 is even .

(32)

(33)

Thus putting (33) into (23) and combining the latter with (20) and (22) yields

tjr

(P˙ rj (t) − γ(t)) ˙ dt =

t(j−1)r

=− =

tjr

t(j−1)r

t(j−1)r

1 < f˙(t), γ(t) ˙ > dt + O( 2r+1 ) d(γ) m

1 < f (t), γ¨ (t) > dt + O( 2r+1 ) d(γ) m

1 O( mr+2 ) 1 O( mr+2+min{1,ε} ) m

tjr


−1

r d(Pjr ), we finally obtain Hence as d( γr ) = Σj=0

d(γ) − d( γr ) =

1 O( mr+1 ) 1 O( mr+1+min{1,ε} )

This completes the proof of Theorem 4.


Denoising Images: Non-linear Leap-Frog for Shape and Light-Source Recovery Lyle Noakes1 and Ryszard Kozera2 1

The University of Western Australia School of Mathematics and Statistics 35 Stirling Highway, Crawley WA 6009, Australia 2 The University of Western Australia School of Computer Science and Software Engineering 35 Stirling Highway, Crawley WA 6009, Australia

Abstract. In 3-source photometric stereo, a Lambertian surface is illuminated from 3 known independent light-source directions, and photographed to give 3 images. The task of recovering the surface reduces to solving systems of linear equations for the gradients of a bivariate function u whose graph is the visible part of the surface [9], [16], [17], [24]. In the present paper we consider the same task, but with slightly more realistic assumptions: the photographic images are contaminated by Gaussian noise, and light-source directions may not be known. This leads to a non-quadratic optimization problem with many independent variables, compared to the quadratic problems resulting from addition of noise to the gradient of u and solved by linear methods in [6], [10], [20], [21], [22], [25]. The distinction is illustrated in Example 1 below. Perhaps the most natural way to solve our problem is by global Gradient Descent, and we compare this with the 2-dimensional Leap-Frog Algorithm [23]. For this we review some mathematical results of [23] and describe an implementation in sufficient detail to permit code to be written. Then we give examples comparing the behaviour of Leap-Frog with Gradient Descent, and explore an extension of Leap-Frog (not covered in [23]) to estimate light source directions when these are not given, as well as the reflecting surface.

1

Introduction

In shape-from-shading the task is to recover an unknown function u : Ω → IR whose graph {(x, y, u(x, y)) : (x, y) ∈ Ω} is the surface of a reflecting surface, typically a Lambertian surface (a perfect light diffuser which scatters light equally in all directions). Here Ω is a domain in the focal plane of a camera, taken as IR2 . Several photographs may be needed for the recovery of u, with different light source directions (and sources considered to be infinitely far away). Each photograph gives an image irradiance equation [9], [11] for u, of the form T. Asano et al. (Eds): Geometry, Morphology, . . . 2002, LNCS 2616, pp. 419–436, 2003. c Springer-Verlag Berlin Heidelberg 2003

420

Lyle Noakes and Ryszard Kozera

p1 ux (x, y) + p2 uy (x, y) − p3 = E(x, y) u2x (x, y) + u2y (x, y) + 1

(1)

which is a non-linear first-order partial differential equation for u. Here p = (p1 , p2 , p3 ) is a unit vector in the direction of the light source, and E : Ω → IR is the image intensity of the photograph. Equations of this form depend only on the gradient ∇u, and therefore are invariant with respect to addition of constants: u → u+c. So, no matter how many photographs we have, u is at most determined up to addition of constants. This may be called the standard ambiguity in shapefrom-shading, and there are other more subtle ambiguities, especially in single image shape-from-shading [9], [11], [12], [15]. In multiple image shape-from-shading (called photometric stereo), the problem of determining u from several images is generically well-posed, at least over the intersection of their respective domains, and modulo the standard ambiguity [9], [16] [17], [24]. The standard method is first to determine the gradient ∇u, and then integrate to recover u up to addition of constants. What complicates the task is that the intensity functions E 1 , E 2 , . . . , E m , corresponding to m > 1 photographs, may be contaminated by noise. This feeds into the estimate v of ∇u. Consequently v is usually non-integrable in the sense that it is not the gradient of any C 2 function. When the noise contamination can be assumed to be addition of uniform Gaussian noise to the gradient estimate v, the problem reduces to finding an integrable vector field nearest to v in a simple and well-defined sense. The approaches to this use linear methods [9], [10], [20], [21], [22], [25], and especially the mathematically clean and computationally effective algorithm of Frankot and Chellappa [6]. Whatever the approach, the underlying mathematical problem is one of quadratic optimization and the difficulty not so much in the mathematical algorithms as the assumption that v is contaminated by uniform Gaussian noise. Sometimes this assumption is reasonable, or at least a useful first approximation, but it seems more natural to assume noise added to the intensity functions E s of the photographs. Unfortunately this reduces to a non-convex optimization problem in a huge number of variables (depending on the resolution of the photographs). The problem simplifies when we can make a good initial guess for u. Gradient Descent is an elementary approach that works fairly well in such circumstances, but regulating step-sizes can be tricky, as in Examples 2, 3, 4, especially considering that the huge number of independent variables makes calculation of Hessians prohibitively costly. The 2-dimensional Leap-Frog Algorithm (2D Leap-Frog) [23] is an iterative scheme, resembling block-Gauss-Seidel [7], but in a non-linear setting, which breaks the local optimization into a sequence of smaller optimization problems in fewer independent variables. Alternatively, this can be seen as an extension of 1dimensional Leap-Frog [13], [19]. Solving the smaller optimization problems can done much more quickly, with a wider range of methods (for instance Hessians can be calculated as need be). This has to be traded off against the need for many small-scale optimizations. So there seems to be a need for some experimentation

Denoising Images

421

to compare the relative merits of Gradient Descent, 2-dimensional Leap-Frog, and possibly other methods. The present paper makes a start on this. In Section 2 we review 2-dimensional Leap-Frog and some mathematical results proved in [23], in sufficient detail to permit code to be written. Then in Section 3 we focus on an important special case of the general method in [23]. In Section 4, we take the opportunity to extend 2-dimensional Leap-Frog beyond what is envisaged in [23], estimating unknown light source directions as well as the reflecting surface. Note however that the latter is ill-posed and further constraints are needed to tighten the inherent ambiguities - see [1], [3], [4], [5], [8] or [26]. Some readers may wish to skip these descriptions and go directly to Section 5, where we apply Leap-Frog to some examples, including estimation of light source directions and comparisons with linear methods and Gradient Descent.

2 2.1

2-Dimensional Leap-Frog Discretization

Our pixels have side 1/M in Ω = [0, 1] × [0, 1], where M > 1 is a fixed integer. For 1 ≤ i, j ≤ M , and (xi , yj ) the centre of the pixel [(i − 1)/M, i/M ] × [(j − 1)/M, j/M ] , set uji = u(xi , yj ) and Eij = E(xi , yj ). From the central-difference approximations ux (xi , yj ) ≈ M (uji+1 − uji−1 )/2,

uy (xi , yj ) ≈ M (uj+1 − uj−1 )/2 , i i

the discrete analogue of (1) is < νîj , p >= Eij , where (M (uji+1 − uji−1 )/2, M (uj+1 − uj−1 )/2, −1) i i , νîj = j−1 2 1 + (M (uji+1 − uji−1 )/2)2 + (M (uj+1 − u )/2) i i

(2)

and 1 < i, j < M . The values of u appearing in (2) are displayed in the M × M tableau   M uM uM · · · · · · uM 2 3 M−2 uM−1 M−1 M−1   uM−1 uM−1 uM−1 · · · · · · uM−1 2 3 M−2 uM−1 uM  1   .  .. .. .. .. .. u =  .. (3)  . . . . .   2 2 2 2 2  u2  u2 u3 · · · · · · uM−2 uM−1 uM 1 u12 u13 · · · · · · u1M−2 u1M−1 with no corners. Identify the space of all such tableaux with IRM νˆ = νˆ(u), and < νîj , p >= Eij in the form f p (u) = E ,

2

−4

, write (4)

422

Lyle Noakes and Ryszard Kozera 2

2

where f p : IRM −4 → IR(M−2) is determined by the light direction p. Three light sources ps , give three images E s , and three systems of equations of the form (4). If the E s are contaminated by independent uniform mean zero Gaussian noise, let uopt be a maximum-likelihood estimate of u [27]. Then uopt minimizes s

3

f p (uopt ) − E s 2 , J (uopt ) = Σs=1

(5)

2

where · is the Euclidean norm on IR(M−2) . 2.2

Ambiguities 2

Let c ∈ IRM −4 be the table all of whose entries are a constant c, then J (u+c) = J (u). So uopt is determined at most up to constant translations. As in the continuous case, this standard ambiguity has to be resolved using additional information. More subtle ambiguities arise as follows, from the discretization 2 of u to u. For 1 ≤ k ≤ 4, define uk ∈ IRM −4 by     1 0 1 0 ··· 0 1 0 0 0 0 0 ··· 0 0 0 0 0 0 0 0 ··· 0 0 0 0 1 0 1 0 1 ··· 1 0 1 0     0 1 0 1 0 ··· 0 1 0 1 0 0 0 0 0 ··· 0 0 0 0      .. .. .. .. .. ..  .. .. .. .. .. .. . . . . . .  . . . . . . · · · .. .. ..   . . . . . . · · · .. .. ..  u1 =   u2 =   0 1 0 1 0 ··· 0 1 0 1 0 0 0 0 0 ··· 0 0 0 0     0 0 0 0 0 ··· 0 0 0 0 1 0 1 0 1 ··· 1 0 1 0     0 1 0 1 0 ··· 0 1 0 1 0 0 0 0 0 ··· 0 0 0 0 0 0 0 0 ··· 0 0 0 0 1 0 1 ··· 1 0 1     0 0 0 0 ··· 0 0 0 0 1 0 1 ··· 1 0 1 0 1 0 1 0 ··· 0 1 0 1 0 0 0 0 0 ··· 0 0 0 0     0 0 0 0 0 ··· 0 0 0 0 1 0 1 0 1 ··· 1 0 1 0      .. .. .. .. .. ..  .. .. .. .. .. .. . . . . . .  . . . . . . · · · .. .. ..   . . . . . . · · · .. .. ..  u3 =   u4 =  . 0 0 0 0 0 ··· 0 0 0 0 1 0 1 0 1 ··· 1 0 1 0     0 1 0 1 0 ··· 0 1 0 1 0 0 0 0 0 ··· 0 0 0 0     0 0 0 0 0 ··· 0 0 0 0 1 0 1 0 1 ··· 1 0 1 0 1 0 1 0 ··· 0 1 0 0 0 0 0 ··· 0 0 0 Then f p (u + 1≤k≤4 ck uk ) = f p (u), where ck ∈ IR. So J (u + ck uk ) = J (u) , 1≤k≤4

and thus uopt is at most determined up to translations by tableaux in a 42 dimensional subspace of IRM −4 . Let {p1 , p2 , p3 } be a basis of unit vectors in IR3 . The following lemma shows these are the only ambiguities (the standard ambiguity is when ck = c, for all k). 2

s

s

ˆ ∈ IRM −4 . If f p (ˆ Lemma 1. [23] Let u, u u) = f p (u) for s = 1, 2, 3, then ˆ = u + 1≤k≤4 ck uk , for some ck ∈ IR. u

Denoising Images

423

Since estimates occur in 4-parameter families, we seek the maximum-likelihood estimate uopt R which also minimizes the sum j j+1 (ui+1 − uji )2 + (ui − uji )2 , (6) JR (u) = of squared horizontal and vertical variations (corresponding to the forward differ ence continuity), for u ∈ {uopt + 4k=1 ck uk }. Because JR is quadratic in the ck , this optimization is routine, and only the standard ambiguity remains. Before this easy task, comes the problem of finding the global minimum of J , where values of u are prescribed at 4 positions in the tableau. In view of Lemma 1, let 2 J0 : U0 → [0, ∞) be the restriction of J to the space U0 ≡ IRM −8 of tableaux u satisfying u21 = u31 = u12 = u22 = 0. Perhaps surprisingly, for a given triple s of noisy images E s sufficiently close to f p (u0 ), where 1 ≤ s ≤ 3, there exists a unique global minimum of J0 over whole U0 : Lemma 2. [23] For an initial guess u00 ∈ U0 , let 1

2

3

E0 = (f p (u00 ), f p (u00 ), f p (u00 )) . For some δ1 > 0 and each triple of noisy images E satisfying E − E0 < δ1 , ˜ 0 ∈ U0 of J0 . there is exactly one non-degenerate global minimum u

3 3.1

2-Dimensional Leap-Frog: A Special Case Snapshot Optimizers

We describe 9 classes of different non-quadratic optimization problems defined locally over a given rectangular sub-array of u (a snapshot) corresponding to different locations. For k < l and M = 2l , Ω has M 2 pixels l = [(i − 1)/2l , i/2l ] × [(j − 1)/2l , j/2l ] , Sij

where 1 ≤ i, j ≤ 2l . Cover Ω also by a family of overlapping squares (snapshots) kl F kl = {Sij }1≤i,j≤2l−k+1 −1 , l , where each comprising 22k pixels Sij kl = [(i − 1)2k−l−1 , (i − 1)2k−l−1 + 2k−l ] × [(j − 1)2k−l−1 , (j − 1)2k−l−1 + 2k−l ] , Sij

and 1 ≤ i, j ≤ 2l−k+1 − 1. In Figure 1 k = l − 1 and l ≥ 2. 2l The bottom-left snapshot S11 is shown in Figure 2 where k = 2. Similarly 8 rt to (5) let J (ul ) : IR → IR be the the local performance index in ul ∈ IR8 2 variables representing u ∈ IRM −4 at pixels in Mrt (xc , yc ) (see Figure 2a) and s defined to reduce the noise in E p (for 1 ≤ s ≤ 3) over pixels from N rt (xc , yc ) (see Figure 2b). Note that J rt depends here also on fixed values v ∈ IR7 and w ∈ IR4 (see Figure 2a). Of course, for U0 the cost function J rt is naturally

424


( l −1) l S21

S31( l −1)l

S11( l −1)l (a)

(b)

(c)

S12( l −1)l

S32( l −1)l

( l −1) l S22

(d)

(e)

(f)

S13( l −1)l ( l −1) l S23

(g)

(h)

(i)

Fig. 1. Covering an image Ω by the family F (l−1)l Sij

(here k = l − 1). Each

v 07

X

u 6l

X

u 3l

X

X

X

w04

w03

v60

v50

u 7l

u 4l

u1l

X

S33( l −1)l

X

w02

u 5l

v 02

w10

u 2l

v10

S112l

(a)

=

2(l−1)

(l−1)l {Sij }1≤i,j≤3

of sub-squares

pixels

M(xrt c ,yc )

v30

u8l

X

v 04

consists of 2

kl

N(rtxc ,yc ) S112l

(b)

2l Fig. 2. (a) Free and fixed variables for the snapshot S11 without 4-pixel value 2l enforcement on u. (b) Selected pixels in S11 over which performance index J is locally minimized

Denoising Images

425

modified to J0rt : IR4 → IR in ul0 ∈ IR4 variables, with u1l , u2l , u3l , u4l set to zero in ul ∈ IR8 . In a similar fashion, the eight remaining cases (see Figure 1(bi)) yield the corresponding local performance indices J rtl : IR6 → IR, J tl : IR8 → IR, J brt : IR6 → IR, J brtl : IR4 → IR, J tlb : IR6 → IR, J br : IR8 → IR, J lbr : IR6 → IR, and J lb : IR8 → IR, respectively. 3.2

2-Dimensional Leap-Frog and Relaxation

2-dimensional Leap-Frog: In this version of Leap-Frog, snapshots are square, with half-snapshot overlaps, and a particular order of snapshots is from left to right. An initial guess u00 ∈ U0 is assumed to be given. For n = 1, 2, . . . repeat the following steps until some halting condition is flagged. kl (for k = l − 1 see Figure 3 (a)) – Start with the left bottom snapshot S11 and apply any non-linear optimization solver to J0rt with respect to1 ul 0 ∈ k k 2l IR2 (2 −2)−4 . This yields a new update un0 c ∈ IR2 −4 . – Pass now to the second snapshot (for k = l − 1 see Figure 3 (b)) of the first kl and optimize J rbl . Adjusting variables in the snapshot yields a new row S21 2l vector un0 c ∈ IR2 −4 . kl – Continue until the last snapshot S(2 l−k+1 −1)1 in the first row (for k = l − 1 tl see Figure 3 (c)) and optimize J accordingly. Adjusting variables in the 2l snapshot yields a new vector un0 c ∈ IR2 −4 . This completes the first row of nth iteration. kl – Pass to the second row. Start with the S12 snapshot and optimize J brt (for k = l − 1 see Figure 3 (d)). Adjusting variables in the snapshot yields a new 2l vector un0 c ∈ IR2 −4 . kl (a generic case) over which we optimize – Pass to the second snapshot S22 brtl (for k = l − 1 see Figure 3 (e)). Adjusting variables in the snapshot J 2l yields a new vector un0 c ∈ IR2 −4 . kl – Continue until the last snapshot S(2 l−k+1 −1)2 in the second row is reached. tlb Over this snapshot optimize J (for k = l − 1 see Figure 3 (f)). Adjusting 2l variables in the snapshot yields a new vector un0 c ∈ IR2 −4 . – Continue row by row (as specified in the previous steps), until the last row is kl reached. Now optimize J br over S12 l−k+1 −1 (for k = l − 1 see Figure 3 (g)). 2l

Adjusting variables in the snapshot yields a new vector un0 c ∈ IR2 −4 . kl – Pass to the second snapshot of the last row S22 l−k+1 −1 over which we optimize J lbr (for k = l − 1 see Figure 3 (h)). Adjusting variables in the snapshot 2l yields a new vector un0 c ∈ IR2 −4 . kl – This continues, up until the last snapshot S(2 l−k+1 −1)(2l−k+1 −1) , in the last row is reached. Over this sub-square optimize J lb for k = l − 1 (see Figure 3 2l (i)). Adjusting variables in the snapshot yields a new vector un0 c ∈ IR2 −4 . 1

More precisely, adjust the variables in the snapshot to minimize J rt

426


This completes the nth iteration, and the resulting updated global values of 2l un0 c are labelled by un0 = un0 c ∈ IR2 −4 . Relaxation: Once uopt ∈ U0 is computed with 2-dimensional Leap-Frog find uopt 0 R ∈ U according to the relaxation scheme (6). 3.3

Convergence

˜0 It remains to be shown that {un0 : n ≥ 1} converges to the global minimum u of J0 , referred to in Lemma 2. ˜ 0. Theorem 1. [23] For some θ > 0, and all E if E −E0 < θ, limn→∞ un = u This establishes convergence for the entire sequence {un0 }∞ n=1 - not just a subsequence. In addition, convergence of {un0 }∞ to the unique global minimum n=1 ˜ 0 of J0 (over U0 ) is assured provided the noisy images E s are close enough to u s the images f p (u00 ) formed by an initial guess u00 . One way of making an initial guess might be to apply methods designed for the mathematically simpler task of estimating u when uniform Gaussian noise is added to the gradient [6], [10], [20], [21], [22], [25].

4

Shape and Light-Source Direction Recovery

The case when light-source directions are unknown is not dealt with in [23], but it is not difficult to modify 2-dimensional Leap-Frog to recover both u and light source directions ps , as shown in this section. In the absence of additional data or constraints imposed on illuminated surface the problem of finding in parallel both the surface and light-source directions for multiple images of type (1) is ill-posed. In fact (see [1], [8] or [26]) the possible ambiguity in finding discrete u and light source directions is reduced to scaled orthogonal transformations B1 = λA−1 and B2 = λ−1 A−1 , where λ is a non-zero real number and A is an orthogonal ˆ = B1 n 3×3 matrix. Indeed for n = (ux , uy , −1) and p = (p1 , p2 , p3 ) define new n ˆ = B2 p, and taking into account A−T = A we have for each pixel the and p corresponding image intensities coinciding as ˆ |ˆ < n|p > = . ˆ p

n

p

n

ˆ A more thorough discussion on determining both multiple light-source directions and shape under various additional constraints can be found in [3], [4], [5] or [26]. The extension of 2-dimensional Leap-Frog in current version offers a computationally feasible scheme of finding one of possible surface and light-source directions, as follows: – Assign initial guesses for shape u00 ∈ U0 and for light-source directions p0 = (p10 , p20 , p30 ) ∈ IR9 to the current 0-iteration 2-dimensional Leap-Frog update of shape and light source-directions

Denoising Images

427

– For n = 1, 2, . . . repeat the following steps until some halting condition is flagged: and pn−1 . Perform 2-dimensional Leap1. take the current updates of un−1 0 Frog over Ω (one iteration only) with pn−1 temporarily fixed. Upon is updated termination of one iteration of 2-dimensional Leap-Frog un−1 0 to un0 . 2. optimize now (5) with respect to pn−1 (in spherical coordinates) with un0 temporarily fixed. Update pn−1 to pn . – Finally, perform the relaxation scheme (6) with un0 l and pnl fixed. Return nl uopt as final estimates of 2-dimensional Leap-Frog shape and R ∈ U and p light-source directions.

5

Examples

Numerical experiments were performed with Mathematica on a 900MHz Pentium III PC with 384Mb RAM. Mathematica’s FindMinimum was used for snapshot optimizations. The first task is to check whether 2-dimensional Leap-Frog significantly outperforms [6] when uniform Gaussian noise is added to the photographs (rather than gradient estimates). Ω = [0, 1] × [0, 1] throughout. Example 1. Figure 3 illustrates results of linear methods in 3-source photometric stereo (a), and 2-dimensional non-linear Leap-Frog (c), applied to images E s where 1 ≤ s ≤ 3. The noiseless table-mountain surface is the graph of u(x, y) = 1 25 2 2 3 (1 − tanh( 6 (4 − 6x + 3x − 6y + 3y ))). The light-source directions are √ √ √ √ π π π 3 1 sin , cos ) , (0, 0, 1), (−1 + 3, 0, 1 + 3)/(2 2), (− sin , − 2 24 2 24 24 with images E s contaminated by Gaussian noise of standard deviation 1/4. This noise is substantial, considering the amplitudes of E s . The discretization is coarse, with M = 16, and snapshot domains are squares of side 1/4. The linear reconstruction in (a) has no apparent relationship to the noiseless surface (b). Variations in height for the surface in (a) are around 100 units, compared with about 1 in (b). Although (b) is not exactly recoverable from the very noisy im ages E s , the 2-dimensional non-linear Leap-Frog estimate (c) is informative.

0.5

100

1 0

50 0 -50

15 10

0.8

-0.5

0.6

-1 0 0.2

5

10

15

10

0.4 5

0.4

5

0.5 0 -0.5 -1

0.2

0.6

5 10

0.8 10

15

(a)

(b)

15

(c)

Fig. 3. Estimates: (a) Linear. (b) Noiseless. (c) Non-linear 2D Leap-Frog

428


1

0.5

0.5

1

15

0

0 0.8

-0.5

10

0.6

-1 0 0.2

5 5

0.4 0.4

10

0.2

0.6 0.8

15

10

(a)

(b)

0.75 0.5 15 0.25 0 -0.25

0.5 0 -0.5 -1

15 10

10

5 5

5

10

5 10

15 15

(c)

(d)

Fig. 4. (a) Initial Guess. (b) Noiseless. (c) Non-linear 2D Leap-Frog. (d) Gradient Descent

So it can be well worth going to the extra trouble of solving a non-convex optimization problem in some cases where photographs are contaminated by noise. Next we compare 2-dimensional Leap-Frog with global Gradient Descent. Example 2. In Example 1, the initial guess u0 was obtained by adding uniform Gaussian noise of standard deviation 1/8 to the noiseless surface Figure 4(b), resulting in a bumpy surface (a), with only slight resemblance to (b) and the 2-dimensional Leap-Frog estimate (c). In comparison, global Gradient Descent applied to J0 over Ω with step-size 0.005 yields Figure 4d after 20 iterations. The performance index after a single iteration are J0 (u10 ) = 8.171 for 2-dimensional Leap-Frog and J0 (u10 ) = 126.758 for Gradient Descent. Then J0 decreases steadily to around J0 (u15 0 ) = 0.0509

and

J0 (u20 0 ) = 34.9927 ,

respectively. So the outcome of Gradient Descent is worse than 2-dimensional Leap-Frog in this example, by several orders of magnitude (the bump for Gradient Descent is much too high). Also, some experimentation was needed to find a suitable step-size for Gradient Descent.

Denoising Images

429

Especially for finer grids, it is prohibitively costly to calculate Hessians in order to determine step-sizes for Gradient Descent, and Newton’s Method is ruled out for similar reasons. Of course Hessians and matrix inverses can easily be worked out for the snapshot optimizations in 2-dimensional Leap-Frog. Our implementation simply used Mathematica’s default optimizer FindMinimum. Example 3. For a finer grid take M = 32. Define the bumpy surface in Figure 5(b) as a graph of u(x, y) =

1 (20f ((x, y), w1 ) − 15f ((x, y), w2 ) + 12f ((x, y), w3 )) , 16

where f (v 1 , v 2 ) = exp(−100(< v 1 − v 2 |v 1 − v 2 >), v 1 , v 2 ∈ IR2 , and w1 = (3/4, 1/2), w2 = (1/4, 1/3), and w 3 = (1/3, 4/5). Adding uniform Gausian noise with standard deviation 1/4 we obtained an initial guess in (a). This guess is so bad that very little of (b) is visible. The data comprised 3 photographic images, obtained from illuminating (b) from the 3 light source directions in Example 1, and then contaminated with uniform Gaussian noise of standard deviation 1/10. After a single iteration the performance index for 2-dimensional Leap-Frog is 41.69, and for Gradient Descent with step-size 0.01 the index is 721.585. After 20 iterations iterations of 2-dimensional Leap-Frog and 120 of Gradient Descent the performance indices are 120 J0 (u20 0 ) = 0.082 and J0 (u0 ) = 36.31 , respectively. After 30 steps of Gradient Descent with step-size 0.01 the surface estimate is as in (d). Then, after some human intervention, the step-size is taken as 0.001 for a further 90 steps, yielding a better estimate (e). By eye, (e) looks worse than the 2-dimensional Leap-Frog estimate (c), but perhaps not as bad as the discrepancy in performance indices suggests. Next we experiment with occluding boundaries ( ∇u = ∞) and discontinuities in u. Example 4. Set M = 32. Define a hemi-sphere-like function:

0.25 − (x − 0.5)2 − (y − 0.5)2 if (x, y) ∈ Ω1 , u(x, y) = 0 if (x, y) ∈ Ω \ Ω1 ,

(7)

where Ω1 = {(x, y) ∈ Ω : (x − 0.5)2 + (y − 0.5)2 ≤ (0.2)2 }. Starting with the noiseless surface forming a graph of u over Ω (see Figure 6(b)), uniform Gausian noise with standard deviation 3/20 is added to give the initial guess u00 in (a). Again the guess is so bad that very little trace of (b) is visible in (a). The data for 2-dimensional Leap-Frog are three images, obtained from illuminating (b) from three given light directions (as in Example 1), then contaminated with uniform Gaussian noise of standard deviation 1/20. A single iteration of 2-dimensional Leap-Frog gives J0 (u10 ) = 1.66, and then the value in equation (8)(i) after a total of 7 iterations. Figure 6(c) shows the corresponding computed shape estimate. Gradient Descent required human intervention, with step-sizes 0.01 (1-10 steps),

430


(el

Fig. 5. (a) Initial Guess. (b) Noiseless. (c) Non-linear 2D Leap-Frog. (d) Gradient Descent with Q = 0.01 upon 1-30 steps. (e) Gradient Descent with Q adjusted to Q = 0.001 upon 31-120 steps

Denoising Images

431

0.005 (11-30 steps), 0.001 (31-50 steps) and 0.0008 (51-180 steps). Starting with Jo(uA) = 811.787 the final performance index is shown in equation (8)(ii), (i) Jo(uL) = 0.012 and (ii) J0(uAs0)= 10.31 , and the shape estimate is shown in (d). Both estimates are good, although the one with Gradient Descent is flatter than it should be, as reflected in a much higher performance index. We next see that if an initial guess is bad then 2-dimensional Leap-Frog may reach a local minimum that is not a global minimum of JO. This is expected as Jo generically defines a non-convex optimization problem, which is only guarnateed to be convex (see Lemma 2) in some neighbourhood of global minimum.

Example 5. Take surface and light-source directions as in Example 1. Take M = 16 and snapshot domains squares of side 114. Two initial guesses are made, by adding Gaussian noise of standard deviations 112 and 2 to the ideal values of u (see Figure 7a,b). With these initial guesses 2-dimensional Leap-Frog finds two

(c)

(4

Fig. 6. (a) Initial Guess. (b) Noiseless. (c) Non-linear 2D Leap-Frog. (d) Gradient Descent after dynamically changing the step size during 180 steps

432


2 1 0 -1

5 15 2.5 0 -2.5

15 10

10 5

5

5

5

10

10

15

15

(a)

(b)

1.5 1 0.5 0

15 10

5 2.5 0 -2.5 -5

15 10 5

5 5 10

5 10 15

15

(c)

(d)

Fig. 7. Initial guess: (a) for σ = 0.5, (b) for σ = 2.0. 2D Leap-Frog estimates for initial guess: (c) with σ = 0.5, (d) with σ = 2.0 different local minima: see (c) and (d), respectively. The initial value of J0 (u10 ) = 12.8839 (or of J0 (u10 ) = 36.1033) after the first iteration of 2-dimensional LeapFrog settles down to 120 J0 (u20 0 ) = 0.00599 and J0 (u0 ) = 5.06008

upon 20 and 130 iterations, respectively.

Finally, we experiment with the algorithm in Section 4 for estimating shape and light-source directions simultaneously. Example 6. Consider the surfaces in Examples 4, 3, illuminated from the three light-source directions: p = (−0.82, −0.27, −0.5), q = (−0, 22, −0.13, −0.97), r = (0.07, 0.11, −0.99) . Set M = 32 and let the E s be contaminated by Gaussian noise of standard deviation 1/20 in each case. In each case the initial guess u00 was formed by adding

Denoising Images

433

the Gaussian noise of standard deviation 2/5 to the ideal values of u, and shown in Figures 8a, c. The initial guess for light-source directions for both surfaces was obtained by adding to the spherical angles Gaussian noise of standard deviation 1/4: p = (−0.88, −0.11, −0.46), q = (−0, 09, −0.07, −0.99), r = (0.03, 0.08, −1.0) . 2-dimensional Leap-Frog used snapshots 4 pixels square and on 20 (respectively 25) iterations gives the estimate in (b) (respectively (d)). The light source direction estimates are p = (−0.82, −0.29, −0.58), q = (−0.14, −0.23, −0.96), r = (0.01, 0.02, −1.0) , p = (−0.8, −0.28, −0.54), q = (−0.1, −0.16, −0.98), r = (0.1, 0.1, −0.99) , for the hemi-sphere-like and bumpy surface respectively. The estimates seem satisfatory.

6

Conclusions and Comments

– 2-dimensional Leap-Frog performs well in recovering shape (and light-source directions) on noisy data, and outperforms the linear methods, at least in some examples, where uniform Gaussian noise is added to photographs. – In general, 2-dimensional Leap-Frog seems fairly robust with respect to the initial guess. – 2-dimensional Leap-Frog seems to cope with the occlusion and discontinuities of the unknown surface. – Another feasible alternative is a global Gradient Descent over Ω. As experimentally tested, 2-dimensional Leap-Frog performs markedly better in terms of performance indices, and somewhat better in terms of the appearance of the reconstructed surface. We attribute this, at least in part, to the greater flexibility available to Leap-Frog in carrying out small-scale optimizations. To eliminate the influence of factors external to the adopted herein model (i.e. the Lambertian reflectance and Gaussian noise distribution) real noisy images have not been here considered. 2-dimensional Leap-Frog can evidently be applied to other reflectance maps, different noise distributions (including real images) and in fact to any optimization problem involving a huge number of parameters. These extensions pose future research tasks. – 2-dimensional Leap-Frog can be used to recover shape and light-source directions modulo ambiguities inherent in the problem. – 2-dimensional Leap-Frog is amenable to parallel computing schemes. – It would be interesting to have a more mathematical description of classes of images for which 2-dimensional Leap-Frog substantially outperforms linear methods and global Gradient Descent. Other relaxation methods might also be considered (see Par. 8.4 in [2] or Trust Region Methods Par 3.3 in [14]). Algorithms requiring estimates of Hessians (especially inversions) are, in general not well suited to this kind of applications because of the enormous size of the matrices.

434


(c)

(dl

Fig. 8. (a) Initial guess for hemi-sphere-like surface. (b) Computed surface (with light-sources unknown). (c) Initial guess for bumpy surface. (d) Computed surface (with light-sources unknown)

Acknowledgment The authors thank both referees for constructive comments and suggestions.

References [I] Belhumeur P. N., Kriegman D. J., Yuille A. L. (1999) The bas-relief ambiguity. Int. J. Comp. Vzs., 35 (1):33-44. 421, 426 [Z] Ciarlet P. G. (1989) Introductzon to Numerzcal Lznear Algebra and Optzmrzatzon. Camridge Uni. Press, Cambridge. 433 [3] Drbohlav O., Sara R. (2001) Unambiguous determination of shape £rom photometric stereo with unknown light sources. Proc. 8th Int. IEEE Conf. Crump. Vzszon, Vancouver, Canada, July 7-14, 2001, IEEE Vol. 2, 581-586. &$I,&6

Denoising Images

435

ˇ [4] Drbohlav O., Sara R. (2002) Specularities reduce ambiguities of uncalibrated photometric stereo. In: Heyden A., Sparr G., Nielsen M., Johansen P. (eds) Proc. 7th European Conf. Comp. Vision, Copenhagen, Denmark, May 28-31, 2002, Springer LNCS 2351, Vol. 2, 46–60. 421, 426 [5] Fan J., Wolff L. B. (1997) Surface curvature and shape reconstruction from unknown multiple illumination and integrability. Comp. Vis. Imag. Understanding, 65 (2):347–359. 421, 426 [6] Frankot R. T., Chellappa R. (1988) A method of enforcing integrability in shape from shading algorithms. IEEE, 10 (4):439–451. 419, 420, 426, 427 [7] Hackbush W. (1994) Iterative Solution of Large Sparse Systems of Equations. Springer, New York, Heidelberg, Berlin. 420 [8] Hayakawa H. (1994) Photometric stereo under a light source with arbitrary motion. J. Opt. Soc. Amer. A, 11 (11):3079–3089. 421, 426 [9] Horn B. K. P. (1986) Robot Vision. McGraw-Hill, New York Cambridge, MA. 419, 420 [10] Horn B. K. P. (1990) Height and gradient from shading. Int. J. Comp. Vis., 5 (1):37–75. 419, 420, 426 [11] Horn B. K. P., Brooks M. J. (1989) Shape from Shading. MIT Press, Cambridge, MA. 419, 420 [12] Hurt N. E. (1991) Mathematical methods in shape-from-shading: a review of recent results. Acta Appl. Math., 23:163–188. 420 [13] Kaya C. Y., Noakes L. (1998) A Leap-Frog Algorithm and optimal control: theoretical aspects. In: Caccetta L., Teo K. L., Siew P. F., Leung Y. H., Jennings L. S., Rehbock V. (eds) Proc. 2nd Int. Conf. Optim. Tech. Appl., Perth, Australia, July 1-3, 1998, Curtin Uni. Techology, 843–850. 420 [14] Kelly C. T. (1999) Iterative Methods for Optimization, Society for Industrial and Applied Mathematics, Philadelphia. 433 [15] Klette R., Schl¨ uns K. R., Koschan A. (1998) Computer Vision - Three Dimensional Data from Images. Springer, Singapore. 420 [16] Kozera R. (1991) Existence and uniqueness in photometric stereo. Appl. Math. Comput., 44 (1):1–104. 419, 420 [17] Kozera R. (1992) On shape recovery from two shading patterns. Int. J. Patt. Rec. Art. Intel., 6 (4):673–698. 419, 420 [18] Milnor J. (1963) Morse Theory. Princeton University Press, Princeton New Jersey. [19] Noakes L. (1999) A global algorithm for geodesics. J. Math. Australian Soc. Series A., 64:37–50. 420 [20] Noakes L., Kozera R. (2001) The 2-D Leap-Frog, noise, and digitization. In: Bertrand G., Imiya A., Klette R. (eds) Digital and Image Geometry, Springer, LNCS 2243, 352–364. 419, 420, 426 [21] Noakes L., Kozera R., Klette R. (1999) The Lawn-Mowing Algorithm for noisy gradient vector fields. In: Latecki L. J., Melter R. A., Mount D. M., Wu A. Y. (eds) Proc. SPIE Conf., Vis. Geom. VIII, Denver, USA, July 19-20, 1999, The Int. Soc. Opt. Engineering, 3811:305–316. 419, 420, 426 [22] Noakes L., Kozera R. (1999) A 2-D Leap-Frog Algorithm for optimal surface reconstruction. In: Latecki L. J., Melter R. A., Mount D. M., Wu A. Y. (eds) Proc. SPIE Conf., Vis. Geom. VIII, Denver, USA, July 19-20, 1999, The Int. Soc. Opt. Engineering, 3811:317–328. 419, 420, 426 [23] Noakes L., Kozera R. Nonlinearities and noise reduction in 3-source photometric stereo. Int. J. Math. Imag. Vis., in press. 419, 420, 421, 422, 423, 426

436


[24] Onn R., Bruckstein A. (1990) Integrability disambiguates surface recovery in two-image photometric stereo. Int. J. Comp. Vis., 5 (1):105–113. 419, 420 [25] Simchony T., Chellappa R., Shao M. (1990) Direct analytical methods for solving Poisson Equations in computer vision problems. IEEE Trans. Patttern Rec. Machine Intell., 12 (5):435–446. 419, 420, 426 [26] Yuille A. L., Snow D. (1997) Shape and albedo from multiple images using integrability. Proc. IEEE Conf. Comp. Vis. Patt. Rec., 158–164. 421, 426 [27] Zubrzycki S. (1970) Lectures in Probability Theory and Mathematical Statistics. American Elsevier Publishing Company Inc., New York. 422

Author Index

Asano, Akira . . . . . . . . . . . . . . . . . . 141 Asano, Tetsuo . . . . . . . . . . . . . . . . . . 58

Ngai, Wang Kay Jacky . . . . . . . . 346 Noakes, Lyle . . . . . . . . . . . . . . 403, 419

Baker, Patrick . . . . . . . . . . . . . . . . . 368 Barron, John . . . . . . . . . . . . . . . . . . 346 Bloch, Isabelle . . . . . . . . . . . . . . . . . 153

Obokata, Koji . . . . . . . . . . . . . . . . . . 58 Ootani, H. . . . . . . . . . . . . . . . . . . . . . 196

Chebihi, Nadia Takki . . . . . . . . . . 301 Coeurjolly, David . . . . . . . . . 101, 356 Danovaro, Emanuele . . . . . . . . . . . 386 Deguchi, Koichiro . . . . . . . . . . . . . .218 Eckhardt, Ulrich . . . . . . . . . . . . . . . 176 Flin, Frédéric . . . . . . . . . . . . . . . . . . 101 Floriani, Leila De . . . . . . . . . . . . . . 386 Fujio, Mitsuhiko . . . . . . . . . . . . . . . 141 Gimel’farb, Georgy . . . . . . . . . . . . 120 Goshen, Liran . . . . . . . . . . . . . . . . . . 72 Hasegawa, Jun-ichi . . . . . . . . . . . . . .22 Hirano, Yasushi . . . . . . . . . . . . . . . . . 22 Hirata, Tomio . . . . . . . . . . . . . . . . . . 86 Hontani, Hidekata . . . . . . . . . . . . . 218 Huang, Fay . . . . . . . . . . . . . . . . . . . . 335 Ichinose, T. . . . . . . . . . . . . . . . . . . . . 282 Imiya, A. . . . . . . . . . . . . . . . . . 196, 282 Katoh, Naoki . . . . . . . . . . . . . . . . . . . 58 Keren, Daniel . . . . . . . . . . . . . . . . . . . 72 Klette, Reinhard . . . . . . . . . . 335, 403 K¨ othe, Ullrich . . . . . . . . . . . . . . . . . 320 Kozera, Ryszard . . . . . . . . . . 403, 419 Linsen, Lars . . . . . . . . . . . . . . . . . . . . 39 Mekada, Yoshito . . . . . . . . . . . . . . . . 22 Mesmoudi, Mohammed Mostefa 386 Miyagawa, Miho . . . . . . . . . . . . . . . 141

Prautzsch, Hartmut . . . . . . . . . . . . . 39 Roerdink, Jos B.T.M. . . . . . . . . . . 165 Ronse, Christian . . . . . . . . . . . . . . . . . 1 Sadakane, Kunihiko . . . . . . . . . . . .301 Shimshoni, Ilan . . . . . . . . . . . . . . . . . 72 Sivignon, Isabelle . . . . . . . . . . . . . . 356 Spies, Hagen . . . . . . . . . . . . . . . . . . . 346 Sugihara, Kokichi . . . . . . . . . . . . . . 231 Tatara, K. . . . . . . . . . . . . . . . . . . . . . 196 Teytaud, Olivier . . . . . . . . . . . . . . . 101 Tokuyama, Takeshi . . . . . . . . 58, 301 Torii, A. . . . . . . . . . . . . . . . . . . . . . . . 282 Toriwaki, Junichiro . . . . . . . . . . . . . 22 Tougne, Laure . . . . . . . . . . . . . . . . . 101 T˘anase, Mirela . . . . . . . . . . . . . . . . .247 Veelaert, Peter . . . . . . . . . . . . . . . . .268 Veltkamp, Remco C. . . . . . . . . . . . 247 Wakazono, Y. . . . . . . . . . . . . . . . . . .282 Wei, Shou Kang . . . . . . . . . . . . . . . 335 Werman, Michael . . . . . . . . . . . . . . . 72 Yu, Linjiang . . . . . . . . . . . . . . . . . . . 120 Zeng, PeiFeng . . . . . . . . . . . . . . . . . . 86 Zhou, Dongxiao . . . . . . . . . . . . . . . .120

Geometry, Morphology, and Computational Imaging

Arabic Computational Morphology

Discrete and Computational Geometry

Discrete and computational geometry

Discrete and Computational Geometry

Combinatorial and computational geometry

Computational Geometry

Systems and Frameworks for Computational Morphology

Computational approaches to morphology and syntax

Computational Geometry - Algorithms and Applications

Computational Geometry: Algorithms and Applications

Computational Geometry: Algorithms and applications

Computational geometry: methods and applications

Computational geometry: algorithms and applications

Computational Geometry - Algorithms and Applications

Suspension Analysis and Computational Geometry

Computational Geometry. Algorithms and Applications

Computational Geometry Algorithms And Applications

Computational geometry: algorithms and applications

Imaging Beyond the Pinhole Camera (Computational Imaging and Vision)

Anatomical Imaging: Towards a New Morphology

Handbook of Computational Geometry

Computational Geometry in C

Computational Geometry: An Introduction

Nonlinear computational geometry

Math Computational Geometry

Computational Line Geometry

Computational algebraic geometry

Parallel Computational Geometry

Computational algebraic geometry

Computational geometry: An introduction

Geometry, Morphology, and Computational Imaging

Arabic Computational Morphology

Discrete and Computational Geometry

Discrete and computational geometry

Discrete and Computational Geometry

Combinatorial and computational geometry

Computational Geometry

Systems and Frameworks for Computational Morphology

Computational approaches to morphology and syntax

Computational Geometry - Algorithms and Applications

Computational Geometry: Algorithms and Applications

Computational Geometry: Algorithms and applications

Computational geometry: methods and applications

Computational geometry: algorithms and applications

Computational Geometry - Algorithms and Applications

Suspension Analysis and Computational Geometry

Computational Geometry. Algorithms and Applications

Computational Geometry Algorithms And Applications

Computational geometry: algorithms and applications

Imaging Beyond the Pinhole Camera (Computational Imaging and Vision)

Anatomical Imaging: Towards a New Morphology

Handbook of Computational Geometry

Computational Geometry in C

Computational Geometry: An Introduction

Nonlinear computational geometry

Math Computational Geometry

Computational Line Geometry

Computational algebraic geometry

Parallel Computational Geometry

Computational algebraic geometry

Computational geometry: An introduction

Recommend Documents