3D Video and Its Applications

3D Video and Its Applications Takashi Matsuyama r Shohei Nobuhara Takeshi Takai r Tony Tung 3D Video and Its Applica...

Author: Matsuyama T. | Nobuhara S. | Takai T. | Tung T.

162 downloads 1490 Views 10MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

3D Video and Its Applications

Takashi Matsuyama r Shohei Nobuhara Takeshi Takai r Tony Tung

3D Video and Its Applications

r

Takashi Matsuyama Graduate School of Informatics Kyoto University Sakyo, Kyoto, Japan

Takeshi Takai Graduate School of Informatics Kyoto University Sakyo, Kyoto, Japan

Shohei Nobuhara Graduate School of Informatics Kyoto University Sakyo, Kyoto, Japan

Tony Tung Graduate School of Informatics Kyoto University Sakyo, Kyoto, Japan

ISBN 978-1-4471-4119-8 ISBN 978-1-4471-4120-4 (eBook) DOI 10.1007/978-1-4471-4120-4 Springer London Heidelberg New York Dordrecht Library of Congress Control Number: 2012940250 © Springer-Verlag London 2012 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’s location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)

To our colleagues, students, and families

Preface

This book addresses 3D video production technologies and applications developed by our laboratory in Kyoto University, Japan, over the past ten years and more. In 1996, we started the Cooperative Distributed Vision project, where a group of network connected active cameras monitor a 3D real world scene to cooperatively detect and track people in real time. At the last stage of the project in 1999, we applied the system to synchronized multi-view video data capture to measure full 3D human shape and motion, which was then followed by the development of texture mapping methods to generate full 3D video around 2000. Since then, we have been conducting successively work to improve multi-view video capture systems in both image resolution and object movable space, implement parallel processing methods to reconstruct 3D shape and motion in real time using a PC cluster system, develop accurate 3D shape and motion reconstruction algorithms as well as high fidelity texture mapping and lighting environment estimation methods. With these 3D video production technologies, in 2002, we started to explore applications of 3D video including interactive 3D visualization, 3D content editing, and data compression methods to cultivate the world of 3D video. This book gives a comprehensive view of the state-of-the-art of 3D video production technologies and applications we developed, as well as related contemporary visual information media technologies which will help graduate students and young researchers to understand the world of 3D video. Since the employed technologies include a very wide range of technical disciplines covering real time synchronized multi-view video capture, object tracking with a group of active cameras, geometric and photometric camera calibration, parallel processing by a PC cluster system, 2D image and video processing, 3D shape and motion reconstruction, texture mapping and image rendering, lighting environment estimation, attractive 3D visualization, visual contents analysis and editing, 3D body action analysis, and data compression, we put as references books and technical survey papers on these fundamental technical areas for readers to understand background knowledge of 3D video. Although we have established technical skills and know-how for implementing multi-view video capture systems, and the quality of generated 3D video has been significantly improved with advanced technologies, a high fidelity 3D video producvii

viii

Preface

tion system and its casual usages in everyday life environments are still our future research targets. We hope this book will promote further explorations of the world of 3D video. Takashi Matsuyama Shohei Nobuhara Takeshi Takai Tony Tung

Acknowledgements

Our research activities on 3D video over the past decade have been supported by the Japanese government under several different programs: Research for the Future Program by the Japan Society for the Promotion of Science (1996–2000), Grant-inAid for Scientific Research (2001–2003) and National Project on Development of High Fidelity Digitization Software for Large-Scale and Intangible Cultural Assets (2004–2008) by the Ministry of Education, Culture, Sports, Science & Technology, and Strategic Information and Communications R&D Promotion Programme by the Ministry of Internal Affairs and Communications (2006–2008). As of 2011, we have got another Grant-in-Aid for Scientific Research (2011–2012) and a collaborative research with Nippon Telegraph and Telephone Corporation to explore further advanced 3D video technologies and applications. We are very grateful for all these supports. Since the 3D video project has long been one of the major research topics in our laboratory, many undergraduate, master and Ph.D. students, as well as faculty members and postdoctoral researchers have been engaged in the project. Some have stayed in our laboratory and contributed to this book, and others are working in universities, research institutions, and companies to develop new information technologies. Among others, Prof. Toshikazu Wada and Dr. Xiaojun Wu established foundations of 3D video production systems, Profs. Norimichi Ukita and Shinsaku Hiura implemented a real time active multi-target tracking system with a group of network connected pan-tilt-zoom cameras, Dr. Atsuto Maki, Mr. Hiromasa Yoshimoto, and Mr. Tatsuhisa Yamaguchi developed a 3D video capture system of an object moving in a wide area with a group of active cameras, and Prof. Hitoshi Habe and Dr. Lyndon Hill implemented a sophisticated 3D video coding method. Mr. Qun Shi developed a gaze estimation method from captured 3D video. Without their efforts and enthusiasm, we could not have explored such a wide range of 3D video production technologies and applications as covered in this book. Needless to say, our everyday research activities are supported by the staffs of our laboratory and university. We would like to thank all of our former and current secretaries for their devoted work on project proposal writing, budget management, travel schedule planning, conference and workshop organization, paper and homeix

x

Acknowledgements

page preparation, and so on. Their charming smiles encouraged us to attack difficult problems in research as well as management. Last, but of course not least, we would like to express our sincere appreciations to our families for their everlasting support and encouragement, especially to our wives, Mrs. Akemi Matsuyama, Yoshiko Nobuhara, Shiho Takai, and Bidda Camilla Solvang Poulsen.

Contents

1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Visual Information Media Technologies . . . . . . . . . . . . 1.2 What Is and Is Not 3D Video? . . . . . . . . . . . . . . . . . 1.3 Processing Scheme of 3D Video Production and Applications References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Part I

. . . . .

. . . . .

. . . . .

1 1 3 5 12

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

17 17 18 19 20 20 22 23 24 25 26 27 29 29 36 41 42 43

Multi-view Video Capture

2

Multi-camera Systems for 3D Video Production 2.1 Introduction . . . . . . . . . . . . . . . . . 2.1.1 Single-Camera Requirements . . . . 2.1.2 Multi-camera Requirements . . . . . 2.2 Studio Design . . . . . . . . . . . . . . . . 2.2.1 Camera Arrangement . . . . . . . . 2.2.2 Camera . . . . . . . . . . . . . . . . 2.2.3 Lens . . . . . . . . . . . . . . . . . 2.2.4 Shutter . . . . . . . . . . . . . . . . 2.2.5 Lighting . . . . . . . . . . . . . . . 2.2.6 Background . . . . . . . . . . . . . 2.2.7 Studio Implementations . . . . . . . 2.3 Camera Calibration . . . . . . . . . . . . . 2.3.1 Geometric Calibration . . . . . . . . 2.3.2 Photometric Calibration . . . . . . . 2.4 Performance Evaluation of 3D Video Studios 2.5 Conclusion . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

3

Active Camera System for Object Tracking and Multi-view Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Fundamental Requirements for Multi-view Object Observation for 3D Video Production . . . . . . . . . . . .

45 45 45 xi

xii

Contents

3.1.2 Multi-view Video Capture for a Wide Area . . . . . . . 3.2 Cell-Based Object Tracking and Multi-view Observation . . . . 3.2.1 Problem Specifications and Assumptions . . . . . . . . 3.2.2 Basic Scheme of the Cell-Based Object Tracking and Multi-view Observation . . . . . . . . . . . . . . . . . 3.2.3 Design Factors for Implementation . . . . . . . . . . . 3.2.4 Cell-Based Camera Control Scheme . . . . . . . . . . 3.3 Algorithm Implementation . . . . . . . . . . . . . . . . . . . 3.3.1 Constraints between Design Factors and Specifications 3.3.2 Studio Design Process . . . . . . . . . . . . . . . . . . 3.3.3 Cell-Based Camera Calibration . . . . . . . . . . . . . 3.3.4 Real-Time Object Tracking Algorithm . . . . . . . . . 3.4 Performance Evaluations . . . . . . . . . . . . . . . . . . . . 3.4.1 Quantitative Performance Evaluations with Synthesized Data . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Quantitative Performance Evaluation with Real Active Cameras . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Designing a System for Large Scale Sport Scenes . . . . . . . 3.5.1 Problem Specifications . . . . . . . . . . . . . . . . . 3.5.2 Camera and Cell Arrangements . . . . . . . . . . . . . 3.6 Conclusion and Future Works . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Part II 4

. . . . . .

46 50 50

. . . . . . . . .

. . . . . . . . .

51 52 54 55 56 62 63 65 65

. .

65

. . . . . .

. . . . . .

74 77 78 80 83 84

. . . .

91 91

. .

93

. . . .

93 96

3D Video Production

3D Shape Reconstruction from Multi-view Video Data . . . . . . 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Categorization of 3D Shape Reconstruction Methods for 3D Video Production . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 Visual Cues for Computing 3D Information from 2D Image(s) . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Full 3D Shape Reconstruction . . . . . . . . . . . . . . 4.2.3 Dynamic Full 3D Shape Reconstruction for 3D Video Production . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Design Factors of 3D Shape Reconstruction Algorithms . . . . 4.3.1 Photo-Consistency Evaluation . . . . . . . . . . . . . . 4.3.2 Visibility and Occlusion Handling . . . . . . . . . . . . 4.3.3 Shape Representation and Optimization . . . . . . . . . 4.4 Implementations and Performance Evaluations of 3D Shape Reconstruction Algorithms . . . . . . . . . . . . . . . . . . . 4.4.1 3D Shape from Multi-view Images . . . . . . . . . . . 4.4.2 Simultaneous 3D Shape and Motion Estimation from Multi-view Video Data by a Heterogeneous Inter-frame Mesh Deformation . . . . . . . . . . . . . . . . . . . . 4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . .

104 106 106 109 113

. . 118 . . 118

. . 131 . . 142 . . 146

Contents

5

6

3D Surface Texture Generation . . . . . . . . . . . . . . . . . . . 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.1 Texture Painting, Natural-Texture Mapping, and Texture Generation . . . . . . . . . . . . . . . . . . . . . . . . 5.1.2 Problems in Texture Generation . . . . . . . . . . . . . 5.2 Geometric Transformation Between a 3D Mesh and a 2D Image 5.3 Appearance-Based View-Independent Texture Generation . . . 5.3.1 Notation and Studio Configuration . . . . . . . . . . . 5.3.2 Generating Partial Texture Images . . . . . . . . . . . 5.3.3 Combining Partial Texture Images . . . . . . . . . . . 5.3.4 Discussions . . . . . . . . . . . . . . . . . . . . . . . 5.4 View-Dependent Vertex-Based Texture Generation . . . . . . . 5.4.1 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 5.4.2 Discussions . . . . . . . . . . . . . . . . . . . . . . . 5.5 Harmonized Texture Generation . . . . . . . . . . . . . . . . . 5.5.1 Background . . . . . . . . . . . . . . . . . . . . . . . 5.5.2 Algorithm Overview . . . . . . . . . . . . . . . . . . . 5.5.3 Mesh Optimization . . . . . . . . . . . . . . . . . . . 5.5.4 View-Dependent Texture Deformation . . . . . . . . . 5.5.5 Experimental Results . . . . . . . . . . . . . . . . . . 5.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Estimation of 3D Dynamic Lighting Environment with Reference Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Lighting Environment Estimation Methods . . . . . . . . . . . 6.2.1 Direct Methods . . . . . . . . . . . . . . . . . . . . . 6.2.2 Indirect Methods . . . . . . . . . . . . . . . . . . . . . 6.3 Problem Specifications and Basic Ideas . . . . . . . . . . . . . 6.3.1 Computational Model . . . . . . . . . . . . . . . . . . 6.3.2 3D Shape from Silhouette and 3D Light Source from Shadow . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3.3 Basic Ideas . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Algebraic Problem Formulation . . . . . . . . . . . . . . . . . 6.5 Algorithm for Estimating the 3D Distribution of Point Light Sources with the Skeleton Cube . . . . . . . . . . . . . . . . . 6.5.1 Technical Problem Specifications . . . . . . . . . . . . 6.5.2 Skeleton Cube . . . . . . . . . . . . . . . . . . . . . . 6.5.3 Lighting Environment Estimation Algorithm . . . . . . 6.6 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . 6.7 Surface Reflectance Estimation and Lighting Effects Rendering for 3D Video . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7.1 Generic Texture Generation . . . . . . . . . . . . . . . 6.7.2 Lighting Effects Rendering . . . . . . . . . . . . . . .

xiii

. . 151 . . 151 . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

153 156 158 159 160 162 164 166 169 170 172 173 173 174 177 183 186 192 193

. . . . . . .

. . . . . . .

195 195 196 196 197 198 198

. . 201 . . 203 . . 205 . . . . .

. . . . .

207 207 208 213 216

. . 220 . . 221 . . 225

xiv

Contents

6.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 Part III 3D Video Applications 7

Visualization of 3D Video . . . . . . . . . . . . . . . . . . . . . . 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 3D Video Visualization System . . . . . . . . . . . . . . . . . 7.3 Subjective Visualization by Gaze Estimation from 3D Video . . 7.3.1 3D Face Surface Reconstruction Using Symmetry Prior 7.3.2 Virtual Frontal Face Image Synthesis . . . . . . . . . . 7.3.3 Gaze Estimation Using a 3D Eyeball Model . . . . . . 7.3.4 Performance Evaluation . . . . . . . . . . . . . . . . . 7.3.5 Subjective Visualization . . . . . . . . . . . . . . . . . 7.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

233 233 234 236 238 245 246 248 252 252 254

8

Behavior Unit Model for Content-Based Representation and Edition of 3D Video . . . . . . . . . . . . . . . . . . . . . . 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Topology Dictionary . . . . . . . . . . . . . . . . . . . . . . 8.2.1 Dataset Clustering . . . . . . . . . . . . . . . . . . . 8.2.2 Markov Motion Graph . . . . . . . . . . . . . . . . . 8.3 Topology Description Using Reeb Graph . . . . . . . . . . . 8.3.1 Characterization of Surface Topology with Integrated Geodesic Distances . . . . . . . . . . . . . . . . . . 8.3.2 Construction of the Multi-resolution Reeb Graph . . . 8.3.3 Robustness . . . . . . . . . . . . . . . . . . . . . . . 8.3.4 Advantage . . . . . . . . . . . . . . . . . . . . . . . 8.4 Behavior Unit Model . . . . . . . . . . . . . . . . . . . . . 8.4.1 Feature Vector Representation . . . . . . . . . . . . . 8.4.2 Feature Vector Similarity Computation . . . . . . . . 8.4.3 Performance Evaluation . . . . . . . . . . . . . . . . 8.4.4 Data Stream Encoding . . . . . . . . . . . . . . . . . 8.4.5 Data Stream Decoding . . . . . . . . . . . . . . . . . 8.5 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.1 Behavior Unit Edition . . . . . . . . . . . . . . . . . 8.5.2 Semantic Description . . . . . . . . . . . . . . . . . 8.6 Performance Evaluations . . . . . . . . . . . . . . . . . . . 8.6.1 Topology Dictionary Stability . . . . . . . . . . . . . 8.6.2 3D Video Progressive Summarization . . . . . . . . . 8.6.3 Semantic Description . . . . . . . . . . . . . . . . . 8.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . .

. . . . . .

. . . . . .

255 255 257 259 260 264

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

264 265 267 268 270 270 271 276 278 278 280 280 282 283 283 286 288 289 291

Contents

9

xv

Model-Based Complex Kinematic Motion Estimation . . . . . . . . . 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Skin-and-Bones Model for Kinematic Motion Estimation from 3D Video . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Reliability Evaluation Methods . . . . . . . . . . . . . . . . . . . 9.3.1 Reliability Measure Based on the Surface Visibility . . . . 9.3.2 Reliability Measure Based on the Photo-Consistency . . . . 9.4 Kinematic Motion Estimation Algorithm Using the Reliability Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . 9.5.1 Quantitative Performance Evaluation with Synthesized Data 9.5.2 Qualitative Evaluations with Real Data . . . . . . . . . . . 9.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10 3D Video Encoding . . . . . . . . . . . . . . . . . . . . . . 10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . 10.1.1 Encoding 3D Visual Media into 2D Video Data 10.1.2 Problem Specification for 3D Video Encoding . 10.2 Geometry Images . . . . . . . . . . . . . . . . . . . . 10.2.1 Overview . . . . . . . . . . . . . . . . . . . . . 10.2.2 Cut Graph Definition . . . . . . . . . . . . . . 10.2.3 Parameterization . . . . . . . . . . . . . . . . . 10.2.4 Data Structure Constraints . . . . . . . . . . . . 10.3 3D Video Data Encoding . . . . . . . . . . . . . . . . 10.3.1 Resolution . . . . . . . . . . . . . . . . . . . . 10.3.2 Encoding and Decoding . . . . . . . . . . . . . 10.4 Stable Surface-Based Shape Representation . . . . . . 10.4.1 Stable Feature Extraction . . . . . . . . . . . . 10.4.2 Temporal Geodesic Consistency . . . . . . . . . 10.4.3 Stable Surface-Based Graph Construction . . . 10.5 Performance Evaluations . . . . . . . . . . . . . . . . 10.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

295 295 297 300 300 302 303 304 304 308 311 312 315 315 316 319 321 321 322 324 328 328 329 329 330 331 331 332 335 336 340

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343

Part I

Multi-view Video Capture

Drastic advances of digital technologies and the Internet in this decade have made digital (still and video) cameras ubiquitous in everyday life. Most of mobile phones, tablets and laptop PCs are equipped with cameras, and a huge amount of pictures and video streams are exchanged over the Internet every second. These days, moreover, 3D cinemas and 3D TV monitors have been put into the market and are gradually getting popular. In city areas a large number of cameras are placed for security and traffic monitoring. A variety of computer vision technologies such as auto-focusing on human faces and image stabilization against hand vibrations have been used in modern cameras. Quite recently, moreover, new types of camera with highly advanced computer vision technologies have been commercialized to enhance utilities of cameras in everyday life: a 2.5D range imaging camera for interactive video games [1] and a light-field camera for interactively shifting focused points in a recorded picture [2–4]. Since the major objectives of these cameras are to make pictures beautiful, attractive, and enjoyable, many artificial visual effects are introduced to enhance “image qualities” based on characteristics of human visual system and psychology. Hence captured image data lose quantitative physical grounds. For 3D video production, instead, cameras are used as physical sensors to conduct processing based on geometry, photometry, and dynamics, which requires geometric, photometric, and dynamical calibrations of cameras. Since their accuracies define qualities of produced 3D video data, the selection of cameras and their calibration methods are one of the most important processes in 3D video production. Moreover, since 3D video data is produced from multi-view video data captured by a group of cameras surrounding an object(s) in motion, their layout design and mutual geometric, photometric and dynamical calibrations are required in addition to individual camera calibrations. In this part, Chap. 2 discusses geometric, photometric, and dynamical design and calibration methods of multi-view camera systems including practical implementation technologies. Chapter 3 presents a novel multi-view active camera system for tracking and capturing an object moving in a wide spread area, which allows us to produce 3D video of sports like ice skating.

16

References 1. 2. 3. 4.

Microsoft: Kinect (2010) Levoy, M., Hanrahan, P.: Light field rendering. In: Proc. of ACM SIGGRAPH, pp. 31–42 (1996) Raytrix GmbH: Raytrix-R29 3D lightfield-camera (2011) Lytro: Lytro camera (2012)

Chapter 2

Multi-camera Systems for 3D Video Production

2.1 Introduction As discussed in the previous chapter, 3D video records full 3D shape, motion, and surface texture of an object in motion rather than a pair of stereo video or 2.5D range data. To produce such data, the entire 3D object surface should be captured simultaneously. The practical method for this is to employ a group of video cameras,1 place them to surround an object in motion, and reconstruct its 3D shape, motion, and surface texture from a group of multi-view video data recording partial 2D or 2.5D object views. While several advanced 3D video capture systems [5] are being developed introducing Time-Of-Flight cameras [25] and/or active-stereo cameras with structured lights to capture 2.5D range video data in addition to ordinary cameras, we do not consider such 2.5D cameras in this book and present 3D video production methods by reconstructing 3D object shape, motion, and surface texture from multi-view 2D video data. General limitations of current 3D video production technologies are: • In principle, multiple objects in motion can be captured at the same time. In practice, however, since their mutual occlusions degrade the quality of 3D video data, most of 3D video data are produced for a single object. Thus, in what follows we assume a 3D video stream of one object in motion is produced, except when we explicitly refer to multiple objects. • Since the problem of reconstructing 3D object shape, motion, and surface texture in natural environments is very difficult due to dynamically changing background objects and lighting environments, most of 3D video data are produced from multi-view video data captured in well-designed studios. As will be discussed in the next part, even though we assume a single object in motion in a well-designed studio, there remain many technical problems to be solved for producing high fidelity 3D video. 1 In

what follows, we simply refer video cameras as cameras.

T. Matsuyama et al., 3D Video and Its Applications, DOI 10.1007/978-1-4471-4120-4_2, © Springer-Verlag London 2012

17

18

2


Table 2.1 Camera parameters and their effects Parameter

Effect

Iris

The smaller the size, the deeper the depth of field, but the darker the image becomes

Gain

The smaller the gain, the less noisier, but the darker the image becomes

Shutter

The faster the shutter, the less motion blurred, but the darker the image becomes

Zoom

The smaller the zooming factor, the less deeper depth of field and the wider the field of view, but the smaller the image resolution becomes

This chapter presents and discusses requirements, design factors, and implementation methods of a multi-view camera studio for 3D video production (3D video studio, for short). The basic policy we employed is to implement 3D video studios with off-theshelf devices rather than develop specialized ones for 3D video production. This is not only to develop cost effective systems for casual usages but also to investigate essential problems in 3D video production. Thus, all devices introduced in this and the next chapters can be easily prepared to start research and development of 3D video.

2.1.1 Single-Camera Requirements Firstly, the requirements for 3D video studios can be classified into two categories: single-camera requirements and multi-camera requirements. The former include the following. 1. A camera should be kept well focused on the object during its motion. 2. Captured video data should not contain any motion blur even if the object motion is fast. 3. The dynamic range of a camera should be adjusted to lighting environments in a studio to capture color data accurately. 4. The resolution of a camera should be high enough to capture detailed object surface textures. 5. The field of view of a camera should be wide enough to capture an object in motion. To satisfy these requirements, the camera parameters should be adjusted: focus, iris (aperture size), gain, color balance, shutter speed (exposure time), zoom (focal length, or field of view), and position and orientation (pan and tilt). Table 2.1 summarizes effects by some of these parameters, which show mutual dependencies and hence trade-offs among them. For example, while closing the iris and shortening the exposure time as much as possible are useful to satisfy the requirements 1 and 2 above, very powerful lightings are required to satisfy the requirement 3. Moreover, the requirements 4 and 5 are in a trade-off relation, whose practical solution with

2.1 Introduction

19

active cameras will be given in Chap. 3. Thus, we need to find an acceptable set of the parameters by considering trade-offs between them.

2.1.2 Multi-camera Requirements While the single-camera requirements are well known and various types of knowhow have been developed in photography and cinema production, multi-camera requirements are rather unique in computer vision and some modern cinematography with multiple camera systems. They include: 1. Accurate 3D positions and viewing directions of multiple cameras should be known to integrate captured multi-view video data geometrically. 2. Multiple cameras should be accurately synchronized to integrate captured multiview video data temporally. 3. Accurate brightness and chromatic characteristics of multiple cameras should be known to integrate captured multi-view video data chromatically. 4. All object surface areas should be observed by at least two cameras to reconstruct their 3D shapes by stereo-based methods; while visual cues in a single image such as shading can be used to reconstruct 3D object shape, absolute 3D depth cannot be computed and, moreover, many assumptions which are not always valid in the real world are required. Requirements 1, 2, and 3 imply that cameras should be well calibrated geometrically and photometrically as well as synchronized. While these requirements can be satisfied in some accuracy with modern camera calibration methods, the last requirement is rather hard to satisfy. Especially for objects with loose clothes such as MAIKO and objects playing complex actions such as Yoga, it is not possible to satisfy this requirement. As will be discussed in detail in the next part, moreover, the multi-view surface observability plays a crucial role in the 3D shape and motion reconstruction (Chap. 4) and the texture generation (Chap. 5) for 3D video production. Consequently, the layout design of cameras should be done carefully to allow as many object surface areas as possible to be observed. As the first step toward 3D video production, this chapter establishes technical understandings about how we can find a feasible set of camera parameters which satisfies the above-mentioned requirements in practice. Section 2.2 first discusses design factors of 3D video studios and introduces three 3D video studios we developed. Then, Sect. 2.3 presents geometric and photometric camera calibration methods. The introduction and calibration of active cameras for tracking and capturing multi-view video of an object moving in a wide spread area will be presented in Chap. 3. The calibration or estimation of lighting environments will be presented in Chap. 6, since it has much to do with the texture generation in Chap. 5. Section 2.4 evaluates the performance of the three 3D video studios we developed, where the accuracy of the geometric camera calibration is quantitatively evaluated. Section 2.5 concludes this chapter with discussions and future works.

20

2


2.2 Studio Design Here we discuss technical problems to design a 3D video studio with static cameras. While, as will be discussed later, such system constrains the object movable space to satisfy the requirements described above, most of 3D video studios developed so far used static cameras to produce high quality 3D video. The introduction of active cameras, which cooperatively track an object and capture multi-view highresolution video, is a promising method to expand the object movable space. Since such an active multi-view video capture system should satisfy an additional requirement for dynamic camera control synchronized with the object motion, we confine ourselves to static cameras in this chapter and reserve discussions on active cameras for Chap. 3.

2.2.1 Camera Arrangement One of the most important design factors of a 3D video studio is how to determine (1) the number of cameras to be installed and (2) their spatial arrangement to achieve high 3D shape reconstruction accuracy. If we do not have any specific knowledge about the object shape or motion, or if we want to capture a variety of objects in the same studio, one reasonable solution is to employ a circular ring camera arrangement, where a group of cameras placed evenly along the ring observe the object performing actions at the ring center. We may call it a converging multi-camera arrangement. Figure 2.1 illustrates three typical multi-camera arrangements: diverging multi-camera arrangement for omni-directional image capture and parallel multi-camera arrangement for multi-baseline stereo [23, 27] and light-field modeling [15]. Then the next design factors to be specified concern the placement of the camera ring and the number of cameras installed on the ring. In [28], we pointed out: • The best observability of the object surface with a single ring camera arrangement is achieved by locating the ring at the mid-height of the target object. • Shape-from-silhouette methods for 3D shape reconstruction (Sect. 4.2.2.2) require at least nine cameras (40° spacing on the ring), and the reconstruction accuracy can be improved well by increasing the number of cameras up to 16 (23°). Even with larger number of cameras, the accuracy improvement is limited, since shape-from-silhouette methods can only reconstruct an approximated 3D shape of the object by definition (cf. “visual hull” in Sect. 4.2.2.2). • Shape-from-stereo methods require at least 14 cameras (25°) for an optimal balance between matching accuracy and depth ambiguity; the wider the baseline between a pair of stereo cameras becomes (i.e. wide-baseline stereo), the better accuracy of the depth measurement is achieved, while the harder the stereo matching becomes.

2.2 Studio Design

21

Fig. 2.1 Multi-view camera arrangements: (a) converging, (b) diverging, and (c) parallel arrangements

Fig. 2.2 Capturable area. Each camera can capture an object located in its field of view and within the depth-of-field (DoF) without blur. The capturable area of a multi-view camera system is given by the intersection of such “in-focus” areas

Hence, we conclude here that we need at least nine to 16 cameras for a 3D video studio with a single ring camera arrangement. As will be shown later, in Sect. 2.2.7, practical 3D video studios are usually equipped with ceiling cameras in addition to a camera ring(s) to increase the observability of top areas of an object. The camera arrangement constrains the object movable space to guarantee the multi-view observability of the object surface. In general, the 3D observable space of a camera can be represented as a quadrilateral pyramid formed by its projection center and bounded image plane. Thus, intuitively, with a converging multi-camera arrangement, the object movable space is confined within intersections of multiple quadrilateral pyramids (Fig. 2.2). That is, to guarantee the surface observation by at least two cameras, the object can only move in spaces where at least two quadrilaterals intersect. Similar space limitations are introduced also by focusing and zooming. They will be described later in this section. It should be noted that with enough number of cameras, all of them do not need to capture entire object images. That is, as long as all object surface areas can be observed by multiple cameras, some of cameras can capture the object partially by

22

2


Table 2.2 Categorization of video cameras Media production

Machine vision

Consumer

Cost

High

Middle to low

Middle to low

Quality

High

Middle to low

Middle to low

Data transmission

HD-SDI

IEEE1394b, 1000Base-T, CameraLink, USB3.0

USB, IEEE1394a

Synchronization

GenLock + Timecode

Trigger Signal

N/A

Lens

PL- or PV-mount

C-, CS-, or F-mount

Unchangeable

zooming up to increase image resolution. In fact, one of our studios (Studio B in Table 2.3) employed this strategy to increase image resolution.

2.2.2 Camera A large variety of commercial cameras are available in the market. They can be categorized by their application domains (Table 2.2). The first group is for professional media productions designed to achieve high-end quality: high-resolution and high color-depth. The second group is for industrial and machine vision. They are originally designed for factory automation, robot, etc., and relatively low-cost. The last group is for consumer use. They are widely available in the market, but not fully designed to interoperate with other cameras or controllers. Since 3D video studios require the synchronization of multiple cameras, consumer cameras cannot be used. The important difference between media production and machine vision cameras is twofold. The first is in their image qualities. Since media production cameras typically utilize 3CCD system, they offer full 8-bit depth for each color channel. On the other hand, most of machine vision cameras utilize 1CCD system with Bayer color filter [2], and their effective color-depth is reduced into 1/3. The second difference is in their synchronization mechanisms. While both media production and machine vision cameras accept a signal to control video capture timing, there is an important difference in the temporal structures of timing signals allowed. In the GenLock (generator lock) system for media production cameras, the signals should come regularly with a standardized interval such as 24 Hz, 29.97 Hz, etc. On the other hand, trigger systems for machine vision cameras allow signals to arrive at arbitrary timings. When selecting cameras, these two different synchronization mechanisms should be taken into account, especially when both types are employed into a 3D video studio. Note that some of machine vision cameras have yet another synchronization mechanism called “bus-sync”. It makes all cameras on the same bus synchronized automatically without providing additional signals. Other practical factors when selecting cameras are the allowable cable length and the data transmission rate between a camera and its data receiver. HD-SDI (formally SMPTE-292M) connection for media production cameras and 1000Base-T

2.2 Studio Design

23

machine vision cameras (known as “GigE Vision” cameras standardized by AIA) allow 100 m cable length. On the other hand, IEEE1394b (or FireWire800), CameraLink, and USB3.0 connections for machine vision cameras allow only 3 m to 10 m without active repeaters. Note that some non-standard long cables are available in the market. Thus, the camera selection for real time multi-view video capture should be done taking into account the physical size of the 3D video studio, the bandwidth of video data transfer, and the processing speed of computers and storage devices.

2.2.3 Lens While not discussed usually, the lens selection is very important to guarantee high quality multi-view image capture, because a lens specifies the field of view, the amount of incoming light, and the depth of field. The field of view can be computed from the physical imager size and the effective focal length of the lens. Suppose the imager size is W mm × H mm and the effective focal length is f mm. Then the horizontal and vertical field of view angles are simply given by −1 W FOVH = 2 tan , 2f (2.1) H FOVV = 2 tan−1 . 2f Imager sizes are often described by their “format”, such as “1/1.8 inch sensor”. For some historical reasons in optics, this number is equal to the diagonal size of the imager divided by 16; that is, the diagonal length of “1/1.8 inch sensor” is 1/1.8 × 16 = 8.89 mm. The amount of light recorded by an imager through a lens is denoted by Fnumber (or F-ratio, F-stop). The F -number is a dimensionless value given by the focal length divided by the effective aperture diameter of the lens. The larger the F -number, the smaller the lens opening is, and the lesser light comes in. Therefore it is better to use a lens with smaller F -number to capture brighter images of scenes under limited lighting environments. F -number also specifies the depth of field of a lens, which defines the depth range in which images can be captured without blur. A small F -number means a small depth of field. Since the physical pixel size is the finest resolvable point size in an image, blurring within this size does not introduce any effects in a captured image. This size is known as the circle of confusion, the maximum tolerable size of blurring. When a lens is focused at infinity, the farthest distance DH beyond which all object images are not blurred can be computed from the circle of confusion diameter c as follows: DH ≈

f2 , Fc

(2.2)

24

2


where f denotes the focal length and F the F -number. This distance DH is called hyperfocal distance. If the lens is focused at df < DH distance from the optic center, then the nearest and farthest distance between which all object images are not blurred are given as DN ≈

Dh df , D H + df

(2.3)

DF ≈

Dh df , D H − df

(2.4)

respectively. Hence the depth-of-field is DOF = DF − DN =

2DH df2 2 − d2 DH f

.

(2.5)

For example, let the physical pixel size be 4.4 µm × 4.4 µm, the focal length 6 mm, the F -number 1.4, and the focus distance 2.5 m. Then DH ≈ (6 mm)2 /(4.4 µm × 1.4) = 5.84 m, DN ≈ (5.84 × 2.5)/(5.84 + 2.5) = 1.75 m and DF ≈ (5.84 × 2.5)/(5.84 − 2.5) = 4.37 m. This means that when cameras are placed on a ring of 3 m radius, an object located within 1.25 m from the ring center can be captured in good focus without blurs. However, if it moves more than 1.25 m = 3 m − 1.75 m to a camera, then the image captured by that camera will not be well focused. That is, for a 3D video studio with a ring camera arrangement of radius R, the capturable area in terms of the depth of field can be approximated by the intersection of concentric circles of diameter 2R − 2DN and 2R − 2DH as illustrated by Fig. 2.2, which further constrains the movable space of an object.

2.2.4 Shutter The shutter speed controls the amount of motion blur as well as incoming light. By shortening the shutter, we can suppress the motion blur while reducing the amount of incoming light. Similarly to the discussion on the depth of field, if the object motion appears smaller than the pixel size, then the image does not include any effects of motion blur. There are two different types of shutter: global and rolling shutters. With the global shutter, all pixels in the imager start and end exposure simultaneously. In contrast, the rolling shutter makes each pixel line start exposure one by one while a captured image can be transmitted frame-wise. This introduces illusionary deformations into dynamic object images, and makes 3D video production unnecessarily harder. Therefore we suggest global shutter cameras, most of which have CCD sensors.

2.2 Studio Design

25

Fig. 2.3 Single-view and multi-view chroma-keying. (a) In single-view chroma-keying, colored reflections from the background to the object surface are occluded from the camera. On the other hand, in multi-view environment (b), colored reflections are observed from multiple cameras

2.2.5 Lighting In a 3D video studio, the camera arrangement constrains the arrangement of light sources as well as the object movable space. In general, cameras should not observe light sources directly, because strong direct lights damage captured images. While ordinary single-camera systems can avoid this problem by locating light sources next to the camera, such light source arrangement cannot be used for multi-view ring camera systems; a light source placed near by a camera is captured by other cameras. Thus, one reasonable solution is to locate light sources on the ceiling and set viewing directions of cameras so that captured images do not include the ceiling (Fig. 2.3). To facilitate such light source arrangement, 3D video studios should have enough heights as ordinary TV studios and theaters. As discussed before, to enhance multi-view image capture capabilities of a 3D video studio, the amount of incoming light to an image sensor is reduced (1) with a smaller iris to make the depth-of-field wider and the capturable area wider, and (2) with a shorter shutter speed to avoid motion blur. To compensate for these darkening effects, we should increase lighting or the sensor gain, which usually reduces the SN ratio of captured images. Typical lighting systems consist of halogen lamps, fluorescent tubes, LEDs, etc. While they have different characteristics on their initial cost, energy efficiency, life time, color, and so on, an important point for the 3D video studio design is whether it does flicker or not. In particular fluorescent tubes without inverters blink at 100 or 120 Hz (double of AC input), and make the global illumination level drift periodically. This should be avoided in a 3D video studio. Besides these continuous lighting devices, we can use lighting devices which flash synchronously to camera exposures. For example, we can use projectors as programmable lights, or strobe lights to “freeze” object images in quick motion [31]. To make full use of such dynamic lighting, well-designed synchronization controls should be developed to coordinate video capture and lighting.

26

2


Another augmentation of lighting is the introduction of structured lights [1, 26] to realize active-stereo analysis. Since high beams of structured lights may disturb human actions to be captured, infra-red structured lighting systems are used. In fact, advanced 3D video systems being developed [5] employ such active sensing devices in addition to ordinary cameras. While studio and theater lighting designs have been well studied and effective lightings are very important design factors to produce attractive visual contents, this book does not cover them except for Chap. 6, which presents a method of estimating 3D shapes, positions, and radiant intensities of distributed dynamic light sources.

2.2.6 Background As will be discussed in Chap. 4, multi-view object silhouettes are very useful for the 3D object shape reconstruction. In particular, the accurate silhouette contour extraction is very crucial, since it directly defines the accuracy of the visual hull geometry (Sect. 4.2.2.2). In fact, the visual hull is often used as the initial estimation of the 3D object surface in practical algorithms (Sect. 4.4). One straightforward solution for the silhouette extraction is to employ background subtraction or chroma-keying techniques. In the former, an object silhouette is given as the difference between a captured object image and the background image taken beforehand without any object. In the latter, on the other hand, the background with a known uniform color is prepared and an object silhouette is extracted as image regions having colors different from the background color. Both techniques are well studied and produce images in media production quality for studio setup. However, it should be noted that the chroma-keying for multi-view camera studio introduces non-negligible color bias into captured images (Fig. 2.3). That is, blue or green lights reflected from the background illuminate the object. In single-view chroma-keying, widely used for cinema and broadcast media production, this is known as “blue (or green) spill”. It appears typically only around the occluding boundary, because most of the reflected lights are occluded by the object. In 3D video studios, on the other hand, all surface areas are lit by colored reflections from the background. To avoid this color bias, we can use the gray background as used in Studios A and B in Fig. 2.4, or estimate lighting environments in a 3D video studio by such methods as presented in Chap. 6 and neutralize the illumination bias. The latter approach is left for future studies. While we do not discuss object silhouette extraction methods in this book, even with the state-of-the-art computer vision technologies, it is still not possible to achieve the perfect accuracy. Especially, when an object wears very colorful clothes like MAIKO with FURISODE, the chroma-keying does not work well and, moreover, wrinkles of her loose FURISODE are covered with soft shadows, and decorations in gold thread generate highlights. To cope with such complicated situations, ordinary 2D image processing methods alone are not enough and hence advanced

2.2 Studio Design

27

Fig. 2.4 Three 3D video studios developed at Kyoto University. The left column shows their interior scenes and the right the camera arrangements, respectively. The colored quadrilateral pyramids in the camera arrangements illustrate the projection centers and fields of view of the cameras

methods which integrate both the multi-view 2D silhouette extraction and the 3D shape reconstruction should be developed [8, 9, 13, 29, 32]. In summary, the problem of 3D video studio design can be regarded as the optimization of the object surface observability by a group of cameras, i.e. the surface coverage by multi-view images of well-focused, high spatial resolution, and high fidelity color. Since an object freely moves and performs complex actions, it is not possible to compute the optimal design analytically. Chapter 3 derives algebraic constraints in designing a 3D video studio with active cameras and analyzes their mutual dependencies to obtain a feasible solution. Finally, it should be noted that the 3D video studio design should be done based on real world physics, while the camera calibration discussed below is conducted based on a simplified algebraic model.

2.2.7 Studio Implementations Figure 2.4 and Table 2.3 show three 3D video studios and their specifications we developed so far, respectively. They were designed for different objectives. Studio A was designed to develop a 3D video studio with multi-view active cameras, which track and capture an object moving in a wide spread area. Its computational algorithm and technical details will be presented in Chap. 3.

28

2


Table 2.3 Specifications of three 3D video studios developed at Kyoto University

Feature

Studio A

Studio B

Studio C

Wide area

Accurate shape and color

Transportable

Shape

Square

Dodecagon

Rounded square

Size

10 m × 10 m 2.4 m height

6 m diameter 2.4 m height

6 m diameter 2.5 m height

Camera

high and low double rings

high and low double rings

single ring

Arrangement

with ceiling cameras



Camera

Sony DFW-VL500 ×25

Sony XCD-X710CR ×15

Pointgrey GRAS-20S4C ×16

Imager

1/3 inch 1CCD

1/3 inch 1CCD

1/1.8 inch 1CCD

Image format

VGA/RAW

XGA/RAW

UXGA/RAW

Lens

Integral 5.5 mm to 64 mm

C-mount 6 mm & 3.5 mm

C-mount 6 mm & 3.5 mm

Pan/tilt/zoom

Active (with pan/tilt unit)

Static

Static

Frame rate

12.5fps

25fps

25fps

Capture PC

25

15

2

Connection

IEEE 1394a 20 m cable

IEEE 1394a 20 m cable

IEEE 1394b 10 m cable

Datarate

3.66 MB/s

18.75 MB/s

45.78 MB/s (366 MB/s per PC)

Background

Gray plywood

Gray plywood

Green screen

Lighting

Overhead inverter fluorescent lights

Studio B was designed to produce 3D video with accurate object surface geometry and texture for digital archiving of Japanese traditional dances. Most of multiview video data used in this book were captured in this studio. Its gray static background eliminates the color bias discussed before and allows high fidelity colored surface texture generation, which is an important requirement for digital archiving, especially for colorful Japanese clothes, KIMONO. Note, however, that chromakeying with gray background often introduces errors in object silhouettes: soft shadows at small wrinkles on object clothes are captured as gray regions. To remove such errors, image segmentation and/or 3D shape reconstruction methods should employ the constraints on the connectivity of silhouette regions and the inter-viewpoint silhouette consistency [22]. Studio C was designed as a transportable 3D video studio to realize on-site 3D video capture. To minimize the studio equipments, it employs only two PCs to receive 16 UXGA video streams, and the green screen background for easier silhouette extraction.

2.3 Camera Calibration

29

2.3 Camera Calibration Following the 3D video studio design, its geometric and photometric calibrations should be done for obtaining multi-view video data usable for 3D video production.

2.3.1 Geometric Calibration 2.3.1.1 Camera Model The geometric camera calibration is the process that estimates parameters of the geometric transformation conducted by a camera, which projects a 3D point onto the 2D image plane of the camera. Figure 2.5 illustrates the camera model used in this book. Note that this pinhole camera model simplifies geometric transformations conducted by a physical camera and hence cannot represent important physical characteristics required to design a 3D video studio such as the depth of field. While closely related, therefore, the 3D video studio design and the camera calibration should be considered as separate processes. As shown in Fig. 2.5, the position of a 3D point in the scene is described by vector Wp = (x, y, z) in the world coordinate system W . Wp is transformed to the camera coordinate system C by W p C p = R Wp + T = (R | T ) , (2.6) 1 where R and T are the rotation matrix and the translation vector which describes the position and posture of the camera in the world coordinate system. Then the point Cp in the camera coordinate system is transformed to (u, v) , the ideal position in the image coordinate system without considering the lens distortion: ⎛ ⎞ ⎛ ⎛ ⎞ ⎞⎛ ⎞ u α γ u0 ku s u0 f 0 0 λ ⎝v ⎠ = A Cp = ⎝ 0 β v0 ⎠ Cp = ⎝ 0 kv v0 ⎠ ⎝ 0 f 0⎠ Cp, (2.7) 1 0 0 1 0 0 1 0 0 1 where λ is a scale parameter which normalizes the third component of the left-hand side vector to 1. By definition λ is equal to the z-value (depth) of Cp. f denotes the effective focal length of the camera in pixel. ku and kv denote the aspect ratio of the pixel, s denotes the skew parameter, and (u0 , v0 ) the intersection point of the optic axis with the image screen represented by the image coordinate system. Given (u, v) , its observed position (u , v ) , which is transformed with lens distortions, is modeled as a mapping in the normalized camera coordinates: Nx Nx

2 4 (2.8) Ny , Ny = 1 + k1 r + k2 r

30

2


Fig. 2.5 Camera model. A 3D point is first projected onto the ideal position (u, v) in the 2D image plane, and then shifted to the observed position (u , v ) by lens distortions

2

2

where r 2 = Nx + Ny . k1 and k2 are the radial distortion parameters. The normalized coordinate system is given by ⎛N ⎞ x λ ⎝Ny ⎠ = Cp. (2.9) 1 In other words, the matrix A in Eq. (2.7) of the normalized camera is the identity matrix. Finally, (u , v ) is given as ⎛N ⎞ ⎛ ⎞ x u ⎝v ⎠ = A ⎝Ny ⎠ . (2.10) 1 1 In this camera model, R and T are called extrinsic parameters. A is called the intrinsic parameter since it is independent of the camera position and posture. k1 and k2 are also independent of the extrinsic parameters, but are called lens distortion parameters in particular. The geometric calibration is a process which estimates these extrinsic, intrinsic, and lens distortion parameters by observing some reference objects in the scene.

2.3.1.2 Computational Methods for Static Camera Calibration In general, • The camera calibration should be done by placing reference objects around the 3D local area where an object to be captured in 3D video performs actions. This is because the accuracy of the camera calibration is guaranteed only around the reference objects. • The camera calibration should employ a non-linear optimization like the bundle adjustment as the final step to minimize a geometrically meaningful error metric such as the reprojection error.


31

Fig. 2.6 Planar pattern for the camera calibration. Left: observed image. Right: rectified image using estimated intrinsic and lens distortion parameters

This section introduces a practical four-step calibration procedure while any calibration procedures can be used as long as the above-mentioned points are satisfied: Step 1. Step 2. Step 3. Step 4.

Intrinsic and lens distortion parameters estimation by Zhang [33]. Extrinsic parameter calibration by 8-point algorithm [10]. Non-linear optimization (bundle adjustment). Global scale and position adjustment.

2.3.1.2.1 Intrinsic and Lens Distortion Parameters Estimation The most standard camera calibration method is a planar pattern-based method proposed by Zhang [3, 4, 33]. Given a set of planar reference 3D points whose positions on the plane are known, it estimates the camera position and posture with respect to the reference, and the intrinsic and lens distortion parameters. Figure 2.6 shows the planar pattern used for the calibration. In this method, the planar pattern defines the world coordinate system. This method, however, cannot be used in the calibration of the multi-camera system in the 3D video studio. • With the ring camera arrangement, the placement of the planar pattern is very limited to guarantee the simultaneous observation by all cameras. While a possible placement to satisfy the simultaneous multi-view observation is to place it on the floor, the accuracy of the pattern detection in observed images is degraded because the cameras can observe the plane at very shallow angles. • The placement limitation can also degrade the overall calibration accuracy; the reference plane should be placed in the object action space to guarantee the calibration accuracy. Note that a transparent planar pattern would solve these problems, while its specular surface reflections would introduce another placement limitation from lighting environments. Thus, we use Zhang’s method only for the intrinsic and lens distortion

32

2


parameter estimation, which can be done for each camera independently, and employ a multi-view extrinsic parameter estimation method at the second step. With Zhang’s method, the intrinsic parameters represented by A in Eq. (2.7) and the lens distortion parameters k1 and k2 in Eq. (2.8) are estimated. Figure 2.6 compares a captured image of the reference pattern and its rectified image with the estimated parameters.

2.3.1.2.2 Extrinsic Parameter Estimation Given the intrinsic and lens distortion parameters for each camera, we can compute the relative positions of multiple cameras by linear 8-point [10], non-linear 5-point [20], or trifocal-tensor-based algorithms [6] from 2D-to-2D point correspondences (Fig. 2.7). To implement a practical extrinsic parameter estimation method, we have to develop methods to (1) obtain accurate 2D-to-2D point correspondences, and (2) calibrate multiple cameras from the 2D-to-2D point correspondences. For (1), we can make full use of the synchronized multi-view image capture. That is, move a uniquely identifiable reference object(s) scanning the possible object action space. Then, regard reference object positions in simultaneously captured multi-view images as corresponding points. To make this method work well, feature point(s) on the reference object should be designed as view-invariant: for example, 2D chess corners or a center of a 3D sphere (Fig. 2.8). A simple solution for (2) above is to use the 8-point algorithm for estimating the relative position and posture of each camera pairs. Since the 8-point algorithm estimates only the pair-wise relative position up to a scale factor, we should determine the relative positions of all cameras by the following process. Let us consider three cameras A, B, and C as the minimal setup for multi-camera calibration. 1. Suppose we use camera A as the reference, i.e., we are going to describe positions and postures of B and C in the camera A coordinate system. 2. Estimate the relative position and posture for each pair A ↔ B, B ↔ C, and C ↔ A. Note here that we have unknown scale factors for each pair of cameras: λAB , λBC , and λCA (Eq. (2.7)). Let the relative posture and position of Y w.r.t. X be XR Y and XT Y which transforms a point Yp in the camera Y coordinate system to the X coordinate system by Xp = XR Y Yp + λXY XTY . Here we can assume |XT Y | = 1 without loss of generality. 3. Let A 0 denote the origin of the camera A coordinate system. 4. The origin of the camera B coordinate system is represented by ARB B 0 + λAB ATB = λAB ATB in the camera A coordinate system. 5. Similarly, the origin of the camera C coordinate system is represented by λAC ATC in the camera A coordinate system. 6. On the other hand, the origin of the camera C coordinate system is represented by λBC BTC in the camera B coordinate system, which is represented by λBC ARB BTC in the camera A coordinate system. Then, the origin of the camera C coordinate system is represented by λBC ARB BTC + λAB ATB .


33

Fig. 2.7 Extrinsic parameter estimation. With several known 2D-to-2D point correspondences in a pair of observed images (p1 to p1 , . . . , pn to pn ), the relative 3D position and posture of two cameras (R and T ) can be estimated up to scale

Fig. 2.8 2D-to-2D point correspondences by using chess corners (left, by [4]) and sphere centers (right)

7. By equating the above two representations of the origin of the camera C coordinate system, we can obtain the constraint for three scale factors. That is, the three coordinate systems of cameras A, B, and C are integrated into the common coordinate system with one scale factor. By iteratively applying this method for the other cameras one by one, we can describe all the camera positions and postures in the camera A coordinate system with a scale factor. Notice that this process obviously accumulates calibration errors through the iteration. However, this is not a serious problem since the following non-linear optimization will reduce these errors. From a practical point of view, we can use this extrinsic parameter calibration to verify whether or not the calibration processes and the multi-camera system are working correctly. That is, if the 8-point algorithm fails to estimate the pair-wise positions and postures, that is, if calibration errors such as sum of the reprojection errors (described in the next section) are not acceptable, then examine if 1. The corresponding point estimation may have introduced errors due to falsepositive and/or true-negative detections, or 2. The multi-camera synchronization may not be working properly to produce erroneous point correspondences. Since both the calibration and the synchronization are the most crucial requirements for 3D video production, it is highly recommended to check the calibration errors before optimizing the parameters.

34

2


2.3.1.2.3 Bundle Adjustment By the previous two steps, all calibration parameters have been estimated. One standard metric to evaluate the accuracy of the estimated parameters is the reprojection error. That is, for each corresponding point pair pik and pjk of camera Ci and Cj , compute the 3D point P k from them by triangulation, and reproject P k onto the image planes again. Let p˘ ik and p˘ jk be the reprojection of P k on the image planes of cameras Ci and Cj , respectively. Then the reprojection error is defined by E(Ci , Cj ) =

p k − p˘ k 2 + p k − p˘ k 2 . i i j j

(2.11)

k

The goal of the non-linear optimization is to minimize this error for all cameras. That is, it optimizes a set of parameters which minimizes E= E(Ci , Cj ), (2.12) Ci =Cj ∈C

where C is the set of cameras. This optimization is called the bundle adjustment; it optimizes the calibration parameters by adjusting the bundle of light rays from each camera center to its image feature points so that corresponding rays from multiple cameras intersect each other in the 3D space. In practice this non-linear optimization is done by Levenberg–Marquardt algorithm. Furthermore the sparse implementation of Levenberg–Marquardt algorithm can perform better since the Jacobian of Eq. (2.12) is significantly sparse. In addition, as pointed out by Hernandez et al. [11], modifying camera position T has very similar computational effects to shifting image center (u0 , v0 ) in particular for circular camera arrangements, and hence fixing (u0 , v0 ) through the optimization can perform better. One important point in implementing the extrinsic parameter estimation is the estimation method of P k from pik and pjk . As discussed in [10], it is not a good idea to estimate P k by the midpoint of the common perpendicular to the two rays through pik and pjk , since it is not projective-invariant. Instead, [10] suggested to use linear triangulation methods or to solve a 6-degree polynomial.

2.3.1.2.4 Global Scale and Position Adjustment The last step of the geometric calibration is to transform the world coordinate system used for the extrinsic parameter estimation into a physical one: determine the scale parameter of the common coordinate system to which all camera coordinate systems were transformed in the extrinsic parameter estimation. One simple practical method for it is to measure three points po , px , and py on the studio floor. po defines the origin of the physical coordinate system, the directions from po to px and py defines X- and Y -axes, respectively. The Z-direction is given by the cross


35

Fig. 2.9 Global scale and position adjustment using a chessboard on the floor

product of the X- and Y -directions. For example, place a chessboard designed with physical measures on the floor (Fig. 2.9). Let {Ri , Ti } (i = 1, . . . , N ) (N : number of cameras) denote the optimal extrinsic camera parameters obtained by the bundle adjustment. Then, select two cameras i and i (i = i ) which can best observe the chessboard and apply Zhang’s method [33] to estimate the rotation and translation w.r.t. the floor as Rˆ j and Tˆj (j = i , i ). The global scale parameter is given by comparing the distance between camera i and i in the two different coordinate systems. That is, λ=

|Tî − Tî | |Ti − Ti |

(2.13)

is the global scale parameter to be applied for the result of the bundle adjustment. Finally, in order to describe the camera positions and postures w.r.t. the floor, {Ri , Ti } (i = 1, . . . , N ) should be transformed to Ri = Rˆ i Ri Ri

Ti = λ Rˆ i Ri Ti − Rˆ i Ri Ti + Tî ,

(2.14)

which represent the positions and postures of cameras in the physical coordinate system. With this representation, we can easily design object actions in the 3D video studio. Note that the calibration accuracy of the above process does not affect the reconstruction accuracy of 3D object because it uniformly transforms all camera coordinate systems by a rotation and a translation. The accuracy of camera calibration in each 3D video studio we developed will be shown later.

2.3.1.3 Active Camera Calibration While all geometric parameters of static cameras are fixed, those of active cameras can be dynamically changed during video capturing. Typical controllable parameters of active cameras include pan, tilt, dolly, and zoom. While pan, tilt, and dolly controls modify only the position of the projection center geometrically, zooming changes all camera parameters including the focal length, the projection center, the lens distortion, and the image resolution, since the zoom control modifies the entire optical system configuration of a camera.

36

2


Thus from a viewpoint of camera calibration, active cameras without zooming are a reasonable class of practically usable active cameras; the camera calibration process is required to estimate the position of the projection center dynamically while the other parameters are kept fixed. In [30], we developed the fixed-viewpoint pan-tilt camera, where (1) the pan and tilt axes intersect with each other and (2) the projection center is aligned at the intersecting point. With this camera, the projection center is fixed during any pan-tilt controls, and hence it can be calibrated just as a static camera, which greatly facilitates the development of active object tracking systems to monitor 3D motion trajectories of objects [17] as well as high-resolution panoramic image capture systems. One important technical problem when employing active cameras is the synchronization between the camera control and the image capture. That is, since these two processes usually run asynchronously, some synchronization mechanisms should be introduced to associate the viewing direction of a camera with a captured image. In [16], we proposed the dynamic memory architecture to virtually synchronize asynchronous processes. With this mechanism, each captured video frame can be annotated by synchronized pan and tilt parameter values. Note that pan and tilt values obtained from the camera controller are not accurate enough to be used as calibration parameters and hence the ordinary camera calibration should be done using them as initial estimates. The calibration of active cameras, except for the fixed-viewpoint pan-tilt camera, involves many difficult technical problems including the camera model itself and hence its accuracy is limited. We will discuss them in Chap. 3 in detail.

2.3.2 Photometric Calibration A camera records light flux converging to its projection center as a 2D array of pixel intensity values. While the geometric calibration models geometric aspects of this imaging process, the light flux has photometric characteristics such as colors (i.e. wave length of light) and powers (i.e. irradiance), which are also transformed through the imaging process. The goal of the photometric calibration is to rectify the photometric transformations by a camera. Here, we consider the following two practical characteristics for the photometric calibration. Gain: The gain defines the transformation from incident light intensities to image pixel values. First of all, to use cameras as physical sensors, the γ correction should be done to make this transformation linear; most cameras transform incident light intensities nonlinearly to image pixel values to make captured images look natural on displays or printed papers. Since ordinary color cameras employ the RGB decomposition of incident light to record RGB image intensity values for each pixel, the gain is defined for each color channel. Then, the adjustment of RGB gains, which is called color balance


37

or white balance, should be done to capture high fidelity color images. Moreover, image sensor sensitivity and electronic circuit characteristics vary from camera to camera even if they are of the same type, making color calibration of multicamera systems much harder. Vignetting: Ordinary lens systems introduce vignetting: central areas of an image become brighter than peripheral areas. That is, the latter can receive less light rays compared to the former due to (1) multiple optical elements in a lens system (optical vignetting) and (2) the angle of incoming light (natural vignetting by the cosine fourth law). Compared to color calibration, vignetting elimination is rather easy if lens parameters are not dynamically changed. In multi-camera systems, each camera observes a different part of the scene from a different viewpoint. This means that lighting environments vary from camera to camera. To calibrate lighting environments in a 3D video studio, 3D distributions of light sources and inter-reflections in the studio have to be modeled. These will be discussed in Chap. 6. In this section, we assume we can prepare uniform lighting environments for the photometric calibration and present two practical photometric calibration methods for multi-camera systems: relative and absolute methods. The former normalizes photometric characteristics to be shared by all cameras, while the latter establishes their transformations to standard ones defined by reference data.

2.3.2.1 Relative Multi-camera Photometric Calibration A standard idea of gain and vignetting correction is to measure a specified point in the scene by different pixels of an image sensor by moving a camera. That is, align the point at central and then peripheral image pixels one by one, and estimate parameters of a vignetting model. Kim and Pollefeys [14] proposed a method which estimates vignetting parameters from overlapped image areas in a patch-worked panoramic image. This method suits well for mobile camera and can calibrate spatial gain bias and vignetting of single-camera systems. For multi-camera systems, we proposed an idea of object-oriented color calibration in [21]. The idea is to optimize vignetting and gain parameters of cameras to minimize observed color differences of a specified 3D object surface. The following process is applied to each color channel independently. Let p denote an identifiable point on the 3D object surface and pCi the pixel representing the projection of p on the camera Ci image plane. Then, the ideal intensity value l at pCi is transformed first by a simplified Kang-and-Weiss model [34] representing the lens vignetting: l =

1 − ar l, (1 + (r/f )2 )2

(2.15)

where r denotes the distance from the image center (u0 , v0 ) to p. f and a denote the vignetting parameters. Then the intensity is transformed by the gain adjustment

38

2


Fig. 2.10 (a) Originally captured multi-view images, (b) photometrically calibrated multi-view images. ©2009 IPSJ [22]

process as follows, assuming the γ correction has been done already: l = αl + β,

(2.16)

where α and β denote the scale and bias factors. Reversing these transformations, the ideal intensity can be estimated from the observed intensity:

(l − β)(1 + (r/f )2 )2 l = F l = . α(1 − ar)

(2.17)

Then, the goodness of the gain and vignetting parameters for p can be evaluated by

E(p) = VAR FCi ICi (pCi ) ,

(2.18)

where Ci denotes a camera which can observe p without occlusion, ICi (pCi ) the observed intensity of pCi , FCi the function defined in Eq. (2.17) for Ci , and VAR{·} the function to compute the variance. Note that p should be on a Lambertian surface because its radiance should be independent of viewing angles of Ci s. Let P denote a set of Lambertian surface points. Then, apply Levenberg– Marquardt method to estimate the optimal gain and vignetting parameters which minimize the following objective function. E= E(p). (2.19) p∈P

Figure 2.10 shows the result of the photometric calibration of multi-view images. Figure 2.11 demonstrates that photometric characteristic variations of uncalibrated cameras can introduce visible artifacts in images rendered from 3D video. Here the simplest view-independent texture generation method in Sect. 5.3 is used to demonstrate the color differences across original images. Notice that the relative photometric calibration normalizes photometric characteristics of multi-view cameras so that multi-view observations of a 3D surface point give the same pixel intensity value. Hence it does not guarantee that the calibrated color is the “true” color of the object.


39

Fig. 2.11 Textures generated from Fig. 2.10(a) and (b), respectively. The red arrows indicate texture boundaries introduced by photometric characteristics variations among the cameras. ©2009 IPSJ [22].

Fig. 2.12 Macbeth color checker. The triplet of hexadecimal values attached to each color patch denotes approximated 8-bit RGB values [24]

2.3.2.2 Absolute Multi-camera Photometric Calibration Assuming that the vignetting calibration is done, the absolute color calibration adjusts RGB color channel gains of a camera so that RGB values for reference color patterns coincide with predefined standard responses. Figure 2.12 shows a wellknown color pattern called Macbeth color checker, where each color patch is associated with predefined standard RGB values [24]. The color calibration with a standard color pattern also requires standard lighting environments: the pattern should be uniformly lit by a standard light source such as defined by ISO/IEC standards. As is well known, since RGB values denote spectral integrals, the accuracy of the above-mentioned RGB-based color calibration is limited. Thus, physics-based color calibration should be employed to attain the truly absolute color calibration: estimate spectral filtering characteristics of RGB channels from a reference pattern and a light source whose spectral radiance and radiant characteristics are known, respectively. To evaluate the usability of standard color samples, such as Munsell standard colors, in the physics-based color calibration, we measured spectral characteristics of radiance intensities of 1,016 color samples lit by a standard light, where spectral characteristics of each color sample is represented by 176 radiance intensity values from 380 nm to 730 nm with 2 nm sampling pitch. Then, we computed the major principal components. Table 2.4 shows eigen values and residual errors for 16 major principal components. From these results, we can observe that spectral characteristics of Munsell color samples can only be represented by several major spectral bases. This implies that detailed spectral characteristics of cameras and lighting en-

40 Table 2.4 Dimensionality reduction of Macbeth colors by PCA

2 # of principal component

Multi-camera Systems for 3D Video Production Eigenvalue

0

Approx. error

100.000

1

2.1544e−02

18.011

2

5.1743e−04

9.592

3

1.3787e−04

5.486

4

4.3802e−05

3.228

5

1.1529e−05

2.290

6

4.7269e−06

1.766

7

3.1202e−06

1.311

8

1.7696e−06

0.961

9

7.4854e−07

0.766

10

4.4186e−07

0.624

11

2.6615e−07

0.519

12

1.9256e−07

0.428

13

1.6722e−07

0.329

14

8.3086e−08

0.269

15

5.5517e−08

0.218

16

3.2762e−08

0.182

vironments cannot be estimated with such color samples; the dimension of spectral characteristic space is degenerated. To estimate spectral characteristics of cameras, we need to utilize additional reference measurements given by special optical systems such as spectrometer [19], multi-spectral camera [18], and hyper-spectral sensor [7]. These techniques play an important role on digital archiving of cultural assets such as ancient tapestries, statues, etc. In addition, knowledge about spectral characteristics of reference objects can help to calibrate such sensors. ISO/TR 16066:2003 [12] provides spectral color data of more than 50 thousand common objects as well as their reflectance and transmittance characteristics in order to calibrate spectral response of image sensors. While the absolute photometric calibration can be conducted for each camera independently before installation in a 3D video studio, lighting environments of the studio should be estimated to obtain calibrated RGB values. As will be discussed in Chap. 6, the lighting environment estimation itself involves difficult problems. Especially, it would be almost impossible to estimate the 3D spatial distribution of detailed spectral characteristics of lighting environments, because an object in motion disturbs lighting environments by its shadows as well as inter-reflections with the background scene. In summary, it would be a practical method for multi-camera photometric calibration to employ the relative multi-camera photometric calibration and then normalize RGB values based on the RGB responses of an absolutely calibrated camera.

2.4 Performance Evaluation of 3D Video Studios

41

Fig. 2.13 Multi-view videos captured by the three studios. Each line of the subtitles shows the title, captured length, and feature, respectively

2.4 Performance Evaluation of 3D Video Studios Figure 2.13 shows multi-view videos captured by the three studios in Kyoto University described in Sect. 2.2.7. Each of them has different features such as active tracking, complex and non-rigid object shape, complex motion, etc. They will be used as input for our 3D video production algorithm described in the following chapters.

42

2


Table 2.5 Performances of the three studios described in Sect. 2.2.7

Studio A

Studio B

Studio C

3.0

3.0

4.0

Capture space (m) Cylinder diameter Cylinder height

2.0

2.2

2.2

Effective resolution (mm/pix)

3.9

2.0

2.0

Calibration accuracy (mm)

4.3

2.4

3.4

Table 2.5 reports the performance measures achieved in the three studios. The capture spaces are approximated by cylinders where the requirements for 3D video production are satisfied. In the table, the diameters and heights of the cylinders are described. As in most of 3D video studios, the object movable space is very limited to guarantee the high object surface observability. The effective resolution implies the average physical distance between two neighboring pixels at the center of the capture space. The calibration accuracy is computed as the average 3D distance between a pair of rays from a pair of corresponding points in different views. The accuracy in 2D, that is, the reprojection errors of corresponding points are all in sub-pixel level. The lower resolution and accuracy of Studio A can be ascribed to its lower camera resolution (VGA); Studio A was developed for tracking and multi-view object observation with pan/tilt/zoom active cameras. Studio C, on the other hand, was designed to realize a wider object movable space with the almost same number of cameras as Studio B. To this end, the field of view was increased by employing a larger imager (1/1.8 inch) as well as improving the camera resolution (UXGA). With these designs, the effective resolution of Studio C attained the same level as that of Studio B, while the calibration accuracy was degraded due to its enlarged capture area. In summary, to enlarge the capture space as well as improve the effective resolution and calibration accuracy, we need to increase the number of cameras or employ active pan/tilt/zoom cameras. This problem is discussed in the next chapter.

2.5 Conclusion This chapter discussed the design factors of multi-camera systems for 3D video studios and introduced our three implementations. While advanced imaging device, computer, and computer vision technologies make it rather easy to implement 3D video studios, many problems are left in (1) camera selection and arrangements to guarantee multi-view observability of an object in motion, (2) geometric and photometric camera calibrations to realize the “seamless” integration of multi-view video data, and (3) design and calibration of lighting environments. These are crucial requirements for successful 3D video production for Part II. As noted at the beginning of this chapter, we designed and implemented 3D video studios with off-the-shelf cameras and lenses. Specially developed cameras

References

43

such as 4K and 8K cameras with professional lenses will improve the performance measures of 3D video studios shown in Table 2.5, while algorithms and technologies to solve the problems (1) and (2) above are left for future studies. The second generation of 3D video studios will employ new imaging sensors such as time-of-flight cameras or active-stereo systems to directly obtain 2.5D video data. Their calibration, synchronization, and data integration with ordinary video cameras will require the development of new technologies. Similarly, it would be another interesting augmentation of 3D video studios to introduce audio capturing devices such as microphone arrays for recording 3D acoustic environments. To integrate 3D visual and acoustic scenes, cross-media synchronization and calibration methods should be developed.

References 1. Batlle, J., Mouaddib, E., Salvi, J.: Recent progress in coded structured light as a technique to solve the correspondence problem: a survey. Pattern Recognit. 31(7), 963–982 (1998) 2. Bayer, B.E.: US Patent 3971065: Color imaging array (1976) 3. Bouguet, J.-Y.: Camera Calibration Toolbox for Matlab. http://www.vision.caltech.edu/ bouguetj/calib_doc/ 4. Bradski, G.: The OpenCV Library (2000). http://opencv.willowgarage.com 5. Virtualizing Engine. Private communication with Profs. Takeo Kanade and Yaser Sheikh, Robotics Institute, Carnegie Mellon University, PA (2011) 6. Fitzgibbon, A.W., Zisserman, A.: Automatic camera recovery for closed or open image sequences. In: Proc. of European Conference on Computer Vision, pp. 311–326 (1998) 7. Gevers, T., Stokman, H.M.G., van de Weijer, J.: Colour constancy from hyper-spectral data. In: Proc. of British Machine Vision Conference (2000) 8. Goldlüecke, B., Magnor, M.: Joint 3D-reconstruction and background separation in multiple views using graph cuts. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 683–688 (2003) 9. Guillemaut, J.Y., Hilton, A., Starck, J., Kilner, J., Grau, O.: A Bayesian framework for simultaneous matting and 3D reconstruction. In: Proc. of International Conference on 3-D Digital Imaging and Modeling, pp. 167–176 (2007) 10. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000) 11. Hernandez, C., Schmitt, F., Cipolla, R.: Silhouette coherence for camera calibration under circular motion. IEEE Trans. Pattern Anal. Mach. Intell. 29(2), 343–349 (2007) 12. ISO/TR 16066: Standard Object Colour Spectra Database for Colour Reproduction Evaluation (SOCS) (2003) 13. Ivanov, Y., Bobick, A., Liu, J.: Fast lighting independent background subtraction. Int. J. Comput. Vis. 37(2), 199–207 (2000) 14. Kim, S.J., Pollefeys, M.: Robust radiometric calibration and vignetting correction. IEEE Trans. Pattern Anal. Mach. Intell. 30(4), 562–576 (2008) 15. Levoy, M., Hanrahan, P.: Light field rendering. In: Proc. of ACM SIGGRAPH, pp. 31–42 (1996) 16. Matsuyama, T., Hiura, S., Wada, T., Murase, K., Toshioka, A.: Dynamic memory: architecture for real time integration of visual perception, camera action, and network communication. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 728–735 (2000) 17. Matsuyama, T., Ukita, N.: Real-time multitarget tracking by a cooperative distributed vision system. Proc. IEEE 90(7), 1136–1150 (2002)

44

2


18. Miyake, Y., Yokoyama, Y., Tsumura, N., Haneishi, H., Miyata, K., Hayashi, J.: Development of multiband color imaging systems for recordings of art paintings. In: Proc. of SPIE, pp. 218–225 (1998) 19. Morimoto, T., Mihashi, T., Ikeuchi, K.: Color restoration method based on spectral information using normalized cut. Int. J. Autom. Comput. 5, 226–233 (2008) 20. Nister, D.: An efficient solution to the five-point relative pose problem. IEEE Trans. Pattern Anal. Mach. Intell. 26(6), 756–770 (2004) 21. Nobuhara, S., Kimura, Y., Matsuyama, T.: Object-oriented color calibration of multiviewpoint cameras in sparse and convergent arrangement. IPSJ Trans. Comput. Vis. Appl. 2, 132–144 (2010) 22. Nobuhara, S., Tsuda, Y., Ohama, I., Matsuyama, T.: Multi-viewpoint silhouette extraction with 3D context-aware error detection, correction, and shadow suppression. IPSJ Trans. Comput. Vis. Appl. 1, 242–259 (2009) 23. Okutomi, M., Kanade, T.: A multiple-baseline stereo. IEEE Trans. Pattern Anal. Mach. Intell. 15(1), 353–363 (1993) 24. Pascale, D.: RGB coordinates of the ColorChecker (2006). http://www.babelcolor.com/ main_level/ColorChecker.htm 25. PMDTechnologies GmbH: CamCube3.0 (2010) 26. Salvi, J., Pagès, J., Batlle, J.: Pattern codification strategies in structured light systems. Pattern Recognit. 37(4), 827–849 (2004) 27. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 7–42 (2002) 28. Starck, J., Maki, A., Nobuhara, S., Hilton, A., Matsuyama, T.: The multiple-camera 3-d production studio. IEEE Trans. Circuits Syst. Video Technol. 19(6), 856–869 (2009) 29. Toyoura, M., Iiyama, M., Kakusho, K., Minoh, M.: Silhouette extraction with random pattern backgrounds for the volume intersection method. In: Proc. of International Conference on 3-D Digital Imaging and Modeling, pp. 225–232 (2007) 30. Wada, T., Matsuyama, T.: Appearance sphere: Background model for pan-tilt-zoom camera. In: Proc. of International Conference on Pattern Recognition, pp. A-718–A-722 (1996) 31. Yamaguchi, T., Wilburn, B., Ofek, E.: Video-based modeling of dynamic hair. In: Proc. of PSIVT, pp. 585–596 (2009) 32. Zeng, G., Quan, L.: Silhouette extraction from multiple images of an unknown background. In: Proc. of Asian Conference on Computer Vision, pp. 628–633 (2004) 33. Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000) 34. Zheng, Y., Yu, J., Kang, S., Lin, S., Kambhamettu, C.: Single-image vignetting correction using radial gradient symmetry. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)

Chapter 3

Active Camera System for Object Tracking and Multi-view Observation

3.1 Introduction As discussed in the previous chapter, most of 3D video studios developed so far employ a group of static cameras and hence the object’s movable space is rather strictly constrained to guarantee the high resolution well-focused multi-view object observation. This chapter presents a multi-view video capture system with a group of active cameras, which cooperatively track an object moving in a wide area to capture its high resolution well-focused multi-view video data.

3.1.1 Fundamental Requirements for Multi-view Object Observation for 3D Video Production In general, the multi-view object observation for 3D video production should satisfy the following basic requirements. Requirement 1: Accurate camera calibration, Requirement 2: Full visual coverage of the object surface, and Requirement 3: High spatial image resolution. Requirement 1 is crucial to reconstruct 3D shape, motion, and surface texture of an object in motion. It includes both geometric and photometric camera calibrations as discussed in the previous chapter. From a practical point of view, large efforts and computations are required in calibrating a large number of cameras, which, moreover, should be repeated when the cameras are misaligned due to hard object actions or are replaced due to malfunctions. Thus, it is a too naive idea to introduce a huge number of cameras to satisfy requirements 2 and 3. The second requirement means that every point on the object surface must be observed by at least two cameras to estimate its 3D position by shape-from-stereo methods. Obviously, this requirement is very hard or often impossible to satisfy for an object in action; objects performing complex actions like Yoga or wearing T. Matsuyama et al., 3D Video and Its Applications, DOI 10.1007/978-1-4471-4120-4_3, © Springer-Verlag London 2012

45

46

3


loose clothes like MAIKOs introduce heavy self-occlusions, which prevent object surface areas from being observed by cameras. Moreover, action scenes by multiple performers continuously introduce mutual occlusions, which significantly limit the observability. Note that as will be discussed in Chap. 4, 3D shape reconstruction methods can estimate, i.e. interpolate, 3D object surface shape even for those parts which are not observable from any cameras. In practice, therefore, it becomes an important design factor of 3D video studios to configure a group of cameras so that the object surface observability is maximized. The last requirement is very important to make 3D video an attractive visual media as ordinary 2D video. Besides employing high resolution cameras, zoom-up video capture can increase the spatial resolution. However, since it limits observable object surface areas, the trade-off with the second requirement must be solved in the 3D video studio design. These three requirements set the design principles of the 3D video studio development and constrain the capture space where an object can move. This is mainly because the mutually contradicting requirements 2 and 3 above should be compromised with a limited number of cameras. In fact, as shown in Tables 2.3 and 2.5, and Fig. 2.4, the capture spaces of our 3D video studios are constrained within rather small 3D spaces, which prevents 3D video production of sports actions like ice skating and gymnastics.

3.1.2 Multi-view Video Capture for a Wide Area There are two possible technical solutions to enlarge the capture space: (1) increase the number of cameras or (2) employ active cameras which track an object dynamically.

3.1.2.1 Space Partitioning with a Group of Camera Rings The former is a simple solution, but it is less cost-effective in both equipment and calibration labor. For example, suppose a 3D video studio is equipped with N static cameras arranged along a ring as discussed in Sect. 2.2.1. Its capture space is defined by the intersection among the fields of view of the cameras (Fig. 3.1(a)). Then, by doubling the camera number to generate a pair of camera rings and placing the rings as illustrated in Fig. 3.1(b), we can easily enlarge the capture space. Here, it should be noted that neighboring capture spaces should have overlapping spaces to guarantee the following. • The consistent global camera calibration; by placing reference patterns for the extrinsic camera parameter calibration in the overlapping spaces, the camera calibration within each ring can be integrated. That is, both intra-ring and inter-ring camera calibration should be done by making use of the overlapping spaces.

3.1 Introduction

47

Fig. 3.1 Capture space of multi-camera systems. (a) Capture space by the ring camera arrangement (Sect. 2.2.1). (b) Doubled capture space with a pair of camera rings

• The continuous observation of a moving object; multi-view video data captured in a ring should be seamlessly integrated with those captured in another ring based on the overlapping space between two rings. With this system architecture, however, the M times enlargement of the capture space requires at least M times more cameras, and, moreover, M times intra-ring and M(M − 1)/2 times inter-ring calibration labors, while N × (M − 1) cameras are not used for object video capture when a single object performs actions in the studio.

3.1.2.2 Object Tracking with Active Cameras The introduction of object tracking with active cameras [20] would be a more costeffective system design. That is, pan/tilt/dolly/zoom camera controls can solve the trade-off between requirements 2 and 3 even with a limited number of cameras; by controlling camera positions and postures according to the object motion, all cameras can capture multi-view object video as designed. In fact, while in 2D, camera works in cinema productions enable zoom-up video capture of a person during his/her large space motion. However, in such active multi-camera systems requirement 1 above for accurate camera calibration becomes hard to satisfy: • Since pan/tilt/dolly/zoom controls are conducted by mechanical devices, their accuracies are limited compared to the pixel level accuracy for 3D shape reconstruction and texture generation. • Since camera position and posture parameters should be obtained well synchronized with video frame capture timing, accurate synchronization mechanisms between video capture and camera control are required. Although it is not impossible to solve these problems by developing sophisticated pan/tilt/dolly/zoom mechanisms, it would be better to develop active camera calibration methods from observed video data; special devices cost much and prevent people from producing 3D video in everyday life environments.

48

3


Recall that the fixed-viewpoint pan/tilt camera described in Sect. 2.3.1.3 can be calibrated just as a static camera. However, it is hard to augment it with the zoom control; the pan and tilt axes should be adaptively shifted so that their intersecting point is aligned at the projection center, which moves depending on zooming factors dynamically. Moreover, it cannot change the viewpoint to increase the observability of the object surface. Thus we have to develop calibration methods for general pan/tilt/dolly/zoom cameras. There have been proposed two approaches to the active camera calibration: model-based and image-based calibrations. 3.1.2.2.1 Model-Based Active Camera Calibration There have been proposed advanced studies on modeling and calibration of active cameras [1, 6]. As a realistic pan/tilt/zoom unit modeling, Jain et al. [5] proposed a calibration method in which the pan and tilt axes can be located at arbitrary positions. In addition, the projection center can move according to zooming. In practice, however, it requires the exact pan/tilt/zoom values at each image acquisition timing, and only allows prefixed zooming factors. While the pan/tilt/dolly control modifies positions and postures of a camera, the zoom control modifies all of intrinsic, lens distortion, and extrinsic parameters. Moreover, complex optical mechanisms are employed to control zooming, and their accurate mathematical modeling itself is rather difficult. For example, Lavest et al. [7–9] proposed a method of zoom lens calibration based on a thick lens model rather than a pinhole camera model, and developed a 3D shape reconstruction method by zooming; since zooming shifts the projection center, multi-zooming can realize multi-viewpoint for shape-from-stereo. Sarkis et al. [15] proposed an example-based calibration method, which interpolates arbitrary intrinsic parameters from their neighboring real measurements based on a moving-least-square approximation. 3.1.2.2.2 Image-Based Active Camera Calibration The image-based active camera calibration has been widely studied as self calibration of moving cameras [2, 3, 11–14, 19] and image mosaicing [18, 21]. In [14], Pollefeys et al. proved that the calibration of the intrinsic and extrinsic parameters except the skew parameter is possible from multiple video frames captured by a moving camera. In [18], Sinha and Pollefeys proposed a fixed-viewpoint pan/tilt/zoom and lens-distortion calibration method from observed images taken under different pan/tilt/zoom parameters. One important assumption used in these approaches is that there exist a sufficient number of static feature points trackable over multiple frames or static overlapping areas across images taken under different pan/tilt/zoom parameters. In 3D video studio environments, these static features or image areas should be taken from the background scene, since the object is moving. However, we are faced with the following problems.

3.1 Introduction

49

• While the off-line calibration can use well designed and uniformly placed markers, the active calibration should extract them dynamically from the background scene. As discussed in Sect. 2.2.6, however, the background of 3D video studios should be made as uniform as possible to facilitate object silhouette extraction as well as lighting environment estimation. • As pointed out in the previous chapter, the calibration should be done by minimizing reprojection errors in the local area where the object surface exists. Calibration using background features does not satisfy this requirement and hence will introduce larger reprojection errors on the foreground object surface. • The main task of active cameras installed in a 3D video studio is to capture high resolution multi-view object images zooming up the object. As a result, background areas in captured images are limited due to occlusion by the object. Thus it is hard to observe a sufficient number of uniformly distributing background features for calibration. One idea to avoid these problems is to employ features on the object surface for the camera calibration. That is, first background features provide an initial guess of calibration parameters for each active camera, and then object surface features are used to determine 2D-to-2D correspondences across cameras. Finally reprojection errors at such 2D-to-2D correspondences are minimized by a non-linear optimization. While we developed a pan/tilt camera calibration method based on this idea, its reprojection errors stayed more than two times larger than those by the off-line calibration. The reasons for this are: • In the off-line calibration, positions of feature points can be estimated at subpixel accuracy based on the knowledge about calibration patterns like a regular chessboard or a sphere, which, moreover, increases the accuracy of their matching across cameras. • Texture patterns on the object surface are not known and their geometry is not planar, i.e. not affine invariant in general. As a consequence, the accuracy of their detection and matching across cameras is limited. • While calibration patterns can be placed to uniformly cover the capture space, the distribution of object surface features is usually biased. In short, whichever of the model-based or the image-based methods are employed, it is hard to make their calibration accuracy comparable to the off-line calibration.

3.1.2.2.3 Requirement for Camera Control Besides the camera calibration, the most distinguishing problem in employing active cameras for 3D video studios is that the following new requirement should be satisfied. Requirement 4: Track a moving object in real time while satisfying requirements 2 and 3.

50

3


In [20], we developed a cooperative multi-target tracking system with a group of active cameras to detect, track, and compute 3D object motion trajectories in a room. However, since its objective was the multi-target tracking, the accuracy of the camera calibration was limited and, moreover, requirements 2 and 3 were not taken into account in the system design. Hence, it cannot capture multi-view object video data usable for 3D video production. As noted in Sect. 2.3.1.3, the active camera control requires sophisticated realtime processing methods to continuously track an object whose motion is not know a priori. They include fast computation, prediction of the object motion, modeling of active camera dynamics, and real-time coordination among multiple cameras. In other words, the 3D video studio design with active cameras requires integrated analysis methods based on not only geometry and photometry but also dynamics, which we believe is a challenging problem to cut open a new frontier of computer vision technologies. In summary, while the multi-view video capture for a wide area space will exploit new applications of 3D video in sports like ice skating and gymnastics, its realization is a challenging issue. This chapter introduces a novel idea named cell-based object tracking and presents a 3D video studio with active cameras which can capture high resolution well-focused multi-view video data of an object moving in a wide area. The novelty of the idea rests in the integration of both space partitioning with a group of camera rings and object tracking with active cameras to satisfy all four requirements. Section 3.2 introduces the concept of the cell-based object tracking, followed by the practical algorithm of the cell-based multi-view video capture for 3D video production in Sect. 3.3. We derive algebraic constraints on studio design factors to satisfy the four basic requirements specified before. Section 3.4 evaluates the performance of the algorithm with synthesized and real data. To prove the practical utility of the cell-based object tracking in the real world, Sect. 3.5 designs a multi-view video capture system for 3D video production of ice skating. Sect. 3.6 concludes the chapter with future studies.

3.2 Cell-Based Object Tracking and Multi-view Observation 3.2.1 Problem Specifications and Assumptions The following are problem specifications and assumptions employed: 1. A single object of roughly known 3D size moves freely on the flat floor in the scene of a known size. Note that the flat floor assumption is used just for simplicity and the presented algorithm can be applied to a flying object that moves freely in the 3D space. In this chapter, we assume that the 3D shape of the object can be modeled by its bounding cylinder of radius r and height h. 2. No a priori knowledge about the object motion except its maximum velocity is given.

3.2 Cell-Based Object Tracking and Multi-view Observation

51

3. Requirement 3 about image resolution is specified by the lowest allowable resolution. 4. The active cameras employed are PTZ (pan/tilt/zoom) cameras without the dolly control. Thus the movements of their projection centers are limited. 5. The cameras are arranged to surround the scene uniformly, which guarantees the high observability specified by requirement 2. The distance of the camera arrangement from the scene and the zoom control are mutually dependent and will be designed by the studio design algorithm to satisfy requirement 3. 6. The developed cell-based object tracking algorithm enables the active cameras to track the object continuously and capture well-calibrated multi-view video data satisfying requirements 1 and 4, in addition to requirements 2 and 3.

3.2.2 Basic Scheme of the Cell-Based Object Tracking and Multi-view Observation In [22], we proposed a novel idea of the cell-based object tracking and multi-view observation for 3D video production. Its objective is to capture high resolution wellfocused multi-view video data of an object moving in a wide area with a limited number of active cameras. Figure 3.2 illustrates its basic scheme. Step 1: Cell formation Partition the object movable space into a set of regular disjoint subspaces named cells (cf. Fig. 3.2(a)). Step 2: Camera parameter determination For each camera, determine its pan/ tilt/zoom parameters to best observe each cell (cf. Fig. 3.2(b)). That is, the cameras do not observe the object but the cells where the object moves. Step 3: Camera calibration With the pan/tilt/zoom parameters for each cell, conduct the static camera calibration using a reference pattern located in each cell (cf. Fig. 3.2(c)). Step 4: Object tracking While tracking and capturing multi-view video of an object, some cameras observe the object in cell A with the pan/tilt/zoom parameters predetermined for cell A, and others observe its neighboring cell(s) B with the pan/tilt/zoom parameters predetermined for cell B. Depending on the object motion, cameras for the multi-view object observation are dynamically switched from one group to another (Fig. 3.2(d)). The ideas behind this scheme are: • The calibration of each active camera is conducted by fixing pan/tilt/zoom parameters for each cell, which allows us to employ ordinary calibration methods for static cameras with high accuracy. Thus, requirements 1, 2, and 3 can be satisfied by assigning an appropriate group of cameras to each cell. • The dynamic control of each active camera is conducted on a cell-by-cell basis. That is, it can be done by computing which cell each camera should observe now and next rather than tracking the object continuously. Thus, requirement 4 can be satisfied together with requirements 1, 2, and 3.

52

3


Fig. 3.2 Basic scheme of the cell-based object tracking and multi-view observation

• Video data captured during camera motions are not used for 3D video production, because they are not calibrated. That is, only when an active camera fixes its gaze to a cell with the object stably, its video data become calibrated and are used for 3D video production.

3.2.3 Design Factors for Implementation To implement a practical algorithm, the following three design factors should be fixed. Space partitioning and cell arrangement: When an approximate object motion trajectory is known a priori, cells should be arranged to cover the trajectory. The system presented in [22] used this knowledge and aligned a sequence of cells along the trajectory. Without such knowledge, here, it is reasonable to partition

3.2 Cell-Based Object Tracking and Multi-view Observation

53

Fig. 3.3 Hexagonal cell partitioning and cell-based camera control rule (see text)

the object movable 2D floor using a regular tessellation. Among others, the hexagonal tessellation realizes the most isotropic cell shape, which enables us to control active cameras isotropically. Moreover, the number of cells connected at a corner is three in the hexagonal tessellation whereas those in the square and triangular tessellations are four and six, respectively. Hence, it facilitates the cellbased camera control, as will be discussed below. It should be noted here that we do not need to introduce any overlapping areas between hexagonal cells: compare Figs. 3.1 and 3.3. Even without overlapping areas, the cell-based camera control realizes the continuous object observation, which will be explained in the next section. Camera arrangement: Since the object can move freely on the floor and each camera should observe each cell in higher resolution than specified, a camera ring surrounding the entire floor is a reasonable design of the basic camera arrangement (Fig. 3.4). Recall that the camera ring is usually augmented with a group of ceiling cameras as shown in Fig. 2.4. As will be discussed later, the practical camera arrangement should be designed based on controllable ranges of pan/tilt/zoom parameters as well as the cell size, the floor size, and the object maximum velocity. Cell-based camera control: As discussed before, when the object is in cell A as illustrated in Fig. 3.3 and observed by a group of cameras assigned to cell A, the other camera groups should become ready to observe the cells adjacent to A to ensure the continuous multi-view object observation. If this condition is always satisfied, the continuous multi-view object observation is achieved even if the cells do not have any overlaps. In other words, when the object crosses a cell border, the object is observed by a pair of camera groups which are assigned to a pair of neighboring cells sharing the border; the field of view of each camera is set wide enough to cover its assigned cell, as will be described later. Thus, the cell-based camera control can be realized by solving a camera-to-cell assignment problem. The detailed discussions on this problem are given below, since the cell-based camera control should be designed to satisfy all four requirements specified before and has much to do with the above two design factors.

54

3


Fig. 3.4 Cell groups and their corresponding camera groups (see text)

3.2.4 Cell-Based Camera Control Scheme Since an arbitrary object movement can be expressed by a movement from the current cell to one of its six neighboring cells, a naive camera-to-cell assignment method would be to partition cameras into seven groups and assign a group to a cell where the object exists, say cell A in Fig. 3.3 and the other six to its neighboring cells B, C, D, E, F , and G. Then, when the object moves into cell D next, three camera groups assigned to cells C, E, and G are controlled to observe H , I , and J , respectively. This camera-to-cell assignment, however, is not cost-effective, because 6/7 of the installed cameras are not used for the object observation. If the object trajectory is known a priori, then it can be represented by a sequence of cells. In this case, since each cell has only two neighboring cells, the set of cameras can be partitioned into two groups: one group observes the cell with the object and the other group is controlled to observe one of the neighboring cell depending on which of the neighboring cells the object is heading to. With this camera-to-cell assignment, the number of idle cameras can be reduced to 1/2. In [22], we developed a more sophisticated camera control method which maximizes the object observability and reduces the idle cameras. Even without knowledge about the object trajectory, we can reduce the number of the idle cameras by assuming the cameras can be controlled from one cell to another faster than the object motion. Suppose the object in cell A is approaching to the corner shared with cells B and C as illustrated with x in Fig. 3.3. Then, a pair of camera groups should be assigned to cells B and C to prepare for the object

3.3 Algorithm Implementation

55

observation. If the object changes its motion and approaches the corner shared with cells B and D as illustrated with x in Fig. 3.3, then the camera group assigned to C should be controlled to observe from cell C to D before the object crosses the border of cell D. With such quickly controllable cameras, the following camera-tocell assignment reduces the number of idle cameras to 2/3. Note that, as will be proven later in Sect. 3.4.1.1, the average number of idle cameras is much smaller than 2/3 in practice, because the field of view of each camera is set wide enough to cover its assigned cell and as a result, each camera can observe some parts of cells neighboring to its assigned cell. Figure 3.4 illustrates the proposed camera-to-cell assignment. The set of hexagonal cells are partitioned into three disjoint cell groups so that three cells sharing a corner should belong to different cell groups. The set of cameras are also partitioned into three groups, and the one-to-one correspondence is established between the cell groups and the camera groups so that each camera group observes the cells in its corresponding cell group by dynamically controlling pan/tilt/zoom parameters depending on the object motion. Three cells sharing a corner are given a unique cell cluster ID, within which, as shown in Fig. 3.4, each cell is assigned a unique cell ID, i, j , where i denotes the cell cluster ID and j the camera group ID. The camera control is conducted as follows. Suppose the object is in cell 4, 1 in Fig. 3.4 and camera group #1 is capturing its multi-view video. Then, depending on which corner is the closest, camera groups #2 and #3 are controlled to observe cells sharing that closest corner with cell 4, 1. Thus, if the object is wandering in cell 4, 1, camera group #2 may change the target cell among cells 1, 2, 2, 2, and 4, 2, and camera group #3 among 1, 3, 4, 3, and 3, 3, respectively. This corner based camera control also validates the hexagonal tessellation, because in the square and triangular tessellations, four and six cells would share a corner, respectively, hence increasing the number of idle cameras. In summary, with our cell-based object tracking and multi-view observation scheme, by increasing the number of cameras three times, the high resolution wellfocused multi-view object observation can be realized even if the object moves freely in a wide spread area.

3.3 Algorithm Implementation To implement a practical algorithm for the cell-base object tracking and multiview observation presented above, we should design the camera arrangement, the cell size, the calibration method, and the real-time tracking algorithm. This section presents the design of their practical implementations based on the specifications: the approximate object size, its maximum speed, the entire 3D scene space, and the minimum allowable image resolution.

56

3


3.3.1 Constraints between Design Factors and Specifications Suppose the camera ring is arranged to surround the scene as shown in Fig. 3.4 and its radius is given. The cell radius R is designed based on (1) the maximum object speed, (2) the camera control speed, and (3) the minimum allowable image resolution required for 3D video production. As illustrated in Fig. 3.3, intuitively, the cell radius R should be large enough to ensure that the camera group observing cell C can be switched to cell D before the object at x arrives at D. On the other hand, R should be small enough to ensure the object observation to be in more than the minimum allowable image resolution. This section first gives intuitive descriptions of how the design factors and the specifications are related, and then derives algebraic constraints on them.

3.3.1.1 Camera Control Rule and Cell Size As the most fundamental constraint, here we derive the constraint on the cell radius R by the maximum object speed and the camera control speed, and explain how the design factors and the specifications are related qualitatively. Figure 3.5 illustrates the worst case which requires the quickest switching from one cell to another. Suppose the object is at p in cell 1, j and is going straight to q in cell 3, j . To catch up with this object motion, the camera group j observing cell 1, j should be switched to cell 3, j when the object crosses at the midpoint of p and q, since the corner point q becomes the closest corner point from the object. Then, the pan/tilt/zoom parameters of the camera group j , which were adjusted for cell 1, j , should be changed and fixed to those for cell 3, j before the object arrives at q. Let v [m/sec] denote the maximum object velocity and τ [sec] the maximum allowable control time for a camera to switch its pan/tilt/zoom parameters from one cell to another. Then, R should satisfy R ≥ 2τ v.

(3.1)

This equation implies that the maximum image resolution is achieved when R = 2τ v,

(3.2)

since the smaller the cell is, the larger the appearance of the object in the image becomes. If the maximum possible image resolution given above does not satisfy the specification, we should modify the specifications by slowing down the object motion, reducing the minimum allowable image resolution, employing a faster pan/tilt/zoom camera device, or enlarging the radius of the camera ring. The dashed thick color lines in Fig. 3.5 denote rule borders, at which the camera control from√ a cell to another is triggered. They form another hexagonal tessellation with radius 3R. The color hatched areas of width τ v in Fig. 3.5 illustrate the controllability constraint of the active cameras. That is, when the red hatched area


57

Fig. 3.5 Designing the camera control rule for camera group j and the cell radius R. Cells are represented by black and gray solid lines. Dashed thick color lines denote rule borders and colored hatched areas controllability constraints of active cameras (see text). ©2010 IPSJ [22]

does not overlap with cell 1, j , the camera group j can be switched to that cell before the object comes in from other neighboring j group cells. The same condition holds for the blue hatched area. The width of such hatched areas, i.e. τ v, can be reduced with fast controllable active cameras or enlarging the radius of the camera ring. When active cameras are equipped with a large zooming function to observe cells from rather far away, the radius of the camera ring can be increased to reduce τ . This is because the camera angle (i.e. pan/tilt parameters) to be changed for the cell switching is reduced when the distance between the camera and cells becomes large. In addition, a pair of cells between which a camera is switched can be observed with almost same zooming factors from a distant camera, and the time for its zoom control is also reduced. Note that strictly speaking, the camera control time varies depending on a camera position as well as a pair of cells between which its pan/tilt/zoom parameters are switched. That is, the widths of the hatched areas illustrated in Fig. 3.5 vary from cell to cell as well as from rule border to rule border. To disentangle the mutual dependencies among the design factors and the specifications, and to determine R to satisfy the specifications, the following sections derive algebraic constraints on the possible range of R, which are categorized into (1) image observation constraints and (2) camera control constraints. The former are derived to guarantee sufficiently high resolution object observation and the latter to realize sufficiently fast camera control for the cell switching. Note that as will be discussed below, the distance from a camera to a cell, d, and R are tightly coupled with each other and hence, most of the constraints include both. In practice, d can be determined based on the capture space and the possible camera arrangement space, which then constrains R. Note also that all analyses below are conducted on a 2D floor for simplicity, hence neglecting the object height and the heights of camera positions. This is because the objective of the analyses is just to show how fieldof-view, image resolution, and depth-of-field parameters as well as pan/tilt/zoom control speed constrain the cell radius R. Once a radius R satisfying all constraints is obtained, the detailed 3D parameter design can be done easily by simulation.

58

3


Fig. 3.6 Constraint between the cell radius R and the field-of-view of a camera

3.3.1.2 Image Observation Constraints As discussed in Chap. 2, a camera has a set of controllable parameters to be optimized to capture object images sufficient for 3D video production: focal length, focus, shutter speed, iris size, and gain. Here, we assume that gain and shutter parameters have been already optimized to capture the object in good exposure and high S/N ratio, because their adjustments should be done by taking into account or controlling lighting environments, which is out of the main scope of this book. In what follows, we analyze constrains on R by field-of-view, image resolution, and depth-of-field, which are determined by focal length, iris size, and focus parameters.

3.3.1.2.1 Constraint by Field-of-View Figure 3.6 illustrates the constraint between the field-of-view (i.e. zooming factor) and the cell size. Let d [mm] denote the distance from the projection center to the cell center and r [mm] the radius of the bounding cylinder of the object. To capture the object in this cell,1 the field-of-view at the distance of d should be larger than 2 × (R + r). That is, 2(R + r) W ≤ d f ⇔

R≤

dW − r, 2f

(3.3) (3.4)

where W [mm] and f [mm] are the image width and focal length, respectively. Note that r and W are fixed constants, and f and d design parameters for R [mm]. This constraint implies that the maximum possible cell radius is obtained when the camera is observing the nearest cell with the shortest focal length, i.e. the largest field-of-view. In other words, since the shortest focal length, the image width, and the object size are fixed, the minimum value of distance d determines the maximum possible cell radius R. That is, 1 We

regard the object to be in a cell when the axis of its bounding cylinder is included in the cell.


59

Fig. 3.7 Constraint between the cell radius R and the image resolution

Camera arrangement constraint 1: to make the cell as large as possible, which facilitates the camera control and reduces calibration work, place the cameras as far as possible from the scene.

3.3.1.2.2 Constraint by Image Resolution Figure 3.7 illustrates the constraint between the cell radius R and the image resolution. The maximum pixel coverage s [mm/pix], which is inversely proportional to the image resolution, is given when the object surface is located at d + R + r from the projection center: s=

(d + R + r)W , fN

(3.5)

where N denotes the pixel count of the image width W . Note r, N , and W are fixed constants, f and d control parameters to optimize s and R. This constraint implies that given R, the minimum possible image resolution is obtained when the camera is observing the farthest cell with the largest focal length, i.e. the smallest field-of-view. That is, Camera arrangement constraint 2: to make the minimum image resolution larger than the specified, place each camera so that with its largest focal length, the image resolution at the farthest scene point gets larger than the specified minimum allowable image resolution. In other words, to increase the image resolution, place cameras as close as possible to the scene, which obviously contradicts with the camera arrangement constraint 1. In designing the arrangements of cameras and cells, the largest focal length is fixed and the minimum allowable image resolution is specified. Thus we can compute the maximum possible cell radius R as follows: R = sf

N − d − r, W

(3.6)

where s and f denote the maximum allowable pixel coverage and the largest focal length, respectively.

60

3


Fig. 3.8 Constraint between the cell radius R and the depth-of-field

3.3.1.2.3 Constraint by Depth-of-Field As discussed in Chap. 2, multi-view video data for 3D video production should capture the object without blur. This indicates that the cell size should be designed so that each camera can capture the object within its depth-of-field. Consider a cell located at d [mm] from the camera as shown in Fig. 3.8. In this figure the near and far clips DN and DH of the depth-of-field are given by Eqs. (2.3) and (2.4). Hence this constraint can be expressed as follows: d − R − r ≥ DN ≈

DH df f 2 df = 2 , DH + df f + F cdf

DH d f f 2 df = 2 , d + R + r ≤ DF ≈ D H − df f − F cdf

(3.7)

where F denotes the F -number, c = W/N the diameter of circle-of-confusion, i.e., the physical size of a pixel, and df the distance at which the lens is focused. In this constraint, c = W/N is a fixed constant, df , F and f are control parameters to optimize DN , DF and R. This constant implies that larger F -number (smaller iris) and shorter focal length (wider field-of-view and lower image resolution) make the maximum possible cell radius larger. Since the minimum image resolution is also given as a design specification, making F -number large by using a stronger illumination is the option without trade-offs.

3.3.1.3 Camera Control Constraints While the image observation constraints described above are derived to capture multi-view video data usable for 3D video production, the camera control constraints specify conditions to ensure the continuous object tracking with the given pan/tilt/zoom control mechanisms.


61

Fig. 3.9 Constraint between the cell radius R and the pan speed. If the object moves from p to q, then the camera should start panning from cell 1, j to 3, j when the object goes beyond the midpoint between p and q

3.3.1.3.1 Constraint by Pan/Tilt Control Since we assumed that the object moves on a 2D plane and the cameras are arranged on a ring, the pan control plays a major role in active object tracking. Here, therefore, we analyze the constraint by the pan control alone; the constraint by the tilt control will be satisfied when that by the pan control is satisfied, because both controls share similar dynamics and can be done in parallel. Figure 3.9 illustrates the situation where a camera should switch its focusing cells by the fastest pan control. As explained in Fig. 3.5, the camera should switch the cells within R/2v [sec], where v denotes the maximum object speed. Let ωp [rad/sec] and T denote the angular velocity of the pan control and the computation delay, respectively. Then, we have the following constraint: 2 tan−1 (3R/2d) R +T ≤ . ωp 2v

(3.8)

This constraint implies that the farther a camera is placed from the scene, the larger the cell radius is allowed, which is consistent with camera arrangement constraint 1. 3.3.1.3.2 Constraint by Zoom Control Figure 3.10 illustrates the situation where a camera should switch its focusing cells by the fastest zoom control. Similarly to the above pan control, the camera should switch the cells within R/2v [sec]. Figure 3.11 illustrates that the focal length to capture the entire cell at distance d is given by f [mm] =

(2d − R − r)W . √ 2 3(R + r)

(3.9)

Therefore to zoom out from cell 3, j to 1, j , we need to shorten the focal length by √ 3RW (2(d − 3R) − R − r)W (2d − R − r)W [mm]. (3.10) − = √ √ R +r 2 3(R + r) 2 3(R + r)

62

3


Fig. 3.10 Constraint between the cell radius R and the zoom control speed. If the object moves from p to q, then the camera zooms out from cell 3, j to 1, j when the object goes beyond the midpoint between p and q

Fig. 3.11 The field-of-view of a camera in the case of Fig. 3.10

Hence the constraint by the zoom control is given as √ 3RW R +T ≤ , z(R + r) 2v

(3.11)

where z [mm/sec] is the zooming speed. Note that this constraint does not include d and hence purely specifies the allowable range of R given the zoom control speed. By solving this inequality for R, we have √ √ R≥ 12v 2 W 2 + 8 3v 2 zT − 4 3rvz W + 4v 2 z2 T 2 + 4rvz2 T + r 2 z2 √ + 2 3vW + 2vzT − rz (2z)−1 . (3.12)

3.3.2 Studio Design Process As shown in the previous section, the key design factors, the cell radius R and the camera distance d, should be determined to satisfy all constraints described above. Note that while not discussed above, d is rather strictly constrained by physical studio environments where physical cameras can be placed. Hence, it is reasonable to take the following studio design process: 1. Select active cameras with high image resolution, small F -number, large zooming factor, and fast pan/tilt/zoom control. 2. Place the cameras uniformly to surround the capture space. Their distances to the scene should be made as far as possible while satisfying the camera arrangement constraint 2. Note that depending on the shape of the capture space, the camera ring should be deformed to cover the space uniformly. This book assumes the


3. 4.

5.

6.

63

capture space is almost isotropic, where the ring camera arrangement can be employed. The optimal camera arrangement for an anisotropic capture space is left for future studies. Lighting should be made as bright as possible to guarantee sufficient depth-offield and reduce motion blur with small iris size and fast shutter speed. Then, compute the cell radius R which satisfies the constrains described by Eqs. (3.4), (3.6), (3.7), (3.8), and (3.12) assuming the cell placement rule described next. Note that the cell radius computation should be done for each camera and we have to find such R that satisfies all constraints for all cameras. Since the constraints are valid only for the 2D scene, some margin should be introduced to determine R. One idea to reduce the margin is to use the maximum or minimum 3D distances from a camera to the capture space as d in the constraints. If no R satisfies the constraints, then employ higher performance cameras, modify the camera arrangement, and/or reduce the capture space, and recompute R. Partition the scene into a group of hexagonal cells of radius R. That is, the 2D floor is partitioned into the hexagonal tessellation with radius R by fixing a cell at the center of the floor. This cell placement should be taken into account in the previous process of computing R. Conduct the 3D simulation to verify that the continuous object tracking can be realized while capturing its multi-view video satisfying the specifications. If necessary, adjust the camera arrangement and/or the cell partitioning to satisfy the specifications or enhance the system performance.

We will show three practical studio designs later in this chapter: two for laboratory environments and one for a large scale real world scene, an ice skate arena.

3.3.3 Cell-Based Camera Calibration Once the cell partitioning is fixed, the next step is to (1) optimize the camera control parameters to maximize the spatial resolution of captured images, and (2) calibrate the cameras for each cell. Let G1 , G2 , and G3 denote three camera groups, each of which is assigned to three cell groups 1, 2, and 3, respectively, as illustrated in Fig. 3.4. Then the camera control parameter optimization and calibration are conducted as follows. Step 1 Cell-wise control parameter optimization and calibration: For each cell i, j , Step 1.1 For each camera in the associated camera group Gj , optimize the control parameters so that it observes the entire cell i, j with the highest spatial resolution. In this optimization, we assume that the 3D object shape is modeled by a cylinder of the specified radius and height. Let Ei,j denote the set of the optimized control parameters for the cameras in Gj to capture cell i, j .

64

3


Step 1.2 Calibrate the intrinsic and extrinsic parameters Ai,j , Ri,j , Ti,j of the cameras in Gj under Ei,j using the static camera calibration algorithm described in Chap. 2. Step 2 Pair-wise integration of intra-camera-group calibrations: For each pair of neighboring cells i, j and i , j , Step 2.1 Suppose camera groups Gj and Gj observe cells i, j and i , j with the optimized control parameters Ei,j and Ei ,j , respectively. Step 2.2 Control the other camera group Gj to observe both cells i, j and i , j simultaneously. Step 2.3 Estimate the intrinsic and extrinsic parameters of cameras in Gj . Notice that the extrinsic parameters of each camera group are described in the intra-group local coordinate system. Step 2.4 Place at least three 3D reference points in each of cells i, j and i , j , respectively, and estimate their 3D positions in each intral group local coordinate system. Let kpi,j denote the lth 3D point in the cell i, j described in the camera group k coordinate system. l Step 2.5 Solve the absolute orientation problem [10] between jpi,j and l jp i,j .

This provides the optimal rigid transformation of the extrinsic parameters Ri,j , Ti,j into the group Gj coordinate system. Step 2.6 Similarly we can also transform Ri,j , Ti,j in the group Gj coordinate system, and as a result, the rigid transformation between Ri,j , Ti,j and Ri,j , Ti,j are obtained. By repeating the above pair-wise integration process, all cell-based extrinsic camera parameters can be integrated to a single coordinate system. Note that, since the extrinsic camera parameter integration does not modify the intrinsic, lens distortion, and extrinsic camera calibration parameters computed for each cell, the accuracy of the camera calibration inside a cell is preserved, although the inter-cell camera calibration may introduce some spatial discontinuity between neighboring cells. While the intra-cell camera calibration should be done for each cell, the number of the inter-cell calibration processes can be reduced. That is, camera group Gj in the above method can be controlled to observe multiple cells associated with Gj and Gj simultaneously. Then we can integrate intra-cell calibrations in such cells into a single local coordinate system defined by Gj at once. To reduce calibration labor more, we can select some cameras and make them observe the entire scene for ˆ denote the group of the selected establishing the global coordinate system. Let G ˜ j (j = 1, 2, 3) three camera groups excluding the selected ones. By cameras and G ˆ and G ˜ j , we can describe the positions and postures observing 3D points both by G ˆ of the cameras in Gj by the G camera coordinate system, i.e. the global coordinate system. For n cells, while the original pair-wise integration requires n(n − 1)/2 times repetition of Step 2.2 above to transform all intra-cell extrinsic camera parameters into the global coordinate system, this simplified procedure requires only n.

3.4 Performance Evaluations

65

3.3.4 Real-Time Object Tracking Algorithm As described before, the cell-based object tracking and multi-view observation algorithm does not track or observe an object in motion continuously. Instead, it controls pan/tilt/zoom parameters of each camera group to make its member cameras focus to its associated cell for the multi-view object observation and switch focusing cells from one to another according to the object motion. The cell-based object tracking and multi-view observation is realized by the following real-time camera control method. Step 1 The object starts actions from the initial position in the specified cell. For example suppose the object starts from x in cell A of Fig. 3.3. Then, capture the multi-view object video with the camera group assigned to cell A. Depending on the object initial position in cell A, the other two camera groups are controlled to observe two neighboring cells of A, respectively (cells B and C in Fig. 3.3). Step 2 The camera group focusing on the cell with the object detects the object from observed multi-view video frames, and computes its 3D position in real time. Here a rough 3D shape reconstruction method is employed and 3D video production is conducted later from captured video data. Note here that if the camera group focusing on a cell adjacent to the cell with the object is not in motion, multi-view video data captured by such camera group are recorded for 3D video production. Consequently, all cameras can be used to capture multi-view object video data. In other words, the algorithm guarantees that at least 1/3 of the cameras capture multi-view object video. Step 3 If the object crosses the rule border, then switch the camera group to the new cell as described in Sect. 3.3.1.1. And when the camera group stops motion, start capturing multi-view object video even if the object is not in its focusing cell. Step 4 Go back to Step 1.

3.4 Performance Evaluations 3.4.1 Quantitative Performance Evaluations with Synthesized Data To evaluate quantitatively the performance of the cell-based object tracking and multi-view observation algorithm (the cell-based algorithm, in short), we first conducted experiments using a pair of 3D video sequences produced by the previous system [22] as ground truth data, whose original multi-view video data were captured in Studio A (Fig. 2.4 and Table 2.3). As shown in Fig. 3.12, the 3D video sequences represent a person walking in the studio. The red curves in Fig. 3.13 illustrate a pair of object motion trajectories reconstructed by the previous system. Regarding these 3D video sequences as actual ground truth object motions, we designed the following simulation setups:

66

3


Fig. 3.12 3D video sequence used for the simulation. ©2010 IPSJ [22]

Fig. 3.13 The object motion trajectories and the cell arrangement used for the performance evaluation. The hexagonal cells are given unique cell IDs consisting of cellclusterID, cameragroupID. ©2010 IPSJ [22]

Scene space: 4 [m] × 4 [m] as illustrated in Fig. 3.13. Object size: approximated by a cylinder of 0.9 [m] diameter and 1.8 [m] height. Active cameras: the same pan/tilt/zoom control parameters and image resolution as those of the real active cameras installed in Studio A. Maximum time for a camera to conduct the cell switching: τ = 1.0 [sec].


67

Fig. 3.14 Camera arrangement. The squares and circles represent the camera positions. The former are placed at 2,200 mm height and the latter at 500 mm to increase the observability of the object surface. ©2010 IPSJ [22]

Lowest allowable image resolution: pixel coverage s ≤ 8 [mm/pix]. Maximum object velocity: v = 0.3 [m/sec]. Camera arrangement: camera ring of 8 [m] radius with 24 cameras as illustrated in Fig. 3.14. To increase the observability of the object surface, a half of the cameras are placed at 2,200 [mm] height and the other at 500 [mm]. Then, the cell radius was determined as R = 0.6 [m] and the cell partitioning illustrated in Fig. 3.13 was obtained. Note here the following. • The cell radius is set rather small to guarantee enough image resolution, because the image resolution of the active cameras in Studio A is just VGA (640 × 480). • To satisfy the fundamental constraint, R ≥ 2τ v, the slow object motion speed and the large radius of the camera ring were employed; the camera ring radius was set much larger than the actual on in Studio A (see Table 2.3). The practical process of designing the cell partitioning will be shown in the third experiment presented in this section. As a baseline method, we designed the following multi-camera system: • The same camera ring as illustrated in Fig. 3.14 was employed. • Pan/tilt/zoom parameters of 24 cameras on the ring were adjusted to observe the entire capture space and were fixed to regard the active cameras as static ones. Then, using the multi-view video data virtually captured by both the cell-based and the baseline methods, new 3D video sequences were produced and compared with the original one for their performance evaluation. Here we used the volumetric 3D shape reconstruction method in Sect. 4.4.1.2, and the view-independent texture generation technique in Sect. 5.3. For the performance evaluation, we used the following measures. Viewpoint usage: This evaluates how many cameras are observing the object at each video frame. Pixel usage: This evaluates how many pixels cover the object surface.

68

3


3D shape reconstruction accuracy and completeness [16]: This evaluates how accurately and completely the object shape can be reconstructed from observed multi-view video data. Peak signal-to-noise ratio (PSNR) of rendered images: This evaluates the overall quality of the captured multi-view video data by comparing free-viewpoint object images generated from them with those from the original 3D video sequences.

3.4.1.1 Viewpoint Usage This measure specifies how many cameras among the installed ones are used to observe the object. Since the baseline method by design observes the object with all cameras, its viewpoint usage is always 1. On the other hand, the viewpoint usage of the cell-base method varies depending on the object motion. Figure 3.15 shows the timing charts of the active camera control for the two test sequences. For each sequence, the upper graph shows stop & go action profiles for each camera group and the lower the viewpoint usage. From these results, we can get the following observations: • The cell-based method worked well to continuously capture multi-view object video; in all video frames, the object was observed by at least one of three camera groups. • Since three camera groups are controlled to observe a cell cluster consisting of three neighboring cells that share a common corner, the object can be observed even from the camera groups whose assigned cells do not include the object. That is, the viewpoint usage of the cell-based method usually exceeds 1/3 and the full viewpoint usage (=1) can be attained quite often. These results proved that the cell-base method is really effective in realizing the active tracking and multi-view observation of an object moving in a wide area with a limited number of cameras.

3.4.1.2 Pixel Usage The pixel usage measures how many pixels are covering the object surface. It is defined for video frames including object images as follows: (Pixel usage at frame k) =

NO 1 i , |G(k)| NiI

(3.13)

i∈G(k)

where G(k) denotes the set of cameras observing the object at frame k, NiO the number of pixels occupied by the object in the camera i image, and NiI the total number of pixels of the camera i image. Figure 3.16 shows the temporal profiles of the pixel usage for each test sequence by the cell-based and baseline methods, and Table 3.1 their average values and standard deviations. From these results, we can make the following observations:

3.4 Performance Evaluations Fig. 3.15 Timing charts of the active camera control for two test sequences. For each sequence, the upper graph shows the stop&go actions: the boxes along the horizontal axis represent the temporal intervals when each camera group fixes all views of its member cameras onto a cell to observe the object in its interior area or to wait for the object to move into the cell, and gaps between the boxes indicate that the camera group was in motion. Numbers in the boxes represent the cell cluster ID observed by each camera group. ©2010 IPSJ [22]

Fig. 3.16 Pixel usage. ©2010 IPSJ [22]

69

70

3


Table 3.1 Pixel usage. Each number shows the average ± standard deviation per sequence. ©2010 IPSJ [22]

Cell-based [%]

Baseline [%]

Sequence 1

6.7 ± 0.32

1.9 ± 0.10

Sequence 2

7.2 ± 0.45

2.0 ± 0.16

• The average pixel usage of the cell-based method is limited to about 7 %. This is because the camera parameters are controlled to observe the entire cells rather than the object. Even though the pixel usage is limited, high resolution object images can be captured with higher resolution cameras. • Compared with the baseline method with wide-view static cameras, the cell-based method attained about three times better average pixel usage. This implies that the cell-based method can capture object images in 1.8 times finer resolution approximately. These results proved that the cell-base method is effective to capture finer resolution images of an object in motion with active pan/tilt/zoom controls.

3.4.1.3 Shape Reconstruction Accuracy and Completeness With the given pair of 3D video sequences, the accuracy and completeness of the 3D object shape reconstructed from multi-view object images observed by the cellbased and baseline methods can be evaluated. Figures 3.17 and 3.18 show 90 % accuracy and 10 mm completeness for each frame in the two sequences by the cell-based and baseline methods. 90 % accuracy denotes the distance d (in cm) such that 90 % of the reconstructed surface is within d cm of the ground truth, and 10 mm completeness measures the percentage of the reconstructed surface that are within 10 [mm] of the ground truth [17]. The viewpoint usage is also illustrated at the bottom of each figure to evaluate the performance. Figure 3.19 summaries their averages and standard deviations. From these results, we can observe the following:

Fig. 3.17 10 mm-completeness and 90 %-accuracy for sequence 1. ©2010 IPSJ [22]


71

Fig. 3.18 10 mm-completeness and 90 %-accuracy for sequence 2. ©2010 IPSJ [22]

Fig. 3.19 Average performance of the cell-based and baseline methods in terms of 10 mm-completeness and 90 %-accuracy. ©2010 IPSJ [22]

• On average, the cell-based method performed equally well or better than the baseline method. • At frames where the viewpoint usage dropped to 1/3, the completeness and accuracy by the cell-based method were degraded. This is because the observability of the object surface with 1/3 of the cameras is limited. • At frames where the viewpoint usage was more than 2/3, the cell-based method outperformed the baseline method in both completeness and accuracy.

72

3


Fig. 3.20 Shape reconstruction results for frame 135 of sequence 1. The first row shows the error distributions of the reconstructed 3D object shapes, where colors indicate distances to the ground truth object surface: blue denotes large error and red small. The bottom row shows the rendered object images and their PSNR with respect to the ground truth image, i.e. the left image. ©2010 IPSJ [22]

To further analyze the effects of the viewpoint usage, we examined frames 135 and 95 of sequence 1 in detail, where the viewpoint usage values were 1/3 and 2/3, respectively. Figure 3.20 shows the results of the 3D shape reconstruction and object image generation for frame 135 of sequence 1, where only 1/3 of the cameras could observe the object in the cell-base method. The color pictures on the top row show the error distributions of the 3D shape reconstructed from the multi-view object images observed by the cell-based and baseline methods, respectively. While large errors in blue are distributed widely in the baseline method, they are concentrated at the middle area in the cell-based method. These errors are introduced because the observability of such areas was greatly reduced due to the self-occlusion by the object arm. That is, 1/3 of the cameras could not capture enough multi-view object images to resolve the self-occlusion. The bottom row shows object images rendered


73

Fig. 3.21 Shape reconstruction results for frame 95 of sequence 1. ©2010 IPSJ [22]

based on the original, and two reconstructed 3D object shapes, where PSNR with respect to the ground truth image, i.e. left image, were computed. (The PSNR computation method will be described below.) These results proved that the cell-based method can produce 3D video of comparable qualities to the baseline method even with 1/3 of the camera. To guarantee the high observability, the total number of cameras should be increased; eight cameras are not enough to attain the sufficient observability of an object in action. To evaluate the performance of the cell-based method with the high viewpoint usage, we analyzed frame 95 in sequence 1, where 2/3 of the cameras observed the object. Figure 3.21 shows the same data as in Fig. 3.20. Comparing these figures, the performance of the cell-based method is much increased and outperforms the baseline method in all measures: accuracy, completeness, and PSNR. The major reason for this is the increased object image resolution, i.e. the higher pixel usage. Figure 3.22 compares a pair of images taken by camera 1 at frame 95 in sequence 1 by the cell-based and baseline methods, which demonstrates the effectiveness of active zooming in the cell-based method. The increased object image resolution contributes significantly to both 3D shape reconstruction and texture generation.

74

3


Fig. 3.22 Images captured by camera 1 at frame 95 in sequence 1. ©2010 IPSJ [22]

3.4.1.4 PSNR of Rendered Images As the final and overall performance evaluation, PSNR was computed for object images rendered based on the 3D object shapes reconstructed from multi-view video data virtually captured by the cell-based and baseline methods. Object images rendered from a pair of given 3D video sequences were used as the ground truth data. The object image rendering was done in the following setups. Viewpoint: For each video frame, place the viewpoint of the virtual camera 3 m away from the object front side. Field of view: 32 degrees. Image size and resolution: 250 × 450. Texture generation: The view-independent texture generation in Sect. 5.3. Figure 3.23 illustrates the dynamic PSNR profiles of the cell-based and baseline methods for two 3D video sequences. This demonstrated that the cell-based method can produce much better 3D video data thanks to the finer resolution of captured object images.

3.4.2 Quantitative Performance Evaluation with Real Active Cameras To evaluate the performance of the cell-based method in the real world, we implemented it in Studio A, whose specifications are shown in Fig. 2.4 and Table 2.3. Figure 3.24 illustrates the camera arrangement. Note that two cameras placed at the center of the ceiling were not used for the experiment here to realize the camera ring arrangement with 23 active cameras: 13 cameras installed on the floor and 10 cameras on the ceiling. Each active camera was implemented by mounting a Sony DFW-VL500 camera with a computer controllable zoom lens on a pan-tilt unit PTU46 by Directed Perception, Inc. They were connected to a PC cluster system with 24 nodes: one master and 23 camera nodes to capture and control 23 active cameras, respectively. The nodes were connected by 1000Base-T Ethernet, and communications between nodes were implemented by UDP. The system clocks were synchronized by NTP. As is shown in Fig. 3.24, the physical object movable space i.e. the capture space, is limited to about 3 × 3 [m], since the physical studio size is limited to about 8 [m] square and the active cameras cannot take focused object video when it comes close


75

Fig. 3.23 PSNR of rendered images. ©2010 IPSJ [22]

Fig. 3.24 3D video studio for active object tracking and multi-view observation (Studio A in Fig. 2.4 and Table 2.3). The circles and squares indicate camera positions: the former are placed 500 mm above the floor looking upward and the latter at the ceiling 2200 mm above the floor looking downward. The numbers denote camera IDs, and the red/blue/green border colors indicate cell groups (cf. Table 3.2). ©2010 IPSJ [22] Fig. 3.25 Object used in the real studio experiment: a stuffed animal toy on a radio controlled car. ©2010 IPSJ [22]

to the cameras. Thus we used a small object rather than a human for the experiment: a stuffed animal toy on a radio controlled car shown in Fig. 3.25. It moves at most v = 0.208 [m/sec], and its volume is approximated by a cylinder of 0.800 [m] diameter and 0.500 [m] height. Note that this miniature size experiment is not due to the cell-based method itself but to the limited studio space. The next section will give a system design for a large movable space, i.e. ice skating arena.

76

3


Fig. 3.26 Cell arrangement and object trajectory in the studio experiment. ©2010 IPSJ [22]

Table 3.2 Camera groups and cells assigned to each group. ©2010 IPSJ [22]

Group

Member camera IDs

No. of assigned cells

1

2 7

9

19

20

21

22

23

7

2

3 5

8

13

14

16

17

18

5

3

1 4

6

10

11

12

15

5

By specifying the maximum time for a camera to conduct the cell switching as τ = 1.2 [sec] and the minimum allowable image resolution as pixel coverage s = 5 [mm/pix], we obtained the cell arrangement shown in Fig. 3.26, where the estimated object trajectory is overlaid. Table 3.2 shows the numbers of member cameras and assigned cells for each camera group. Figure 3.27 shows the temporal profile of the viewpoint usage. As designed, the object was observed by more than seven cameras at all frames. Since the object moved back and forth, it traveled across rule borders several times around frame 400, which decreased the viewpoint usage; a camera group has to change its focusing cell when the object crosses its corresponding rule border. Note that even in such “busy” period, the object was tracked and its multi-view videos were observed continuously by camera groups focusing on the cells including the object.

Fig. 3.27 Viewpoint usage in the studio experiment. ©2010 IPSJ [22]

3.5 Designing a System for Large Scale Sport Scenes

77

Fig. 3.28 Images captured by camera 1 at frames 200 and 400. The upper figure shows the fields of view and the lower the captured images. ©2010 IPSJ [22]

Figure 3.28 shows the fields of view and captured images by camera 1 at frames 200 and 400. We can observe that the object size in images, i.e. the pixel usage, is kept almost constant regardless of the object distance from the camera. To evaluate the quality of 3D video data, we applied the volumetric 3D shape reconstruction algorithm in Sect. 4.4.1.2 and the view-independent texture generation method in Sect. 5.3 to the captured multi-view video data. Figure 3.29 shows a pair of free-viewpoint visualizations of the produced 3D video at frame 395. Even though the object was observed by only one camera group with eight cameras, fine textures on the object surface such as the harness or letters written on the car can be rendered. These results proved that the cell-based method can work well in the real world even with off-the-shelf camera devices and PCs.

3.5 Designing a System for Large Scale Sport Scenes The evaluations in the previous section proved that the cell-based method can successfully track an object moving freely in the scene, and can capture its multi-view

78

3


Fig. 3.29 Produced 3D video at frame 395 with eight cameras. ©2010 IPSJ [22]

video data in sufficient qualities for 3D video production in the laboratory environment. The goal of this section is to demonstrate the practical utilities of the cell-based method by designing a system for real world large scale sport scenes with off-theshelf active cameras. As such real world scenes, we here choose figure skating. According to the standard regulations by the International Skate Union [4], the typical shape of the ice rink for short and free skating programs is defined by a rectangle of 60 × 30 [m]. We here discuss how we can design the camera and cell arrangements taking into account physical characteristics of the target motion, capture space size, and active camera dynamics.

3.5.1 Problem Specifications Figures 3.30 and 3.31 show the virtual ice skate arena we consider in this section. The capture target is a figure skater moving at most 10 [m/sec] on the 2D ice rink.2 The 3D volume of a skater is approximated by a cylinder of 2 [m] diameter and 2 [m] height, which will be enough to encase the skater even if she/he performs complex actions. We assume that the cameras can be installed Horizontally: between 0 [m] and 30 [m] away from the rink (the area surrounding the rink in Fig. 3.31), and Vertically: between 0 [m] (on the floor) and 10 [m] (on the ceiling) above the ice plane. The minimum allowable image resolution is set to a pixel coverage s = 8 [mm/pix] on the object surface. 2 While we do not know if this speed is reasonable, we had to limit the maximum speed due to the camera control speed of off-the-shelf active cameras employed.


79

Fig. 3.30 Virtual ice skate arena

Fig. 3.31 Physical specifications of the virtual ice skate arena

As a typical off-the-shelf pan/tilt/zoom camera, we adopted Sony EVI-HD7V whose specifications are summarized in Table 3.3. For simplicity, we assume that its pan, tilt, and zoom control speeds are constant regardless of their current values. Note that EVI-HD7V does not accept synchronization signals, and as a consequence, multi-view video data cannot be used for 3D video production. We use it only to obtain pan/tilt/zoom control characteristics of off-the-shelf active cameras. In addition to the physical pan/tilt/zoom control speeds of the camera, the object detection and tracking processes add a delay including (1) image data acquisition, (2) object detection and 3D position localization from acquired image data, (3) PCTable 3.3 Specifications of SONY EVI-HD7V pan/tilt/zoom camera

Image sensor

1/3 inch (4.8 mm × 3.6 mm), CMOS

Image resolution

up to 1920 × 1080

Pan angle

±100 degrees

Tilt angle

±25 degrees

Pan speed

300 degrees/sec

Tilt speed

125 degrees/sec

Horizontal angle of view

70 (wide) ∼ 8 (tele) degrees

Focal length

3.4 (wide) ∼ 33.9 (tele) mm

Zoom speed

30.5 mm/sec

Iris

F1.8 ∼ close

80 Table 3.4 Assumed specifications

3

Active Camera System for Object Tracking and Multi-view Observation Object motion

v ≤ 10 [m/sec]

Object radius

r ≤ 1 [m]

Effective image resolution s ≤ 8 [mm/pix] Control delay

T ≤ 0.03 [sec]

Horizontal pixel resolution N = 1920 [pix] Horizontal imager size

W = 4.8 [mm]

Focal length

f = 3.4 ∼ 33.9 [mm]

Angular pan speed

ωp = 300/180 × π = 5π/3 [rad/sec]

Zoom speed

z = 30.5 [mm/sec]

Iris

F ≤ 2.8

to-camera data transfer, etc. We here model this delay by a constant value T = 0.03 [sec]. As described in the previous chapter, the lighting environment plays an important role on designing the shutter, gain, iris parameters, and as a result it affects the depth-of-field. In this section we assume that the cameras can close their irises up to F = 2.8 to capture images in acceptable S/N ratio. Table 3.4 summarizes the employed specifications.

3.5.2 Camera and Cell Arrangements Based on the given specifications, this section derives the arrangements of cameras and cells which satisfy all requirements for 3D video production as well as physical limitations. Firstly, the cameras should be distributed uniformly surrounding the ice rink. Secondly, the distance from the object to each camera does not affect the image resolution as long as it can be compensated by zooming. Thirdly, pan/tilt angles to be changed during the cell switching get larger when an active camera is placed closer to the rink. Hence the cameras should be located as far as possible from the rink. By taking these into account, we designed the camera arrangement as illustrated in Fig. 3.32. In this figure, 24 cameras are mounted on the ceiling edges to make the camera-to-skater distance as far as possible. The biased distribution of the cameras reflects the oblate shape of the rink and makes the camera-to-object angles to distribute evenly. The figure also illustrates the designed cell arrangement. The cell radius is determined as 5.2 [m], and the rink is covered by 32 cells in total. In what follows, we determine the key design factors, the cell radius R and the camera distance d, based on the constraints described in Sect. 3.3.1.


81

Fig. 3.32 Results of the active camera arrangement and the cell arrangement. Small circles represent the camera positions

3.5.2.1 Image Observation Constraints To examine the validity of camera parameter setups, it is sufficient to analyze the nearest and farthest cells from the cameras. In the arena, supposing the cameras are installed on the ceiling of 10 [m] height and 30 [m] away √ from the rink, then the 302 + 102 = 31.623 [m]. shortest distance between a camera and a cell is dmin = √ Similarly the longest distance to the farthest cell is dmax = (30 5 + 30)2 + 102 = 97.596 [m]. Note that 3D distance values rather than 2D ones in Sect. 3.3.1 were used to make the design process practical. By substituting these values with d of the constraints described in Sect. 3.3.1.2, we obtain feasible ranges of focal length and cell radius as illustrated by Fig. 3.33. In this figure, the ranges of possible focal length and cell radius are indicated by the hatched areas where all inequalities hold. From these plots, we can observe that the maximum possible cell radius is given as the intersection of Eq. (3.4) and (3.6). Then, by solving R=

dW −r 2f

N R = sf −d −r W

(3.14)

for f and R, we have √

2dsN + d 2 − 2r − d 2 √ ( 2dsN + d 2 + d)W . f= 2sN

R=

(3.15)

Since we employ a uniform cell arrangement, the maximum possible cell radius is limited by the lower one of the two extreme, i.e. nearest and farthest, cases. Thus,

82

3


Fig. 3.33 Image observation constraints represented by Eqs. (3.4), (3.6), and (3.7) for the two extreme cases: (a) nearest and (b) farthest. The hatched areas denote feasible ranges of focal length and cell radius

by substituting d in Eq. (3.15) with dmin , we obtain R ≤ 5.389 [m],

(3.16)

from the image observation constraints. Note that the camera-to-object distance df in Eq. (3.7) is set to 30.5 [m] and 76.5 [m] for Figs. 3.33(a) and (b), respectively. These values make the center of the depth-of-field coincide with the cell center.

3.5.2.2 Camera Control Constraints As discussed before, the quickest pan control is required when switching between two nearest cells. In the current camera arrangement, the shortest distance between a camera and its nearest cell is dmin . Figure 3.34(a) shows the constraint by Eq. (3.8) in this case, where the horizontal axis denotes R and the vertical axis the difference between the left-hand-side and the right-hand-side in Eq. (3.8). That is, the inequality holds where the value is negative, i.e., R ≥ 0.941 [m]. Equation (3.12) provides the constraint on R in terms of the zooming. By substituting the constant values in this inequality, we obtain R ≥ 5.168 [m].

(3.17)

Figure 3.34(b) shows the plot of the inequality of Eq. (3.11) as a reference. Notice that the constraint on zoom does not depend on the distance between the camera and cell. In summary, we obtain R ≥ 5.168 [m] from the camera control constraints.

(3.18)

3.6 Conclusion and Future Works

83

Fig. 3.34 Camera control constraints represented by (a) Eq. (3.8), (b) Eq. (3.11). In both graphs, the vertical axis denotes the difference between the left-hand-side and the right-hand-side of each inequality, which hence is satisfied in the negative value area

3.5.2.3 Optimal Cell Size By integrating the constraints on R by both the image observation constraints and the camera control constraints, we have for image observation: R ≤ 5.389 [m] for camera control: R ≥ 5.168 [m].

(3.19)

Consequently, the possible cell size is 5.168 ≤ R ≤ 5.389 [m]. Based on this condition, we designed the cell of R = 5.2 [m] to maximize the pixel usage; the smaller the cell, the larger the object image becomes in the cell. This design process also indicates that the zoom speed is the crucial factor which limits the cell size dominantly. In summary, the above design process demonstrated that the cell-based method can be used to track an object and capture well-focused multi-view video of large scale sport scenes like ice skating. In practice, however, the specified image resolution, i.e. pixel coverage s = 8 [mm/pix], is not sufficient to produce high fidelity 3D video compared to those attained in studios with static cameras in Table 2.5. Based on our simulation, s = 7 [mm/pix] will be the minimum possible pixel coverage under the given specifications. To realize multi-view video capture of higher image resolution, it is required to employ active cameras with higher functions than SONY EVI-HD7V in physical image resolution and zooming speed as well as to reduce the computational delay T .

3.6 Conclusion and Future Works This chapter presented a novel multi-view video capture system with a group of active cameras which cooperatively track an object moving in a wide area and capture

84

3


high resolution well-focused multi-view object video data. The novelty rests in the cell-based object tracking and multi-view observation, where the scene space is partitioned into a set of disjoint cells, and the camera calibration and the object tracking are conducted based on the cells. The cell-based system scheme was developed to satisfy the following fundamental requirements for the multi-view video capture for 3D video production. Requirement 1: Requirement 2: Requirement 3: Requirement 4: and 3.

Accurate camera calibration, Full visual coverage of the object surface, High spatial image resolution, and Track a moving object in real time while satisfying requirements 2

To develop a practical cell-based method, the hexagonal tessellation was introduced for the cell arrangement on a 2D floor on which an object moves. Then, the sets of cells and cameras are divided into three groups, respectively, and one-to-one correspondences are established between the cell and camera groups. We analyzed the algebraic constraints among various parameters to satisfy the above requirements, and designed three 3D video studios: two for laboratory environments and one for a large scale real world scene, an ice skate arena. The experimental results with synthesized and real world scenes demonstrated the effectiveness of the cellbased method in capturing high resolution well-focused multi-view video for an object moving in a wide spread area. Throughout this part, we assumed that the cameras should be installed uniformly in the studio, and therefore the ring arrangement is a preferable choice in general. However, real applications such as 3D video for movie productions may require an anisotropic capture space, and consequently, a non-uniform camera distribution. In [22], actually, we developed a cell-based method which arranges a set of cells along the object trajectory specified by a scenario, assuming the camera arrangement is fixed. It is left for future studies to realize the simultaneous optimization of the camera and cell arrangements in various real world applications. In addition, the multi-view video capture of multiple moving objects is another future study direction. While the system we developed in [20] realized versatile multi-object detection and tracking with a group of active fixed-viewpoint cameras, mutual occlusions between objects cannot be resolved with pan/tilt/zoom cameras. To avoid occlusions and guarantee the high observability of multiple object surfaces, the 3D positions of cameras should be controlled by the dolly mechanism, which is also an interesting future research problem to exploit 3D video applications.

References 1. Davis, J., Chen, X.: Calibrating pan-tilt cameras in wide-area surveillance networks. In: Proc. of International Conference on Computer Vision, pp. 144–149 (2003) 2. Fitzgibbon, A.W., Zisserman, A.: Automatic camera recovery for closed or open image sequences. In: Proc. of European Conference on Computer Vision, pp. 311–326 (1998)

References

85

3. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000) 4. International Skating Union: Special Regulations & Technical Rules. Single and Pair Skating and Ice Dance (2008). Rule 342 Required rinks 5. Jain, A., Kopell, D., Kakligian, K., Wang, Y.-F.: Using stationary-dynamic camera assemblies for wide-area video surveillance and selective attention. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 537–544 (2006) 6. Kitahara, I., Saito, H., Akimichi, S., Onno, T., Ohta, Y., Kanade, T.: Large-scale virtualized reality. In: CVPR2001 Technical Sketches (2001) 7. Lavest, J.M., Peuchot, B., Delherm, C., Dhome, M.: Reconstruction by zooming from implicit calibration. In: Proc. of International Conference on Image Processing, vol. 2, pp. 1012–1016 (1994) 8. Lavest, J.-M., Rives, G., Dhome, M.: Three-dimensional reconstruction by zooming. IEEE Trans. Robot. Autom. 9(2), 196–207 (1993) 9. Li, M., Lavest, J.-M.: Some aspects of zoom lens camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 18(11), 1105–1110 (1996) 10. Lu, C.-P., Hager, G.D., Mjolsness, E.: Fast and globally convergent pose estimation from video images. IEEE Trans. Pattern Anal. Mach. Intell. 22(6), 610–622 (2000) 11. Luong, Q.-T., Faugeras, O.D.: Self-calibration of a moving camera from point correspondences and fundamental matrices. Int. J. Comput. Vis. 22, 261–289 (1997) 12. Maybank, S.J., Faugeras, O.D.: A theory of self-calibration of a moving camera. Int. J. Comput. Vis. 8, 123–151 (1992) 13. Mendonca, P., Cipolla, R.: A simple technique for self-calibration. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 637–663 (1999) 14. Pollefeys, M., Koch, R., Gool, L.V.: Self-calibration and metric reconstruction in spite of varying and unknown internal camera parameters. Int. J. Comput. Vis., 7–25 (1999) 15. Sarkis, M., Senft, C.T., Diepold, K.: Calibrating an automatic zoom camera with moving least squares. IEEE Trans. Autom. Sci. Eng. 6(3), 492–503 (2009) 16. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 7–42 (2002) 17. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 519–528 (2006) 18. Sinha, S.N., Pollefeys, M.: Pan-tilt-zoom camera calibration and high-resolution mosaic generation. Comput. Vis. Image Underst. 103(3), 170–183 (2006) 19. Szeliski, R., Kang, S.B.: Recovering 3D shape and motion from image streams using nonlinear least squares. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 752–753 (1993) 20. Ukita, N., Matsuyama, T.: Real-time cooperative multi-target tracking by communicating active vision agents. Comput. Vis. Image Underst. 97, 137–179 (2005) 21. Wada, T., Wu, X., Tokai, S., Matsuyama, T.: Homography based parallel volume intersection: Toward real-time reconstruction using active camera. In: Proc. of CAMP, pp. 331–339 (2000) 22. Yamaguchi, T., Yoshimoto, H., Matsuyama, T.: Cell-based 3D video capture method with active cameras. In: Ronfard, R., Taubin, G. (eds.) Image and Geometry Processing for 3-D Cinematography, pp. 171–192. Springer, Berlin (2010)

Chapter 1

Introduction

1.1 Visual Information Media Technologies Looking back on our modern history, information media technologies innovated our everyday lives and societies. For example, in the 15th century the printing machine by J. Gutenberg enabled mass copying of books and newspapers. The 19th century saw the deployment of the camera and photography to capture vivid real life scenes. Electronic technologies of the 20th century have realized the explosive enrichments of information media with radio, TV broadcasting systems, as well as a variety of media capturing, recording, and presentation devices like video cameras, CD/DVD, and electronic displays. Now in the 21st century, all of our personal and social activities are promoted by multimedia technologies tightly coupled with digital telecommunication technologies such as the Internet, web, and mobile phones. Thus, it is not too much to say that information media technologies embody social innovations. In the area of visual information media technologies, one can observe a line of evolution starting from analog still 2D image (photography) and followed by analog 2D motion picture (movie, TV, and video), which then were enhanced and integrated with telecommunication by digital technologies. At the end of the last century, Computer Vision technologies enabled the exploration of another line leading to 3D images. In fact, the 3D shape of static objects can now be easily captured with off-the-shelf laser rangefinders [18, 21]. It is then natural to explore technologies for 3D motion picture next. Figure 1.1 illustrates the world of 3D visual media technologies, which consists of fundamental disciplines of visual information processing. First of all, the physical visual world includes 3D object(s), camera(s), and light source(s), upon which the following line of processing methods fabricate the world of visual information media. (I) Image Capture: 3D Object ⇒ 2D Image: Light rays emitted from the light source(s) are reflected on the object surface, and are then recorded in the form of 2D image(s) with the camera(s). To derive the computational model of this T. Matsuyama et al., 3D Video and Its Applications, DOI 10.1007/978-1-4471-4120-4_1, © Springer-Verlag London 2012

1

2

1

Introduction

Fig. 1.1 World of visual information media

physical imaging process, camera and light source calibration have to be conducted. Note that the image capture process degenerates the 3D information in the physical world into 2D image data. (II) Image Processing: 2D Image ⇒ 2D Image: A large library of image processing methods [3, 16] can be applied to the captured image data for frequency

1.2 What Is and Is Not 3D Video?

3

and spatial filtering, edge and region segmentation, color, shape, and texture analysis. (III) Computer Vision: 2D Image ⇒ 3D Object: Computer vision technologies [22] enable the reconstruction of 3D object information from the captured 2D image(s): 3D shape from shading, texture, silhouette, motion, stereo vision, and so on. Note that since a camera can only observe the frontal surface of an object, multiple cameras have to be employed to reconstruct full 3D object information including the backside surface information. (IV) Editing/Coding: 3D Object ⇒ 3D Object: To produce attractive visual information media, various types of editing operation should be applied to the reconstructed 3D object: color, texture, shape, and motion editing. Moreover, a virtual scene can be designed by introducing artificially designed objects and/or modifying lighting environments. To facilitate these editing operations and/or reduce the data size for storage and transmission of 3D information, it is common to introduce efficient and flexible data structures and their coding methods. (V) Computer Graphics: 3D Object ⇒ 2D Image: Associated with 3D object editing and coding, computer graphics technologies [4] render virtual 3D objects to 2D image data, from which viewable physical photometric image(s) are generated with display monitors. (VI) Augmented/Mixed Reality: 2D Image ⇒ 2D Image: With immersive displays and see-through head-mounted displays, augmented/mixed reality technologies [2, 6] integrate the real physical and virtual computer-generated scenes to make us perceive the enriched visual world.

1.2 What Is and Is Not 3D Video? Aiming to enrich the world of visual information media, we have been working in the last ten years to develop 3D Video as a new visual information medium and cultivate its utilities in everyday life. Here 3D video [14] is the full 3D image media that records dynamic visual events in the real world as is; it records time varying full 3D object shape with high fidelity surface properties (i.e. color and texture). Its applications cover wide varieties of personal and social human activities: entertainment (e.g. 3D game and 3D TV), education (e.g. 3D animal picture books), sports (e.g. sport performance analysis), medicine (e.g. 3D surgery monitoring), culture (e.g. 3D archive of traditional dances), and so on. S. Moezzi et al. [14] and T. Kanade et al. [10] are pioneers who demonstrated that full 3D human actions could be generated from multi-view video data, and have been followed by many computer vision researchers to explore 3D video production technologies and applications. Now several venture companies [20] are commercially supporting 3D video capture systems and 3D video content generation. 3D video can be best characterized by its full 3D-ness; the full 3D shape and motion of a real world object including its backside is captured as is. While many 3D visual media technologies have been developed, their differences from 3D video can be characterized as follows.

4

1

Introduction

3D TV and Cinema: 3D TV and cinema data are nothing but a pair of 2D stereo video images, from which 3D scenes are perceived in our brains. That is, with special glasses or special display monitors, pop-up motion pictures can be enjoyed. However, their basic limitations constrain the viewing direction of the scene to be fixed; the back, left, or right sides of the scene cannot be seen interactively. Nor can one edit image contents, change object locations or poses, or modify illuminations or shadows. On the other hand, with 3D video data, a pair of 2D stereo video data for 3D TV can be easily generated, and viewing directions and zooming are interactively changed. That is, 3D video enables us to enjoy interactive full 3D scene observations. 3D Depth (Range) Image: Many off-the-shelf devices to capture 3D depth images are available: some employ laser beam time-of-flight [18, 21] and others stereobased methods [19] to measure depth values at object surface points. Usually the captured data are represented by a range image where each pixel value denotes the depth value from the camera center to a surface point. Note that even with this technology no backside surface data can be obtained. In this sense, the captured data should be called 2.5D images rather than 3D. Although 2.5D motion images can be captured in real time with modern range recorders like Kinect [24], 3D video production technologies are required to capture full 3D motion images. Ikeuchi et al. [8] developed a system to reconstruct full 3D shape of huge static objects like big Buddha statues and temples by patch-working a group of 2.5D images observed from different positions. Since the process to capture multiview 2.5D images is time consuming, target 3D objects must be static ones. 3D Motion Capture [12]: The principle of this technology is to attach markers to the object surface and measure their 3D position and motion with a group of cameras surrounding the object. Measured data are just a group of dynamic 3D position sequences of markers, and hence no 3D object surface shape, color, or texture can be obtained. As its name stands for, this enables us just to capture 3D point motions and 3D CG technologies should be employed to generate 3D motion pictures [17]. 3D Animation [11]: The critical difference between 3D video and 3D animation lies in the fact that the former records natural real world objects while the latter designs and generates artificial ones. While 3D CG technologies enable us to generate very sophisticated object shapes, motions, and surface textures, they still stay at natural-looking level. From a technical point of view, although both 3D video and 3D animation often share 3D mesh data for representing 3D object surface shape, they differ in what follows: 1. 3D mesh data of 3D video often change largely from frame to frame since the 3D mesh data of each frame are reconstructed from a set of multi-view image frames, whereas a temporal sequence of 3D mesh data in 3D animation are generated based on the pre-specified 3D motion data keeping the mesh structure (i.e. number of vertices and their connectivities) over time. 2. In 3D video the 3D object motion from a temporal sequence of reconstructed 3D mesh data has to be estimated, whereas in 3D animation 3D object motion data are specified at the designing step.

1.3 Processing Scheme of 3D Video Production and Applications

5

Table 1.1 Functional differences between image-based and model-based methods Processing method

No. of cameras

Image quality

Imaged objects

Viewpoint for image rendering

Lighting environment editing

Content editing

Object motion analysis

Image-based

Several tens— hundred dense arrangement Several—tens sparse arrangement

High

Entire scene natural scene 3D object

Constrained around cameras Completely free

NO

NO

NO

YES

YES

YES

Model-based

Average

3. In 3D video surface texture and color properties of each mesh face should be estimated from observed multi-view images, while those of 3D animation are given at its designing process. 3D CT Image [1]: In medicine 3D CT images are used for detailed medical examination and diagnosis. A 3D CT image is constructed by piling up a set of 2D CT images representing cross-sections of an object. It records the 3D feature distribution in the interior volume of the object, whereas 3D video represents 3D surface shape and texture alone. Even though the 3D CT image may be called a substantially full 3D image, it is still hard to capture objects in motion. Free-Viewpoint TV [9, 15]: This technology has many features in common with 3D Video, in particular, in capturing and visualizing image data: both employ a multi-view camera system where a group of cameras are placed so as to surround an object, and can interactively generate 2D and 2.5D object images viewed from arbitrary viewpoints. Their differences lie in data representation and processing methods: the former employs image-based methods while the latter model-based ones. In the image-based method, a free-viewpoint 2D image of the scene is synthesized based on geometric relations among pixels in a set of multi-view images. To represent the geometric relations, the epipolar image [7], rayspace [5], and light filed [13] representations of the multi-view images were developed. The model-based method, on the other hand, reconstructs explicitly 3D object shape and motion to render free-viewpoint images. Table 1.1 summarizes functional differences between them, from which we believe 3D video has much larger flexibilities. In 3D video, object position, shape, and motion can be analyzed and edited and lighting can be modified to render object images. Moreover, as will be described later in Chap. 7, the object motion can be viewed from the object’s own viewpoint: for example, dancing actions can be viewed from the dancer’s own viewpoint.

1.3 Processing Scheme of 3D Video Production and Applications The processes aligned on the left in Fig. 1.2 illustrate the basic processes to produce a 3D video frame.

6

1

Introduction

Fig. 1.2 Process of 3D video production and applications

(1) Synchronized Multi-View Image Acquisition: A set of multi-view images of an object are taken simultaneously by a group of distributed video cameras. The top row images in Fig. 1.2 show samples of captured multi-view images of a dancing MAIKO, a young traditional Japanese dancer, who wears specially designed Japanese traditional clothes named FURISODE with long sleeves. Her


7

Fig. 1.3 Multi-view video capture system with a PC cluster

hair and cosmetic styles, decorations, design patterns of clothes, and dances are nothing but Japanese cultural assets. From a technical viewpoint, her very long thin sleeves and the free edges of the OBI (a Japanese gorgeous sash) swing largely during her dance, whose 3D shape and motion reconstruction is a technical challenge we attacked in our project. In general, the cameras are spaced uniformly around the object(s) so that the captured images cover the entire object surface (Fig. 1.3). Note that it is often hard to satisfy this requirement of full observation coverage of the object surface. Some parts of the surface are occluded by others even when capturing a single object. Moreover, heavy occlusions become inevitable when capturing multiple objects in action. Thus in order to produce a 3D video, methods that cope with self and mutual occlusions have to be developed. Other technical issues to be solved include geometric and photometric camera calibrations, and layout and calibration of light sources, which will be discussed in Chap. 2. One of the technical challenges we attacked deals with the camera layout and control problem: how to capture a 3D video of an object moving in a wide area while guaranteeing high resolution surface texture observation. We will address these issues and their solutions in detail in Chap. 3. (2) Object Silhouette Extraction: The model-based method requires the segmentation of an object region(s) in each observed image. In a well designed 3D video studio, background subtraction and/or chroma-key method can be applied

8

1

Introduction

Fig. 1.4 Taking multi-view video at a NOH theater: NOH is a Japanese traditional dance and is sometimes played at night lit with a group of small bonfires. In this figure, a MAIKO is illustrated instead of a NOH performer, because 3D video of MAIKO dances will be used in this book

to generate a set of multi-view object silhouettes (second row images from top in Fig. 1.2). Although we ideally want to capture 3D videos in any natural environments as shown in Fig. 1.4, their geometric, photometric, and dynamical complexities forced us to work in well controlled studio environments instead. Nevertheless a 3D video capture system in everyday life environments remains one of our future research targets. (3) 3D Object Shape Reconstruction: Many methods for 3D shape reconstruction from a set of multi-view images have been developed. Even though simple, one of the most popular methods remains the volume intersection. Each object silhouette is back-projected into the common 3D space to generate a 3D visual cone encasing the 3D object. Then, such 3D cones are intersected with each other to generate the voxel representation of the object shape (third picture from top in Fig. 1.2). Since this method utilizes only silhouette information, many concave parts of the object cannot be reconstructed. Hence a 3D shape refinement process based on surface texture and motion information has to be performed. For such surface-based processing, voxel data have to be converted into 3D surface mesh data (third picture from bottom in Fig. 1.2). The second picture from the bottom in Fig. 1.2 illustrates a refined 3D shape obtained with our 3D mesh deformation method. Silhouette, texture, and motion information are integrated to accurately fit the 3D mesh to the object surface. Note that the preceding process of object silhouette extraction can be integrated with the 3D shape reconstruction process. That is, an integrated method of segmentation and 3D shape reconstruction can be developed. In Chap. 4, we will survey methods of 3D shape reconstruction from multi-view video data and in-


9

troduce several algorithms we developed for our 3D video project with experimental results. (4) Surface Texture Generation: With a reconstructed 3D object shape, color and texture on each 3D mesh face are computed from the observed multi-view images (bottom picture in Fig. 1.2). The technical problems to be solved here are: 1. Since a mesh face may be observed differently in multiple images, how to integrate the images to generate smooth and high resolution 3D video? Note that since 3D videos are usually observed interactively by changing viewing directions and zooming factors, texture generation methods should take such viewing directions and zooming factors into account to render high fidelity surface texture. 2. The most difficult problem in surface texture generation lies in the inevitable errors in camera calibration and 3D object shape reconstruction, which makes 3D object shape reconstruction not perfectly consistent with the set of multiview images. Thus, a texture generation method that copes with such inconsistency has to be developed. 3. Since one of the appealing characteristics of 3D video is the interactive observation with changing viewpoints, texture generation should be done in real time even if the production of 3D video data is done off-line. We developed several texture generation methods to solve the above mentioned problems, which will be presented in Chap. 5 with rendered 3D video data. By repeating the above processes for each video frame, one can obtain 3D video data. Note that since motion is an important source of information to conduct each of the above mentioned processing steps, methods can be developed for processing a new frame based on the analysis of the preceding one, or by analyzing multiple frames at once in one process. Practical usages of motion information will be introduced in Chap. 4. (5) Lighting Environment Estimation: As mentioned in Sect. 1.1, light sources are one of the most important components of a visual scene. Especially in traditional and artistic dances, dynamically controlled and spatially designed lighting effects play a crucial role to produce attractive visual events. Thus 3D video capture systems should be able to estimate lighting environments under which 3D object images are captured; characteristics of light sources as well as object shape, motion, and texture data are what one wants to obtain by analyzing multiview video data. As shown in what follows, the estimated lighting information can be used in 3D video visualization. Chapter 6 addresses a method to estimate dynamically changing lighting environments. 3D locations and shapes of distributed light sources such as candle lights are estimated from observed video data. Note that the three problems for the 3D video production described so far, namely the object shape and motion reconstruction, the surface texture and color estimation, and the lighting environment estimation have strong mutual dependencies. Hence it

10

1

Introduction

is still left to future studies to solve the three problems at once without any assumptions, i.e. the 3D video production in arbitrary environments. With the methods mentioned above, 3D video data of object(s) can be produced. What should be done next is to cultivate the world of 3D video by developing 3D video data processing methods. They include the following applications that we developed: (6) Editing and Visualization: With 3D video data, the most straightforward application is visualization. A 3D video editor and visualization system was therefore developed to produce attractive 3D video contents. The functionalities of the current system are limited to rather simple ones, such as 3D layout and duplication of 3D video data in a 3D scene surrounded by an omni-directional video. The system allows us to interactively observe the 3D scene with a 3D display. One sophisticated visualization method which was developed can visualize human actions from his/her own viewpoint: extract 3D gaze directions and motions from 3D video data and render the data based on the 3D gaze information. This is indeed a very unique visualization function that cannot be realized with imagebased methods. With this function, for example, one can observe how a dancer like a MAIKO controls his/her eyes to produce emotional effects, and as well one can learn where to look at while performing complex body actions like juggling. These visualization methods are described in Chap. 7. More sophisticated 3D video editing and visualization systems include functions such as the following. Editing Lighting Environments: Given the lighting environments estimated by the method in Chap. 6, object surface reflectance properties can be estimated: intrinsic color and degree of specularity. 3D video can then be visualized under arbitrary lighting environments, which will enhance utilities of 3D video data to produce a variety of visual media with different atmospheres: the same dance can be observed under daylight, sunset, candle lightings, and so on. Some simplified lighting environment editing examples are given in Chap. 6. Editing Body Actions: While the 3D video editor for visualization previously described does not modify captured 3D video data themselves, the action editor conducts geometric and temporal transformations & modifications to generate new 3D video data. Its functionalities range from a simple spatiotemporal transformation method to sophisticated human action analysis. 1. 3D mesh data edition: With 3D video data, one can easily and freely edit 3D mesh data: a thin body can be inflated and an action can be accelerated or slowed down. One technical problem to be solved for this kind of mesh data editing concerns the texture generation process which should work consistently with such 3D mesh modifications. 2. Behavior unit edition: A sequence of complex actions captured in 3D video data can be decomposed into a group of simple behavior units: standing up, hand shaking, turning, walking, and so on (Fig. 1.5). By extracting such behavior units from captured 3D video data and weaving the behavior


11

Fig. 1.5 3D video data are decomposed into a group of behavior units in order to create new 3D video data of new actions. Here, 201 frames of a MAIKO dance sequence were classified into N behavior units (N : 4, 24, 72, 86). Each unit is represented by a unique color

units, new 3D video data of new object actions can be generated. While the captured 3D video data in each behavior unit are preserved, new 3D video data representing a variety of new sequences of object actions can be generated. Chapter 8 addresses methods to extract and edit behavior units from 3D video in order to produce new 3D video data. 3. Action editor: Estimate a kinematic structure and motion of an object from captured 3D video data and edit object actions based on its intrinsic kinematics: for human action editing, extract his/her bone structure and joint motion from captured 3D video data, and edit the kinematic structure and motion so that his/her action can be well synchronized with that of a virtual object. Chapter 9 describes a method to estimate bone and joint locations and motions from 3D video data of a complex human body action such as Yoga playing. Given such kinematic information, 3D animation technologies like skin-and-bones model can be applied to edit actions. While this function enables us to use 3D video data as marker-less motion capture data, our future problem deals with the estimation of bones and joints from 3D video data of a MAIKO wearing very complex clothes. (7) Data Representation and Coding: As shown in Fig. 1.1 and discussed in Sect. 1.1, data structure design and coding are one of the main technical issues when cultivating the visual information media. In fact, appropriate data structures and coding methods enable the exploration of new functionalities, and make data processing, recording, and transmission much easier. In the 3D video world, 3D mesh data with 2D observed surface texture patterns are produced from a set of 2D multi-view video, which can then be transformed into a 3D skin-and-bones model like 3D animation. Each data model has its own functional advantages and limitations. As shown in Table 1.1, for example, although image-based rendering from multi-view video returns high fidelity images, flexibility of content visualization and editing are limited. Since the naturalness of 3D video comes mainly from both natural motions of clothes and

12

1

Introduction

Fig. 1.6 3D video data compression is achieved by converting 3D video frames (top) into geometry images (bottom), where each pixel value RGB represents the position XY Z of a 3D surface mesh vertex. Thus standard MPEG video coding scheme can be used to stream 3D video data

their surface texture, we cannot use the skin-and-bones model as a universal data structure for 3D video even if it enables body action editing. Another important aspect of the data model design is data compression. As a matter of fact, even with modern computing and telecommunication technologies, it is still of great importance to compress huge data of visual information media for easy storage and transmission. As it is well known, a stream of 2D video data can be compressed efficiently with video coding methods like MPEG [23], which has allowed its wide utilities in everyday life activities. Hence, 3D video data compression is one of its most important applications. Chapter 10 addresses a method we developed for 3D video compression, which suits well to MPEG video coding; stored and transmitted data are nothing else but a stream of 2D array data like 2D video (Fig. 1.6). In summary, this book first addresses 3D video production methods consisting of multi-view video capture, 3D shape and motion reconstruction, surface texture mapping, and lighting environment estimation in Chap. 2, Chap. 3, Chap. 4, Chap. 5, and Chap. 6, respectively, and then introduces its applications for visualization, action editing, and coding in Chap. 7, Chap. 8, Chap. 9, and Chap. 10.

References 1. Becker, C., Reiser, M., Alkadhi, H.: Multislice CT. Springer, Berlin (2008) 2. Bimber, O., Raskar, R.: Spatial Augmented Reality: Merging Real and Virtual Worlds. A K Peters, Welesley (2005) 3. Bradski, G.: The OpenCV Library. Dr. Dobb’s Journal of Software Tools (2000) 4. Foley, J.D., van Dam, A., Feiner, S.K., Hughes, J.F.: Computer Graphics: Principles and Practice in C, 2nd edn. Addison-Wesley, Reading (1996) 5. Fujii, T., Kimoto, T., Tanimoto, M.: Ray space coding for 3D visual communication. In: Picture Coding Symposium, pp. 447–451 (1996) 6. Gutierrez, M., Vexo, F., Thalmann, D.: Stepping into Virtual Reality. Springer, Berlin (2008) 7. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000) 8. Ikeuchi, K., Oishi, T., Takamatsu, J., Sagawa, R., Nakazawa, A., Kurazume, R., Nishino, K., Kamakura, M., Okamoto, Y.: The great Buddha project: Digitally archiving, restoring, and analyzing cultural heritage objects. Int. J. Comput. Vis. 75, 189–208 (2007)

References

13

9. iview: http://www.bbc.co.uk/rd/projects/iview 10. Kanade, T., Rander, P., Narayanan, P.J.: Virtualized reality: constructing virtual worlds from real scenes. In: IEEE Multimedia, pp. 34–47 (1997) 11. Kerlow, I.V.: The Art of 3D Computer Animation and Effects. John Wiley and Sons, New York (2004) 12. Kitagawa, M., Windsor, B.: MoCap for Artists: Workflow and Techniques for Motion Capture. Focal Press, Waltham (2008) 13. Levoy, M., Hanrahan, P.: Light field rendering. In: Proc. of ACM SIGGRAPH, pp. 31–42 (1996) 14. Moezzi, S., Tai, L.-C., Gerard, P.: Virtual view generation for 3D digital video. In: IEEE Multimedia, pp. 18–26 (1997) 15. Eye Vision project: http://www.ri.cmu.edu/events/sb35/tksuperbowl.html 16. Rosenfeld, A., Kak, A.C.: Digital Picture Processing, 2nd edn. Academic Press, San Diego (1982) 17. Rusu, R.B., Cousins, S.: 3D is here: Point Cloud Library (PCL). In: Proc. of International Conference on Robotics and Automation, Shanghai, China (2011) 18. Konica Minolta 3D Laser Scanner: http://www.konicaminolta.com/ 19. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 7–42 (2002) 20. 4D View Solutions: http://www.4dviews.com/ 21. MESA Swissranger: http://www.mesa-imaging.ch/ 22. Szeliski, R.: Computer Vision: Algorithms and Applications. Springer, Berlin (2011) 23. Watkinson, J.: MPEG Handbook, 2nd edn. Elsevier, Amsterdam (2004) 24. Microsoft Kinect for Xbox 360: http://www.xbox.com/kinect

Part II

3D Video Production

This part addresses algorithms for 3D video production from multi-view video data captured by the systems presented in Part I. Before discussing the practical algorithms, here we introduce an overall computational scheme of 3D video production. Figure II.1 illustrates the physical scene components included in the world of 3D video: • • • •

objects O in motion, background B, light sources L , and a group of fully calibrated cameras C .

Multi-view video data of objects O surrounded with background scene B are captured by multiple cameras C under lighting environments L . The light sources generate shadings, shadows, highlights on the object and background surfaces. Moreover, interreflections are observed on closely located surfaces. Based on geometric and photometric analyses of the physical world, we can design a computational model of the world of 3D video production as shown in

Fig. II.1 Physical scene components included in the world of 3D video

88

Fig. II.2 Computational model for 3D video production

Fig. II.2, where circles denote physical entities and rectangles imply visual phenomena induced by interactions among the entities: • Object O is characterized by shape So , texture To , and (generic) reflectance properties Ro . • Background B is characterized by shape Sb , texture Tb , and (generic) reflectance properties Rb . • Camera Ci produces images Ii . The arrows in the figure denote dependencies between entities in the model: A ← B implies A depends on B. In other words, it implies that given B, we can compute A: A = f (B). In this model, the object surfaces are lit by light rays directly from the light sources L as well as those interreflected from the objects O and the background B. We can model light sources as a collection of point light sources even if their 3D shape and radiant intensity change dynamically. The lighting environments, the objects, and the background produce a complex light field in the scene. Here the light field includes all phenomena of interreflections, highlights, shadings, shadows, and occlusions. Then, the multi-view cameras C capture parts of the light field as multi-view videos I , respectively. Positions, directions, and characteristics of the cameras define which parts of the light field are recorded in the video data. Disparities between object appearances are induced by such multi-view observations. In the world of 3D video illustrated in Fig. II.1, we introduce a virtual camera Cˆ as the fifth component of the world. One straightforward reason for its introduction

89

is to model free-viewpoint visualization of produced 3D video. As will be discussed later in this part, moreover, the virtual camera plays an crucial role in 3D video production itself. That is, while most of 3D video production processes are designed to generate view-independent (i.e. generic) 3D shape and texture, the complexity of the light field does not allow us to completely reconstruct such generic properties of the objects from limited multi-view video data. In the surface texture generation, especially, while the accurate estimation of surface reflectance (generic) properties in uncontrolled lighting environments is almost impossible, produced 3D video data should represent shiny surfaces and highlights as is for high fidelity visualization of object actions. To cope with such problem, the virtual camera is employed in the surface texture generation process to realize view-dependent 3D video production. Note that we model all entities except the background to be dynamic, i.e., to have time-varying characteristics. For example, lighting by a torch dynamically changes its shape (distribution of point light sources) and radiant intensity. The solid and dashed arrows in Fig. II.2 illustrate dynamic and static relationships, respectively. In this computational model, the scene reconstruction from observed images I is represented as an inverse problem of generating I from O, B, and L . That is, given a computational model including A ← B, we have to develop an inverse computation algorithm that estimates B from A. For the 3D video production, in particular, we need to estimate from I the object shape (position and normal) So , texture To , reflectance Ro , and lighting environment L surrounded by unknown backgrounds B. Thus, the problem of 3D video production from multi-view video data can be represented as follows: Under the computational model I = f (C , So , To , Ro , L , Sb , Tb , Rb ), estimate So , To , Ro , L , Sb , Tb , and Rb from I , C , and if necessary Cˆ. Obviously, this problem is significantly under-constrained and cannot be solved. As referred to Marr’s Shape from X model [1], computer vision has developed a wide variety of scene reconstruction algorithms which exploit visual cues such as shading, silhouette, disparity, etc., introducing assumptions about some of the unknowns in order to convert the original ill-posed problem into a well-posed one. For example, given an image taken under known (controlled) lighting environments, shape-from-shading algorithms estimate normal directions of object surfaces with Lambertian reflectance properties. To convert the significantly under-constrained problem of 3D video production into a manageable one, the following assumptions are introduced: 1. Assume that the surface reflections Ro and Rb follow Lambertian (isotropic reflection) model throughout this book. Note that this assumption does not mean that our algorithms, especially texture generation methods, cannot cope with shiny object surfaces, even though the 3D video production of transparent objects is out of the scope of this book. 2. Neglect all interreflections between object and background surfaces. 3. As presented in Chap. 2, the multi-view video capture is conducted in a 3D video studio, which enables us to control the background scene: Sb , Tb , and Rb can be

90

considered as known if necessary. In other words, 3D video production in natural environments is left for future research. 4. Take a three step computation process for 3D video production: (1) 3D shape So reconstruction (Chap. 4), (2) surface texture To generation (Chap. 5), and (3) lighting environment L estimation (Chap. 6). Each computation step introduces additional assumptions to fulfill the task, which will be described later in corresponding chapters. 5. The 3D shape reconstruction and texture generation methods presented in this part do not assume the scene includes only one object. That is, they can process multi-view video data of multiple objects in motion. This is partly because the controlled background assumption facilitates the identification of each object and partly because the estimation process of So and To should be designed to manage self-occlusion even when the scene includes a single object in motion. Thus we assume that a 3D video data stream can be produced for each object in the scene, respectively, which then will be processed in the applications presented in the next part. 6. Since multi-view video data record 3D surface shape and texture of object(s) in motion, a temporal sequence of 3D textured mesh data is employed as the data structure for representing a 3D video data stream in this book.

References 1. Marr, D.: Vision. Freeman, New York (1982)

Chapter 4

3D Shape Reconstruction from Multi-view Video Data

4.1 Introduction As illustrated in Fig. II.2, the problem of 3D video production, that is, to compute 3D textured mesh data of object(s) in motion from multi-view video data, is very complicated and significantly under-constrained. Hence we divide the problem into sub-problems and develop a solution method for each sub-problem one by one, introducing appropriate assumptions. As the first step, this chapter addresses 3D shape reconstruction methods from multi-view video data. The assumptions employed here include the following. Surface reflection: All surfaces of foreground and background objects follow the Lambertian reflection model, i.e. no specular surfaces or transparent objects are included in the scene. Lighting environments: The entire scene is illuminated by a uniform directed light. Moreover, lights are not attenuated irrespectively of distances between light sources and illuminated surfaces. Interreflection: No interreflections between foreground and background object surfaces are taken into account even if the surfaces have concavities and/or multiple surfaces come close to face each other. Figure 4.1 illustrate the simplified computational model under these assumptions. The black arrows in the figure denote computational algorithms which transforms input image data to the 3D object shape, while the gray arrows illustrate dependency relations as defined in Fig. II.2. With the above assumptions, interreflection and highlight in Fig. II.2 are simplified into “Lambertian reflection.” Moreover, shading and shadow can be regarded as painted patterns of the surface just as texture. In this simplified model, all pixels in multi-view images corresponding to the same 3D object surface point share the same value. In other words, object surface colors and textures are preserved even if camera positions and directions are changed, which enables 3D shape reconstruction methods to conduct appearancebased, i.e. image-based, matching among multi-view images for estimating 3D surface shape. This process is called shape from stereo. Note that since it is very popular T. Matsuyama et al., 3D Video and Its Applications, DOI 10.1007/978-1-4471-4120-4_4, © Springer-Verlag London 2012

91

92

4 3D Shape Reconstruction from Multi-view Video Data

Fig. 4.1 Computational model for 3D shape reconstruction. This is a simplified model of Fig. II.2. The gray and black arrows illustrate “generation” and “estimation” processes, respectively

that objects have specular surfaces such as shiny silk FURISODEs of MAIKOs, the appearance-based matching process among multi-view images are prone to introduce errors and hence the accuracy of the 3D shape reconstructed by shape from stereo is limited. As will be discussed later in this chapter, some 3D shape reconstruction methods employ object silhouettes to estimate 3D object shape, which can work without the above-mentioned assumptions. Since shape from stereo and shape from silhouette are complementary to each other, their combination (the black arrows in Fig. 4.1) enables the robust and accurate 3D shape reconstruction. Another important difference between Fig. 4.1 and Fig. II.2 is that the virtual viewpoint Cˆ in the latter is not included in the former. This implies that the 3D shape reconstruction is implemented as a view-independent generic property estimation process. Here, an essential question is brought up: “Can the generic object property, i.e. 3D shape, be obtained by appearance-based processing?” While the general answer is “no”, our practical solution taken in this book is as follows. • Since the above-mentioned assumptions are valid as the first-order approximation of the world of 3D video production, the 3D object shape can be reconstructed in some reasonable level of accuracy. • Since one of the major applications of 3D video is the interactive visualization of real world 3D object actions, the surface texture generation process should manage introduced errors, as well as specular surface properties that are neglected in the 3D shape reconstruction by employing the information of virtual view-

4.2 Categorization of 3D Shape Reconstruction Methods for 3D Video Production

93

points. That is, the texture generation and visualization processes should be implemented as view-dependent processing. Chapter 5 will discuss and compare view-dependent and view-independent texture generation methods. • As a step toward obtaining completely the generic properties of objects, i.e. 3D shape and surface reflectance, Chap. 6 will present a method of estimating lighting environments including multiple dynamically changing light sources such as bonfires. The problem of estimating 3D shape and reflectance of objects in motion from multi-view video data under known lighting environments would be a reasonable research topic for future studies. Note that developing view-dependent 3D shape reconstruction methods, that is, introducing the virtual viewpoints Cˆ into the computational model of 3D shape reconstruction in Fig. 4.1 would be a possible future research topic for realizing higher fidelity visualization of 3D video. 3D shape reconstruction has been a major research topic in computer vision and a variety of different algorithms have been developed. The rest of this chapter first overviews and categorizes several important approaches to full 3D shape reconstruction of objects in motion from multi-view video data in Sect. 4.2. The categorization is done based on what object properties, i.e. surface or silhouette properties, are employed for, and how object motion is estimated. Then, Sect. 4.3 discusses three essential computational components in designing 3D shape reconstruction algorithms, followed by several practical algorithms with experimental results in Sect. 4.4. Section 4.5 concludes the chapter with discussions on presented ideas and algorithms and topics for future researches.

4.2 Categorization of 3D Shape Reconstruction Methods for 3D Video Production 4.2.1 Visual Cues for Computing 3D Information from 2D Image(s) Marr [30] proposed the computational model of human visual perception, where several algorithms for 3D visual perception were exploited. Augmenting the computational approach to visual perception, computer vision has enriched visual cues for computing 3D information from 2D image(s). While the algorithms are usually referred to as Shape from X as a whole, some of them compute only surface normals without the depth information and may be called pure shape without position from X, whereas others measure 2.5D depth values of visible object surface points and may be named as depth from X. Still, others estimate full 3D object volumes and may be called full 3D shape and position from X. The following summarizes visual cues for the 3D shape reconstruction. Shading: Shading is a very effective visual cue for human to perceive 3D object surfaces. Horn [16] first proposed an algorithm for estimating surface normals from

94


shading. Note that while the visible 3D object surface shape can be reconstructed by integrating surface normals, another cue should be employed to estimate its absolute depth. The originally introduced assumptions of a non-attenuating uniform directional light and a convex Lambertian surface without texture were relaxed by successive researches to make Shape from Shading work in a real world scene characterized by, for example, an attenuating proximal light source and a concave specular surface with non-uniform texture inducing complex mutual reflections [59]. Whatever models of lighting and reflectance are employed, they are crucial requisites for shape from shading. Texture: Shape from Texture [18] estimates surface normals by analyzing texture pattern deformations introduced by the projective imaging process assuming 3D object surfaces are covered with uniform texture. Note that since spatial texture patterns are a generic surface property, shape from texture can be designed to work without accurate models of lighting. Shadow: Shadows carry useful information about 3D object shape and position, since the geometric relations among light sources, objects, and shadows share much with those among cameras, objects, and silhouettes. Thus given calibrated light sources, effective constraints on the full 3D object shape and position can be computed by Shape from Shadow just in the similar way as will be described below in shape-from-silhouette methods. Chapter 6, on the other hand, will present a method of estimating 3D shapes and positions of light sources from shadows and shading patterns on calibrated 3D reference object surfaces. Silhouette: A silhouette represents a 2D object shape projected on an image plane, which carries rich information about full 3D object shape and position; the principle of computer tomography proves that an object image can be reconstructed from a group of its projected data. While we cannot apply this shape from the projection method in computer tomography directly to our problem,1 a geometric constraint on full 3D object shape and position can be derived by back-projecting an observed 2D silhouette into the 3D scene to generate a visual cone encasing the 3D object. By integrating visual cones generated from multi-view images, full 3D object shape and position can be reconstructed. This is the principle of Shape from Silhouette. While some concave parts of the object cannot be reconstructed, shape from silhouette works stably without accurate models of lighting or surface reflectance. Stereo: Binocular stereo allows us to perceive 3D scene. A variety of computational methods of Shape from Stereo have been developed in computer vision. The core computational process in shape from stereo rests in matching corresponding points in left and right images. Practical implementations of this appearancebased matching define the degree of stability against variations of object surface reflectance properties and lighting environments. To increase the stability of 1 The projection in computer tomography is characterized as an integral projection where rays go through the interior area of an object and their degrees of attenuation are observed in projected data. In computer vision, most of objects in the real world are non-transparent and reflect light rays at their surfaces. Thus the projection in computer vision implies ray blocking or shadowing.


95

matching, active lighting devices which project high contrast light patterns onto object surfaces are introduced [22]. Such methods are referred as Active Stereo. Note here that shape from silhouette and shape from stereo are complementary to each other; the latter employs object surface properties as visual cues for computing corresponding points in left and right images, while surface boundaries play a crucial role to estimate 3D object shape and position in the former. Motion: Starting computational models for human Shape from Motion perception, computer vision has explored computational methods including pure 3D shape from motion by optical flow analysis between contiguous image frames [35], 3D structure from motion by algebraic analysis of position data of feature points in a dynamic sequence of images such as factorization [52], and so on. As in shape from stereo, establishing correspondences between multiple dynamic images is the core computation process in shape from motion, where practical implementations define the degree of stability against variations of object surface reflectance properties and lighting environments. Focus: Changing a focus of camera lens, blurring patterns in observed images vary depending on 3D surface depth ranges. Employing this principle, many 2.5D depth sensing methods named as Shape from Focus [36] and Shape from Defocus [50] have been developed. As in active stereo, well designed imaging devices allow accurate depth sensing even in real time [37]. While shape from (de)focus methods require high contrast surface textures, they work stably against variations of object surface reflectance properties and lighting environments. Deviating from the scheme of shape from X, computer vision developed a straightforward technology to obtain a 2.5D depth image of an object surface. It is referred as range finding based on time-of-flight (ToF), which measures the timeof-flight of a laser beam between its emission and observation of its reflected beam from an object surface. Many commercial laser range finders are available nowadays. They can be applied only to static objects and scenes because the scanning process of a laser beam takes some time to obtain a 2.5D depth image. In other words, laser range finders are a kind of 0D (point-wise) sensors. To overcome this disadvantage, 2D ToF sensors which realize simultaneous sensing of time-of-flights in a certain area have been developed. Even though their resolution and accuracy are still limited, utilizing such cameras for 3D video production can be another research direction. Among those visual cues listed above, most of 3D video production systems employ mainly shape from silhouette and shape from stereo. The reasons for this are: • Shape from Silhouette works stably without accurate lighting and reflectance models since the appearance of 2D silhouette boundaries are not much affected by light reflection properties on the object surface, and • Shape from Stereo can reconstruct precise 3D object shape even at concave parts once it can establish correspondences based on the object appearance. The three assumptions described at the beginning of the previous section allow us to conduct the appearance-based matching for shape from stereo.

96


• Real lighting environments in multi-view video capture systems are very complex and cannot be actively controlled for 3D shape reconstruction. Thus shape from shading and shape from shadow which require accurate calibration of lighting environments cannot be used. • Shape from texture assumes that surface texture patterns are given a priori, which are actually the information 3D video production systems have to compute. • Human actions to be captured are very complicated. Especially in recording intangible cultural assets such as MAIKO dances, motion patterns and shape deformations of long and loose clothes are too complex to be modeled by algebraic formulations used in shape from motion methods. • Active stereo, shape from (de)focus, and range finders require specially designed imaging devices having limited resolutions compared to ordinary video cameras. The introduction of such special imaging devices into 3D video studios in addition to a group of high resolution video cameras will be realized in the next generation 3D video studio systems [6]. In what follows, we will first give an overview of algorithms for full 3D shape reconstruction of a static object based on shape from stereo in Sect. 4.2.2.1 and shape from silhouette in Sect. 4.2.2.2, followed by integrated algorithms based on both shape from stereo and silhouette in Sect. 4.2.2.3. Then, Sect. 4.2.3 will present algorithms for full 3D shape and motion reconstruction of an object in motion.

4.2.2 Full 3D Shape Reconstruction Intensive explorations of 2.5D shape from X algorithms for over two decades from the 1970s as well as rapid advancements of imaging technologies led computer vision researchers to the exploitation of full 3D shape reconstruction in the middle of the 1990s; it is natural to proceed first from 2D to 2.5D, and then from 2.5D to 3D employing multi-view imaging technologies. Two pioneer studies almost at the same time proposed algorithms for full 3D shape reconstruction of a human in motion based on shape from multi-view stereo and shape from multi-view silhouette, respectively. In 1997, Kanade et al. proposed a stereo-based approach [21]. This is a two-step algorithm which first reconstructs 2.5D shapes (depth maps) for each viewpoint by a conventional stereo method and then merges them to generate a full 3D shape. In this sense, it is a straightforward extension of 2.5D multi-baseline stereo [43] for full 3D reconstruction. On the other hand, Moezzi et al. proposed a silhouette-based approach [34]. The algorithm was designed based on shape from silhouette [27], which reconstructs an approximate full 3D shape of the object from multi-view silhouettes directly. They called it volume intersection and 3D textured volume as 3D video. Note that both of these two methods processed video frames one by one, and hence no 3D motion data were reconstructed or employed in their 3D shape reconstruction methods. Following these two studies, [32] and [49] proposed integrated methods of these two reconstruction cues to achieve both robustness and accuracy. This section


97

addresses characteristics of shape from stereo and shape from silhouette first, and then discusses ideas for their integration.

4.2.2.1 Shape from Stereo Algorithms of full 3D shape reconstruction by stereo or by texture matching can be classified into two types.

4.2.2.1.1 Disparity-Based Stereo The first type employs the two-step approach as proposed by Kanade et al. [21]. It first reconstructs multiple 2.5D shapes (depth maps) from different viewpoints, and then merges them to build a full 3D shape. This approach can employ well-studied narrow-baseline stereo methods as long as cameras are placed densely. However, when the number of cameras is reduced to save costs of devices and calibration or when distantly located cameras as well as near cameras are used for stereo to obtain comprehensive 3D information, widebaseline stereo techniques should be used, which are known to be harder than the regular stereo in producing dense correspondences between multi-view images; appearances of an object in wide-baseline stereo images often change too greatly to find correspondences. In addition, since the first step generates depth maps, i.e. partial reconstructions of the 3D shape, from different viewpoints independently of each other, the second step may encounter inconsistent depth values in overlapping areas among the depth maps. Moreover, per-viewpoint occlusion handling in the first step may generate mutually inconsistent occluding boundaries. To solve these inconsistencies among the depth maps is not a trivial problem. Note that algorithms of merging 2.5D depth maps can also be used for full 3D shape reconstruction from multi-view range sensor data. In fact, Ikeuchi et al. developed a full 3D shape reconstruction system for huge cultural assets like big Buddhas and ancient temples [19], where they proposed a global optimization method to seamlessly merge a large number of multi-view range sensor data. Volumetric 3D shape reconstruction algorithm by Hernandez et al. [14] (described later) can be regarded as an integration of depth fusion and volumetric stereo approach.

4.2.2.1.2 Volumetric Stereo The second type of stereo-based approach reconstructs 3D shape directly from multi-view images. While the first type of algorithms processes pixels as basic entities for appearance-based matching, the second type digitizes the 3D space into small unit cubes called voxels and processes them as basic entities for the full 3D shape reconstruction.

98


Kutulakos and Seitz [26] first proposed this concept as space carving in 1999. In this method, starting from far enough outside of the object surface, voxels are examined one by one sequentially from the outside toward the inside to determine whether they satisfy photo-consistency or not. If a voxel does, it remains as a part of the object surface and the carving process along that direction stops. Otherwise it is carved away. This is based on the assumption that if a voxel is on the object surface, the colors of the pixels corresponding to that voxel in multi-view images should be consistent with each other. Note that this assumption holds for objects with Lambertian reflectance surfaces. By applying this test from the outmost voxels of the scene, the algorithm terminates when all “exposed” voxels are identified. Here the resultant data are called photo hull since it is the largest 3D volume which satisfies the photo-consistency. Under the assumption of Lambertian surface reflectance, this approach is free from the inter-viewpoint inconsistencies involved in the first type. Instead, it involves a serious problem about which multi-view images have to be used for the photo-consistency evaluation of a voxel. That is, if all multi-view images were employed, no voxel would survive from carving. This is because, while some of the light rays connecting an exposed voxel with its projected pixels in multi-view images intersect at the same object surface point, others intersect at different surface points occluding the exposed voxel. Hence, while the photo-consistency holds for the former, it is violated for the latter. This problem is called the visibility of cameras against a voxel and is essentially a chicken-and-egg problem. Without knowing the 3D shape, we cannot determine which cameras can observe a point in the scene irrespectively of self-occlusion. Without knowing the cameras which can observe a 3D point in question, we cannot evaluate its photo-consistency and hence cannot estimate the 3D shape accurately. A practical method to resolve this problem is to introduce a visibility model which approximates the real visibility. We will discuss this point in Sect. 4.3.2 in detail.

4.2.2.2 Shape from Silhouette The concept of 3D shape reconstruction from multi-view silhouettes was proposed by Baumgart first in computer graphics in 1974 [3, 4], and then revisited by Martin in 1983 [31] and Laurentini in 1994 [27] in computer vision. This method is widely called shape from silhouette, as well as visual cone intersection or volume intersection.

4.2.2.2.1 Visual Hull The idea of shape from silhouette comes from the following question. Given a set of object silhouettes taken from different viewpoints, what is the largest 3D shape such that its 2D projections regenerate the given silhouettes?


99

Fig. 4.2 Visual hull in 2D

Fig. 4.3 Visual hull and object concavity

Figure 4.2 illustrates the answer in 2D. Suppose all the cameras are in general positions. Then the shape given by n-view silhouettes forms a 2n-polygon called visual hull. Notice that the silhouettes are 1D line segments, and the visual hull is a 2D polygon in this figure. In contrast to shape from stereo, this process does not involve any matching process across multi-view images. In other words, a set of multi-view silhouettes is mathematically equivalent to the visual hull, and therefore once silhouettes are given, then no additional “decisions” will be required for visual hull computation. This is the biggest advantage of shape from silhouette since obtaining accurate silhouettes is relatively more stable than establishing dense (per-pixel) and accurate correspondences among multi-view images. This advantage holds particularly with a controlled background (for example, chroma-keying) and sparse wide-baseline camera setup. On the other hand, shape from silhouette has three important disadvantages. The first is its limited ability to reconstruct the object shape. Intuitively, the visual hull has difficulties in modeling surface concavities as illustrated in Fig. 4.3. This is because the visual hull can only represent 3D surface shape of parts that appear as 2D silhouette contours. The second is that the visual hull may include some false-positive portions called phantom volumes. As illustrated in Fig. 4.4, the two areas B and C can coexist with the real object areas A and D in terms of the silhouette consistency. Thus silhouettes are necessary but not sufficient conditions, as they introduce false-positives. To solve this problem, Miller et al. proposed a concept of safe hull in which phantoms are carved if the carving does not change the 2D silhouettes of the object [33]. This assumption works well for regular scene, but obviously does not always hold.

100


Fig. 4.4 Phantom volume

The last problem is the lack of robustness against false-negatives in 2D silhouettes. Since the visual hull computation is a logical AND operation, one falsenegative in any single viewpoint can spoil the entire result. This can be very critical if we use large number of cameras which observe an object against different background scenes; the object silhouette extraction is required to be robust against varying background images. Basically increasing the number of cameras makes the visual hull closer to the real object shape, since more shape profiles are likely to be seen as 2D silhouette contours. However, it can also falsely carve the visual hull due to the increased possibility of false-negatives in particular around 2D silhouette boundaries (see “Matting” in Chap. 2). To solve this problem, [5, 42, 62] proposed methods of integrating multi-view silhouette extraction and visual hull construction into a single process.

4.2.2.2.2 Voxel-Based Visual Hull Computation The simplest but widely used implementation of shape from silhouette is a voxelbased carving approach (Fig. 4.5). Suppose the entire volume is decomposed into voxels, then each voxel is carved if its projection onto an image falls outside of the silhouette but in the image frame. If it falls outside of the image frame, i.e. the camera cannot observe the voxel, then the voxel is not carved since the silhouette tells nothing about the scene outside of the image frame. The resultant volume consists of a set of voxels such that all their projected images are observed as parts of silhouettes. Note here that we assume that the camera layout in a 3D video studio is designed so that some of observed images cover entire areas of objects and others include enough spaces to separate silhouettes of multiple objects into disjoint regions. These assumptions allow us to reconstruct 3D shapes of objects as a group of disjoint 3D volumes. Unlike the space carving described before, the test for a voxel can be conducted independently of the others. This means that it can be done in arbitrary voxel order in parallel, and therefore is suitable for parallel processing.


101

Fig. 4.5 Volumetric shape from silhouette

However, if we increase the voxel resolution, the computational cost as well as the required memory space increases in cubic order. For example, while 1 cm resolution for 3 m by 3 m by 2 m space requires 18 M voxels, 1 mm resolution results in 18 G voxels. One straightforward solution is to use compact or sparse representation of the 3D space. For example, Szeliski [51] proposed an octree-based approach in 1993 which achieves both memory and processing time efficiency. Other inefficiency, innate in the naive voxel-based implementation, could be found in its numerous number of computations of 3D to 2D perspective projections. Matsuyama et al. [32] proposed an efficient parallel algorithm which eliminates time-consuming 3D to 2D perspective projections by representing the target 3D volume as a set of parallel 2D planes and directly conducting the back projection of multi-view silhouettes into the 3D space: first a 2D image silhouette is back projected onto one of the parallel planes and then its affine transformations give back projections on the other planes. With this 3D space representation, both the visual cone generation from multi-view silhouettes and the intersection of the cones can be done at each parallel plane independently, which can be implemented in parallel to realize real time 3D shape reconstruction of human actions in about 8 [mm] voxel resolution [60].

4.2.2.2.3 Surface-Based Visual Hull Computation Since the essential 3D shape information is carried by 2D silhouette boundaries, the shape from silhouette can be implemented based on boundary data rather than volume data as follows: first generate 3D surfaces of visual cones from 2D silhouette boundaries and then compute the 3D object surface by intersecting the 3D visual cone surfaces. While the computation of 3D surface intersection may be complicated, the computational complexity can be reduced.

102


Fig. 4.6 Frontier points. A frontier point X is a point on a 3D object surface where the epipolar plane c1 c2 X is tangent to the object surface. This means that the epipolar lines are tangent to the apparent silhouette contours at x1 and x2 in the observed images, respectively, and the frontier point is exposed on the visual hull surface Fig. 4.7 Visual cone by a partially captured object silhouette. The top surface of the 3D cone is false-positively generated by the image boundary

Lazebnik et al. [28] proposed an algorithm of this approach. They first generate a polyhedral representation of a visual cone from each 2D silhouette boundary regarding each boundary pixel as a vertex of a polyhedron and each line connecting a vertex with the projection center as an edge of the polyhedron. Then they compute exact intersections among those polygonal cones. This approach makes the best use of silhouette resolution in pixel, and the computed visual hull represents well fine structures in 2D silhouettes. However, their cone-to-cone intersection process depends on accurate estimation of frontier points. Here, a frontier point is a point on a 3D object surface where the epipolar plane is tangent to the object surface (Fig. 4.6). In such cases, the epipolar lines are tangent to the apparent 2D silhouette contours (x1 and x2 in Fig. 4.6), and hence the frontier point is exposed on the visual hull surface. From a computational point of view, the estimation of a frontier point is a process of finding a pair of epipolar lines tangent to the 2D silhouette contours. This process is not practically trivial especially with noisy silhouettes. Franco and Boyer [9] proposed another algorithm without computing frontier points explicitly. It is more robust against noise, and runs reasonably fast. While these methods can solve the explosion of the computational cost and reconstruct a visual hull representing well the 2D silhouette resolution, they rely on the assumption that all cameras observe entire areas of objects. As discussed in Chap. 2, we often want to employ some zoom-up cameras to increase image resolutions for a specific object area such as a human face. For example, see Fig. 4.27,


103

where most of the multi-view images do no cover the entire object. With such cameras, then, captured images do not cover the entire area of an object, which disables the application of the surface-based visual hull computation methods. Moreover, this problem of “partial capture” is often encountered when an object is allowed to move in a wide area as discussed in Chap. 3. Suppose one camera cannot capture the entire object in its image frame. Then, we obtain an apparent silhouette boundary on the image boundary (Fig. 4.7). How to cope with such fake boundaries is not a trivial issue for the surface-based visual hull computation methods.

4.2.2.3 Shape from Stereo and Silhouette As discussed so far, shape from stereo and shape from silhouette work complementary to each other. Shape from stereo can reconstruct the complete 3D object shape if appearance-based texture-matching or photo-consistency and visibility tests work accurately. (See Sect. 4.3.2 for detailed discussions on the visibility examination.) Such conditions, however, are unlikely to be satisfied without knowing a reasonable estimation of the object surface geometry and reflectance. On the other hand, shape from silhouette can reconstruct the 3D object shape as the visual hull without depending on critical conditions, while the reconstructed 3D shape is not only an approximation of the real shape but also erroneous phantom volumes may be included. Thus, recent studies integrate these two 3D shape reconstruction algorithms. A simple integration is to form a two-step approach which first utilizes shape from silhouette to generate an initial guess of the 3D shape and then feeds it to a shape from stereo algorithm. One drawback of this method is that the silhouette constraint is not used in the second step, which loosely follows the shape suggested by the visual hull but does not guarantee the reconstructed shape to coincide with the 2D multi-view silhouettes. For 2.5D shape reconstruction, Fua and Leclerc [10] proposed an algorithm which combines stereo, shading, and silhouette cues. It models the 2.5D surface by a triangular mesh and deforms it so as to satisfy the constraints derived from these three visual cues. This active surface modeling can be regarded as an extension of 2D geometric Snakes [23] into 2.5D shape. Section 4.4.1 will present an active full 3D mesh-deformation method to integrate surface texture and silhouette constraints into a unified computational process. Before concluding this section, it should be noted that 3D shape reconstruction algorithms introduced here can work even when multiple objects are in a scene. Their successful segmentation, however, depends on the camera layout, object shapes, and their mutual positions. In case of 3D video production of human actions, cameras looking straight downward from the ceiling facilitate the object segmentation. Thus, if not specified explicitly, we assume the 3D shape of each object is well segmented from the others and object-wise processing can be done.

104


4.2.3 Dynamic Full 3D Shape Reconstruction for 3D Video Production Regardless of what visual cues are employed, the algorithms reviewed so far are for the full 3D shape reconstruction of a static object. Their straightforward application to 3D video production is to reconstruct 3D shape from each frame of observed multi-view video data. This simple frame-wise 3D shape reconstruction, however, obviously misses an important reconstruction cue, i.e. motion, temporal continuity, or consistency. Consequently, it is natural to augment the algorithms by introducing dynamical properties. In fact, a simultaneous 3D shape and motion reconstruction by a mesh-deformation method [39] and a simultaneous 3D shape sequence reconstruction method [12] were developed. Note here that the former explicitly estimates 3D motion vectors between consecutive 3D mesh data, while the latter does not involve such 3D motion vector estimation. Figure 4.8 illustrates the categorization of 3D shape reconstruction methods from multi-view video data. First they are categorized based on their reconstruction target data, and then characterized according to their reconstruction cues. Notice here we use the term motion as equal to inter-frame correspondence of 3D object surface points, and motion estimation as establishing correspondences between such points.

4.2.3.1 Frame-Wise Shape Reconstruction The first group reconstructs a single 3D shape from multi-view video frame images. In this approach, multi-view video data are decomposed into per-frame multi-view images and processed independently. Since the input is now merely a set of multiview images, this group includes a variety of algorithms for non-3D video purpose, e.g. static object reconstruction [15, 58], and as described in Sect. 4.2.2. Notice that appropriate camera settings (Chap. 2), e.g. dense camera arrangement, can be designed for a static object and high quality multi-view images can be observed. Since some static 3D shape reconstruction methods assume such well designed camera settings, their direct application to multi-view video data may not work well; relative viewing positions and directions toward objects dynamically change as objects move. Recall that a group of active cameras are employed to cope with this problem in Chap. 3. While this frame-wise 3D shape reconstruction is the most popular method,2 one should notice that data structures for representing 3D surface shape data vary a lot even if a sequence of 3D shape data themselves vary smoothly. In fact, vertex–edge structures of 3D mesh data, which are the most widely used data structure to represent 3D video, greatly change frame to frame because their computation processes are conducted independently of others. Thus 3D video application methods should be designed taking into account such temporal variations of the vertex–edge mesh 2 The

reasons for this will be discussed later in this chapter.


105

Fig. 4.8 Categorization of 3D shape reconstruction methods for 3D video production. Top: frame-wise reconstruction approach. Middle: simultaneous shape and motion reconstruction approach. Bottom: multi-frame 3D shape reconstruction approach

structure as will be described in Part II in this book. Section 4.4.2, on the other hand, will present a mesh-deformation method that keeps the vertex–edge structure of the mesh over a sequence of multi-view video frames. 4.2.3.2 Simultaneous Shape and Motion Reconstruction The second group estimates 3D shape and motion simultaneously from multi-view images of a pair of consecutive frames. It includes two approaches: by deformation [39] and by matching [55]. The first approach tries to find a deformation from the 3D object shape at frame t to that of t + 1. Once a deformation is conducted, it yields the 3D shape at t + 1 as well as 3D motion from t to t + 1 represented by dense correspondences or deformation trajectories of points on 3D surfaces. Section 4.4.2 will present a detailed implementation and performance evaluation of this approach.

106


The second approach tries to find a pair of corresponding 3D points in a pair of consecutive frames which satisfies the constraint that the 3D points at frame t and t + 1 should be at the same location on the 3D object surface. A pair of 3D points satisfying the constraint should be spatially photo-consistent across multi-view images at t and t + 1, respectively, as well as temporally photo-consistent across two consecutive frames. Vedula et al. [55] proposed an algorithm which formalizes this idea as a voxel carving in 6D space, a Cartesian product of two 3D voxel spaces corresponding to a pair of consecutive frames. Note that correspondences between 3D surface points over time can facilitate later processing in texture generation and 3D video coding.

4.2.3.3 Multi-Frame 3D Shape Reconstruction The third group reconstructs a sequence of 3D shape data simultaneously from multi-view video data. This approach has a chance to “interpolate” ambiguous or unreliable 3D reconstruction at a certain frame by its neighboring frames automatically. This is a kind of long-term temporal coherency constraint on the object 3D shape. Goldluecke and Magnor [12] proposed an algorithm which extracts 4D hyper surface from 4D (volume + time) space representing a temporal sequence of 3D shape data. With this approach erroneous portions around self-occluded areas were successfully carved out. Note that this algorithm does not establish temporal correspondences between 3D shape data at neighboring frames.

4.3 Design Factors of 3D Shape Reconstruction Algorithms Through a comprehensive review of 3D shape reconstruction algorithms in the previous sections, we identified three essential factors for designing 3D shape reconstruction algorithms for 3D video production: photo-consistency evaluation, visibility and occlusion handling, and shape representations for optimization. In what follows, detailed discussions on each design factor will be given.

4.3.1 Photo-Consistency Evaluation The concept of photo-consistency is first introduced by Kutulakos and Seitz in 1999 [26].3 Suppose we are going to evaluate the photo-consistency at a 3D point p. The photo-consistency of p is given by comparing multi-view images at projections of p. This process involves two major computational factors. The first is how to 3 An

earlier version can be found in the voxel coloring study by Seitz and Dyer in 1997 [44].

4.3 Design Factors of 3D Shape Reconstruction Algorithms

107

Fig. 4.9 Photo-consistency evaluation. The photo-consistency at the central white mesh point is evaluated by computing similarity measures among areas in multi-view images denoting 2D projections of the 3D patch window defined at the mesh point

determine which cameras can observe p, which is called visibility test. We will describe it in detail in Sect. 4.3.2. The second is how to evaluate similarity measures among image projections of p in multi-view images, which will be discussed here. Let us assume a set of cameras C that can observe p. Then, if we compare the pixel colors at projections of p, we can express the evaluation function of their photo-consistency as ρ(p, C). Comparison of single pixel colors is not so discriminative or stable. Instead a window-based method can be helpful. To realize geometrically consistent image window generation, first a planar 3D rectangular patch centered at p is generated and the patch direction is aligned with the surface normal at p. Then, the patch is projected onto the image planes of C to determine image windows for their similarity computation (Fig. 4.9). This computation process requires the surface normal n and curvature κ at p, which define the shape and texture of the 2D image window for each viewpoint in C. Consequently the photo-consistency evaluation function can be expressed as ρ(p, n, κ, C). Here we come up with a chicken-and-egg problem; the photo-consistency evaluation for 3D shape reconstruction requires 3D shape properties. A practical solution is to introduce approximation and/or assumption. For example, the normal n can be approximated by using the visual hull. In this case n of point p is approximated by the normal of such point that is on the visual hull and closest to p. Another possible approximation is to assume that n is oriented to the direction of the observable camera closest to p. The observable camera determination will be discussed later in Sect. 4.3.2. The curvature κ (or the 3D patch size) can be given a priori as knowledge on the target or automatically adjusted by analyzing the texture edge features. For example, if projections of p have rich/poor textures then we can make κ smaller/larger, respectively.

4.3.1.1 Photo-Consistency Evaluation by Pair-Wise Texture Matching One possible implementation of the photo-consistency evaluation is to integrate multiple pair-wise texture-matching results into a single score. If we follow a simple

108


averaging strategy, then the photo-consistency evaluation function is given by ρ(p, n, κ, C) =

1 ρ2 (p, n, κ, c1 , c2 ), |C|

(4.1)

c1 ,c2 ∈C

where |C| denotes the number of cameras in C, and ρ2 (·) computes a pair-wise texture similarity/dissimilarity measure such as Sum-of-Absolute-Difference (SAD), Sum-of-Squared-Difference (SSD), Normalized-Cross-Correlation (NCC), Zeromean SAD (ZSAD), Zero-mean SSD (ZSSD), Zero-mean NCC (ZNCC), Census, defined by SAD(q c1 , q c2 ) = |q c1 − q c2 |1 ,

(4.2)

SSD(q c1 , q c2 ) = q c1 − q c2 ,

(4.3)

q c1 · q c2 , q c1 · q c2 ZSAD(q c1 , q c2 ) = (q c1 − q¯ c1 ) − (q c2 − q¯ c2 )1 , ZSSD(q c1 , q c2 ) = (q c1 − q¯ c1 ) − (q c2 − q¯ c2 ), NCC(q c1 , q c2 ) =

(q c1 − q¯ c1 ) · (q c2 − q¯ c2 ) , q c1 − q¯ c1 · q c2 − q¯ c2 Census(q c1 , q c2 ) = Hamming T (q c1 ), T (q c1 ) ZNCC(q c1 , q c2 ) =

(4.4) (4.5) (4.6) (4.7) (4.8)

where qc = qc (p, n, κ) denotes pixel values in the projected window for camera c ∈ C: the 3D patch window defined at p is rasterized into an m-dimensional vector. | · |1 and · denote l-1 and l-2 norms, respectively. q¯ c is a vector such that its size is equal to that of q c and all elements are set to the average value of the elements of q c , i.e. |q c |1 /m. Hamming(·) denotes the Hamming distance of two binary vectors. T (·) denotes the Census transform T (q c ) =

m ξ q c (i), qˆ c ,

(4.9)

i=1

ξ(a, b) =

0 1

if a ≤ b, if a > b,

(4.10)

where denotes a bit-wise catenation, and qˆ c denotes the pixel value of the image window center position, i.e. the projection of p on the screen of c. ξ is a binary or magnitude comparator. Notice that NCC and ZNCC measure similarities (their values range from 0 to 1 with NCC and −1 to 1 with ZNCC, larger is better) while the others measure dissimilarities (smaller is better, 0 is the best).


109

4.3.1.2 Photo-Consistency Evaluation by Variance Another implementation of the photo-consistency evaluation function is to compute the variance of colors among C [17, 26]. In this implementation, the photoconsistency evaluation function is given as follows: ρ(p, n, κ, C) =

1 (qc − q˜C ), |C| − 1

(4.11)

c∈C

where q˜C denotes the average of qc (·), that is, q˜C =

c∈C qc /|C|.

4.3.1.3 Robustness Against Camera Characteristics and Surface Reflectance Properties The choice of the photo-consistency evaluation methods should reflect assumptions on input multi-view video data and object surface properties. The use of the variance-based photo-consistency evaluation or SSD/SAD for pair-wise texture matching implies that camera characteristics are reasonably identical and the object surface is well modeled as Lambertian. NCC implies that camera characteristics are identical up to per-camera scale factors and the object surface is Lambertian. ZSAD, ZSSD and ZNCC add a tolerance for per-camera biases to SAD, SSD, and NCC, respectively. Census is more robust against per-camera scale factors/biases as well as non-Lambertian specularities because it is sensitive only to differences between pixel values in an image window and that of the center pixel. The variance-based photo-consistency evaluation can also be robust against specularities by discarding the brightest color from visible cameras C. Notice that here exists a trade-off between the discriminability and tolerance to variations of camera characteristics and object surface properties. Census is more robust against non-Lambertian surfaces and poorly color-calibrated cameras, but less discriminative than SSD/SAD.

4.3.2 Visibility and Occlusion Handling As described in the previous section, the photo-consistency evaluation requires knowledge about the visibility of objects. For example, as illustrated in Fig. 4.10, the photo-consistency evaluation of a point p on the object surface should be done by using only images observed from c2 and c3 which can observe p. The other images observed from c1 and c4 must be excluded since p is occluded by the object itself. False visibility decision, such as regarding c1 as a visible viewpoint of p, misleads the photo-consistency evaluation. With the 3D object shape, we can determine the visibility even for a point not on the object surface, q (here q is visible from c1 , c2 , and c3 ), which allows us to eliminate q from possible object surface points through photo-consistency evaluation.

110


Fig. 4.10 Physical visibility of object surface. Given the 3D object shape, the visibility from any viewpoint points can be correctly computed even for a point not on the object surface

Such physical visibility, however, can be determined only when an accurate 3D object shape is given. To solve this chicken-and-egg problem, some visibility models were proposed in literature.

4.3.2.1 State-Based Visibility The state-based visibility [48, 58] approximates the visibility of a point p by the visibility of the nearest point p on a shape proxy Sp of the real object. In many cases the visual hull of the object serves as the proxy. The visibility of p is determined by the standard Z-buffering technique in computer graphics with the proxy shape and camera arrangement. The occlusion Qc for camera c in the computational world model (Fig. II.2) is approximated as Qc ≈ Qc (p, Sp ) ≈ Q p , Sp visible if dc (p ) ≤ Dc (p , Sp ), = occluded otherwise,

(4.12) (4.13) (4.14)

where dc (p ) is the depth (z value) of p in the camera c coordinate, and Dc (p , Sp ) is the nearest depth of Sp at p in the camera c coordinate. For example, the visibility of p in Fig. 4.11 is approximated by that of p , the nearest point of p on the proxy surface, and the state-based visibility evaluation reports that c2 and c3 can observe p since they can observe p . The state-based visibility with a visual hull works well as long as the visual hull approximates well the real object shape. Suppose the visual hull coincides with the


111

Fig. 4.11 State-based visibility. The visibility of a point p is approximated by that of p , the surface point on the visual hull closest to p

object shape. In this case no visibility tests on the visual hull surface return falsepositives; if a camera is marked as visible for a point on the visual hull surface, the camera can observe the point definitely. As long as the visual hull does not occlude a camera, there exist no other occluders in the scene by definition, and therefore the camera can observe the object definitely. On the other hand, there may exist cameras which are false-negatively reported as “not visible”. This is because the visual hull may include phantom volumes which falsely occlude itself. For example, while the camera c2 of Fig. 4.12 can observe both points q and q physically, the state-based visibility evaluation using the visual hull reports that c2 cannot observe q because of the occluding phantom volume in the middle. Notice here that we cannot predict the existence of false-negatives since we do not know which portions are phantom volumes.

Fig. 4.12 False self-occlusions by phantom volumes

112


Fig. 4.13 False visibility by the state-based visibility evaluation. Point p is physically observed by c1 and c2 . However, the state-based visibility evaluation approximates the visibility of p by p , and returns c2 , c3 , c4 and c5 as visible

An important fact here is that the above-mentioned observations are valid only for the points on the visual hull surface. That is, the state-based visibility may not work for parts where the visual hull cannot model well the object shape, especially around concavities as shown in Fig. 4.13. In this figure the point p on the real object surface is not exposed as a part of the visual hull surface. The state-based visibility evaluation approximates the visibility of p by that of the point p , the closest point to p on the visual hull surface, and returns that the camera c4 is the most front-facing one, and then c3 , c5 , and c2 . However, this evaluation result is not acceptable, since only c1 and c2 can physically observe the surface point p. To cope with limitations of the state-based visibility evaluation, it makes sense to employ an iterative strategy which refines both the shape and visibility starting from the visual hull. We introduce an algorithm following this idea later in Sect. 4.4.1.

4.3.2.2 Oriented Visibility The oriented visibility is another representative visibility evaluation method introduced by Lempitsky et al. in 2006 [29]. Given the position and normal direction of a point in the scene, this method determines its visibility based only on the camera arrangement. The occlusion Q of camera c for a point at p is given by using a hypothesized surface normal n as Qc ≈ Qc (p, n) visible if n · dc < θ and (p − oc ) · dc > 0, = occluded otherwise,

(4.15) (4.16)

where dc and oc denote the viewing direction from c to p and the position of c, respectively, and θ is a threshold. If θ = −0.5, then cameras observing p within an angle of 0 (front-facing) to 60 are accepted. Note that the normal n is given (1) as a result of reconstruction area digitization (Sect. 4.3.3.3), or (2) by using a shape proxy (e.g. visual hull). In the first scenario, this model works without intermediate reconstructions or guesses of the object shape, but cannot handle self-occlusions. Therefore this is useful when no


113

reasonable initial guesses are available for the object shape. Once a reliable intermediate reconstruction is obtained, then it is better to switch to the state-based visibility evaluation as suggested by Lempitsky et al. In the second scenario, this works as a modified state-based visibility which is robust to false self-occlusions. For example, the normal of q in Fig. 4.12 is approximated by that of q , say n , and then the oriented visibility Qc (p, n ) returns “visible” for all c1 , c2 and c3 though the state-based visibility falsely reports that c2 is not visible. 4.3.2.3 Integrated Evaluation of Photo-Consistency and Visibility Another idea is to utilize the photo-consistency measure itself for the visibility evaluation. Consider the situation illustrated in Fig. 4.9. If the image windows for a set of cameras C are photo-consistent, the surface point in question is likely to be observable from these cameras as well as to be at the correct position and orientation. Otherwise, it is suggested that the surface position, normal, and/or the visibility is wrong. In such cases, we can simply discard the hypothesized surface point or try to test the photo-consistency using a subset of C. Patch-based surface growing approaches described later in Sect. 4.3.3.3 follow the former strategy. The latter strategy was proposed by Vogiatzis et al. [56] which proposes an occlusion robust photo-consistency evaluation algorithm. If a subset of C forms a photo-consistent set of cameras for the surface point in question, then the unused cameras are labeled as invisible for that point. This approach works well for dense camera setups because the subset of C must have a sufficient number of cameras for reliable photo-consistency evaluation.

4.3.3 Shape Representation and Optimization Three major models can be used to represent the 3D object shape in 3D video production: volume-based, surface-based, and patch-based representations. These representations have their own advantages and disadvantages as well as well-matched optimization methods. In particular the first two representations are widely used with discrete optimization methods based on graph-cuts [25] and belief propagation [8]. A great advantage of these methods is that the resultant 3D shape is guaranteed to be optimal. In what follows, we will discuss three models of 3D shape representation together with 3D shape optimization methods based on the individual models.

4.3.3.1 Volume-Based Shape Representation In the volume-based approach, 3D shape is represented by a set of voxels. The goal of 3D shape reconstruction is then to find binary occupancies of voxels that satisfy the photo-consistency.

114


This binary optimization suits well with computational schemes based on the graph-cuts [25]. In general finding a min-cut of a graph G = {V , E} is equivalent to finding the global minimum of the following objective function: Ed (lv ) + Ec (lvi , lvj ). (4.17) E (l) = v∈V

vi ,vj ∈E

Here v denotes a vertex representing a voxel and lv its binary label: occupied or not occupied by an object. The first term represents a per-voxel objective function which returns a cost to label v as lv and the second term represents a pair-wise objective function which returns the cost to label two connected voxels vi and vj as lvi and lvj , respectively. Vogiatzis et al. [58] first proposed a method for the volumetric reconstruction of full 3D shape with the graph-cuts. In their approach the first term was used to represent a shape prior and the second term for photo-consistency values: vi + vj , (4.18) E =− λ+ ρ 2 v∈Vo

vi ,vj ∈Eo

where Vo and Eo denotes the set of graph nodes (voxels) and edges (connected voxels) of the estimated object. In this formalization the object is estimated as the collection of voxels labeled source side. Notice that λ is a positive constant. Use of larger λ indicates that a larger object shape is preferred. In this sense, λ serves as a ballooning term similarly to Snakes [23]. If λ ≤ 0, the minimization of E simply returns a trivial solution Vo = ∅ and hence Eo = ∅ and E = 0. An important advantage of this method is its flexibility to search in a wide range of 3D shape data with greatly different topological structures. Since it can represent arbitrary 3D shapes (up to the voxel resolution, of course) in terms of the binary occupancy map, a wide range of 3D shapes are evaluated to find the optimal one through the optimization. This means that the above-mentioned method can reconstruct the object 3D surface with the correct surface topology even if the topological structure of the visual hull is different from the real one typically because of phantom volumes. On the other hand, the surface-based representation discussed in the next section has a difficulty in managing such global topological changes. The voxel-based graph-cuts method, however, has two disadvantages: memory consumption and continuity constraint embedding. The first problem is shared by the shape from silhouette methods presented in Sect. 4.2.2.2, and its straightforward solution is to follow a coarse-to-fine or hierarchical strategy [14]. The second problem is an essential limitation of the graph-cuts. Since the graph-cuts algorithm has a strong limitation on the objective functions, it is relatively hard to embed continuitybased constraints such as curvature constraints which are defined over more than two nodes (voxels). One possible solution is to convert a graph with high-order cliques (n-ary relations) into a first-order graph with up to binary relations as proposed by Ishikawa in 2009 [20], but it exponentially increases the graph size and therefore memory consuming.


115

4.3.3.2 Surface-Based Shape Representation In contrast to the volume-based representation, the surface-based approach does not digitize the 3D volume but 3D (connected) mesh data are employed to represent the 3D object surface. Suppose we have an initial guess of the object surface, which is usually given by 3D mesh data transformed from the visual hull. The goal of 3D shape reconstruction is then to find a deformation of this mesh such that the deformed mesh satisfies photo-consistency over the entire object surface. One naive implementation of such deformation can be designed based on the gradient descent optimization. Starting from an initial guess, it searches for a point with a better photo-consistency value for each surface point toward the inside of the surface. It deforms the surface toward these points until it converges. Recall that the deformation should proceed toward inside if we start from the visual hull. This approach fits well with the state-based visibility strategy since it can update the visibility at each deformation step, but obviously it can result in a locally optimal solution. Sections 4.4.1.1 and 4.4.2 will present detailed mesh-deformation algorithms for a static object and an object in motion, respectively. This problem can also be rendered as a multi-label optimization. It digitizes possible deformation destinations of each surface point on the initial shape, and then finds a (semi-)optimal combination of digitized destinations [57]. By considering the digitized destinations as the label-set for each vertex, an optimal solution can be found by standard techniques such as graph-cuts, belief-propagation, or tree-reweighted message passing. By iteratively optimizing the deformation and updating the state-based visibility, we can expect that the process converges to a (semi-)optimal result. Advantages of this representation are its limited memory consumption and continuity constraint embedding in contrast to the volume-based representation. It can represent 3D shape in finer resolution with less memory and smoothly introduce a continuity constraint into its computational process since surface normal, curvature, and locally connected neighbors can easily be computed from the surface representation. The disadvantage is its poor flexibility in searching in a wide range of different 3D shapes, which is the advantage of the volume-based representation. As long as we define the surface deformation as transforming each point on the surface to another position, the surface cannot change its global topology or mesh structure inherited from the initial shape. To solve this problem, Zaharescu et al. proposed an algorithm for topology adaptive mesh deformation [61].

4.3.3.3 Patch-Based Shape Representation The patch-based approach represents a 3D surface by a collection of small patches (or oriented points). This representation can be seen as twofold integrations: (1) integration between the volume-based and surface-based shape representations, and (2) combination of the oriented and state-based visibility evaluations. It can conduct

116


Fig. 4.14 Tetrahedral voxel space [29]. The entire volume space is first decomposed into unit cubes, and then each cube is decomposed into 24 tetrahedra. One of such tetrahedra is denoted in red

search in a wide range of surface topological structures like the volume-based representation, and it can employ surface continuity constraint as in the surface-based representation. The patch-based representation can be implemented with two different computation schemes. One is a “surface growing” approach and the other is a global optimization approach. The surface growing approach [11, 13] takes a best-first search method and starts the surface estimation by finding seed patches with higher confidence in their existence and accuracy. Then it searches for neighboring patches around the seeds. As for the visibility evaluation, this approach starts the evaluation with the oriented visibility first, and then gradually switches to the state-based visibility as the surface grows. One disadvantage of this approach is the lack of “closed surface constraint”. While both the volume-based and surface-based representations can reconstruct closed surfaces anyway, the surface growing approach is not guaranteed to generate such a closed 3D surface. Thus, as a post processing, we should employ some “meshing” algorithms such as Poisson surface reconstruction [24] or moving-leastsquare (MLS) surface [1] which generate smoothly connected surfaces from a cloud of oriented points filling out small gaps and resolving intersections, and/or overwraps. The other approach utilizing the graph-cuts for global optimization was introduced by Lempitsky in 2006 [29]. This approach digitizes the target volume into a collection of tetrahedra, and tries to find the best combination of their faces that represents the object 3D surface. In other words, it generates all possible triangular patches in the target space and searches for their best combination in terms of the photo-consistency. Suppose the entire volume is digitized into the tetrahedral voxel space [29]. Figure 4.14 illustrates how a cubic voxel is decomposed into 24 tetrahedra. Then, let us form a graph where nodes denote the tetrahedra. Each node has n-links to their four neighbors, and s- and t-links to the source and sink. The photo hull in this algorithm is given by a surface which minimizes

E= λ dV + ρ dA. (4.19) V

S


117

Fig. 4.15 Local graph structure for minimizing Eq. (4.20). The red triangle indicates one tetrahedral voxel of Fig. 4.14. Each tetrahedral voxel corresponds to a node of the graph. It has four directed edges called n-links connecting with its neighbors sharing one of its surface triangle. Note that s- and t -links of each node are omitted

The first term represents a ballooning term as a shape prior working over the estimated volume V , and the second term represents photo-consistency measures over the estimated surface S. Its discrete representation is given by v i + vj E= (4.20) , nij , λ+ ρ 2 v∈V

vi ,vj

where vi , vj is a pair of neighboring tetrahedra such that the surface S passes v +v v −v through their boundary triangular “patch” at i 2 j with normal nij = |vjj −vii | . This patch is used for oriented-visibility and photo-consistency evaluations. By assigning the first term as s-link weights and the second term as n-link weights (Fig. 4.15), the min-cut of the graph is exactly equivalent to the minimization of Eq. (4.20) and the surface is given as the source-side tetrahedra. Notice that the n-links are directed edges. The link weight from vi to vj represents the photo-consistency value of a v +v patch located at i 2 j and directed from vi to vj , and the link weight from vj to vi represents the photo-consistency value of the patch located at the same position but in the opposite direction. This ensures that the min-cut always returns a closed 3D surface that has the best photo-consistency. In contrast to the surface growing approach, min-cuts of this graph always generate a closed 3D surface. However, it is not trivial to embed a surface smoothness or continuity constraint due to the limitation of graph-cuts formalization, as discussed for the volume-based representation.

4.3.3.4 Notes on Optimality Both global optimization techniques, graph-cuts and belief propagation, are known to produce a (semi-)optimal result of the objective function. However, this does not mean the output is the best reconstruction of 3D object shape from multi-view video data. It only ensures that the output shape returns the best score in terms of the specified objective function. That is, the objective function essentially includes the photo-consistency evaluation score computed based on the visibility, while its

118


correct value cannot be computed without the exact 3D shape of the object. Hence if the visibility evaluation relies on an approximated 3D shape, then the optimization of the objective function may not lead us to the best photo-hull estimation. Consequently, it should be carefully examined in what sense the output is the best.

4.4 Implementations and Performance Evaluations of 3D Shape Reconstruction Algorithms Here we present several practical 3D shape reconstruction methods implemented taking into account the three design factors discussed in the previous section. First we introduce a pair of 3D shape reconstruction methods for a single frame: surfacebased mesh-deformation and volume-based graph-cuts methods. Then we augment the former to realize simultaneous 3D shape and motion reconstruction. The performances of these methods are evaluated with experiments using real world objects in motion including a dancing MAIKO.

4.4.1 3D Shape from Multi-view Images 4.4.1.1 Intra-frame Mesh Deformation Algorithm As discussed so far in this chapter, 3D shape reconstruction algorithms are characterized by their reconstruction cues, shape representation methods, visibility models, photo-consistency evaluation functions, and optimization methods. Here we introduce an intra-frame mesh-deformation approach, which • uses the visual hull as the initial guess, • represents 3D shape by a triangular mesh surface, • deforms the surface so as to satisfy the ZNCC photo-consistency with state-based visibility as well as a silhouette constraint, and • renders the deformation as an iterative multi-labeling problem solved by loopy belief propagation. Suppose we start the deformation from the visual hull surface represented by a 2-manifold closed triangular mesh structure M = {V , E} where V and E denote the set of vertices and edges in M (Fig. 4.16). The goal is to find a set of vertex positions that are optimal in terms of photo-consistency and two additional constraints preserving the initial mesh structure. 4.4.1.1.1 Frame-and-Skin Model We employ the following three types of constraint to control the mesh deformation. 1. Photometric constraint: A patch in the mesh model should be placed so that its texture, which is computed by projecting the patch onto a captured image, is consistent, irrespectively of the image it is projected on.

4.4 Implementations and Performance Evaluations of 3D Shape Reconstruction

119

Fig. 4.16 3D mesh model. Left: a triangular mesh model. Right: 2- and 4-manifold meshes

Fig. 4.17 Frame-and-skin modeling of the 3D object shape

2. Silhouette constraint: When the mesh model is projected onto an image plane, its 2D silhouette should coincide with the observed object silhouette on that image plane. 3. Smoothness constraint: The 3D mesh should be locally smooth and should not intersect with itself. These photometric, silhouette and smoothness constraints together define a frame-and-skin model [38] which represents the original 3D shape of the object (Fig. 4.17(a)) as follows: • First, the silhouette constraint defines a set of frames of the object (Fig. 4.17(b)). • Then the smoothness constraint defines a rubber sheet skin to cover the frames (Fig. 4.17(c)). • Finally, the photometric constraint defines supporting points on the skin that have prominent textures (Fig. 4.17(d)). In what follows, we describe energy functions at each mesh vertex to satisfy the constraints.

4.4.1.1.2 Photometric Constraint We define an energy function Ep (v) which represents the photometric constraint of a vertex v ∈ V as follows: Ep = Ep (v), (4.21) v∈V

120


Fig. 4.18 Silhouette constraint. ©2008 IEICE [41]

Fig. 4.19 Contour generator and apparent contour

Ep (v) = ρ(pv , n˜ v , κ, C˜ v ),

(4.22)

where ρ(·) is a photo-consistency evaluation function which returns smaller values for better matching. pv denotes the current position of v. n˜ v and C˜ v denote the normal and cameras that can observe v at its initial state. This means that the initial mesh shape serves as the shape proxy for the state-based visibility. In our iterative reconstruction (described later), we first use the visual hull as the initial mesh, and then use the result of the previous iteration for the initial mesh of the next iteration. κ is the local curvature at v given a priori.

4.4.1.1.3 Silhouette Constraint The second constraint is the silhouette constraint which restricts the deformation to preserve the 2D silhouette outlines observed by the cameras (Fig. 4.18). The constraint enforcement is realized by finding contour generators on the object surface from multi-view images first, and then by embedding these points as anchors in the deformation. Contour generators consist of points on the object surface such that their surface normal vectors are tangent to viewing rays passing through the camera center and their corresponding 2D silhouette contour points, respectively (Fig. 4.19). In other


121

Fig. 4.20 Contour generator and camera arrangement

words, each point on the 2D silhouette outline corresponds to at least one 3D point on the object surface along the viewing line connecting it with the camera center. Note here that since the visual hull sometimes just clips the viewing line (Fig. 4.20), the 3D position of some contour generator points cannot be determined uniquely. To solve this problem, we employ dynamic programming to estimate the optimal contour generators while considering continuity (smoothness) between estimated points as follows [53]. Suppose we have N cameras Ci (i ∈ [1, N]) and a binary silhouette image Si for each Ci . We assume the outlines of Si are given as a set of 2D curves. We denote the j th outline by si,j , and the xth pixel of si,j by si,j (x) (x ∈ [1, Nsi,j ]) where Nsi,j is the number of pixels on si,j . Every 2D point of silhouette outlines si,j (x) has one or more corresponding 3D points {Pi,j (x)} lying on the object surface. We can expect that each has a high photo-consistency value with camera image pixels since it is 3D points on the real 3D object surface, and is exposed as a point on the visual hull surface. Hence the object visual hull gives the possible 3D positions of contour generators for each si,j (x). We formalize the contour generator estimation problem as an energy minimization problem of a function Esi,j defined as follows: Esi,j =

x

ρ(px ) + λ

Ed (px , px+1 ),

(4.23)

x

where px denotes the selected 3D point from {Pi,j (x)} corresponding to si,j (x), ρ(px ) the photo-consistency term at px as defined in the previous section, and Ed (px , px+1 ) the distance between px and px+1 representing a smoothness term. Since we assumed that si,j forms a curve and parameterized by a single variable x, the smoothness term of x depends only on its neighbor x + 1. Hence this minimization problem can be solved efficiently by dynamic programming whereas a min-cut

122


Fig. 4.21 Smoothness constraint as a spring model

or belief-propagation framework would be computationally expensive. We denote by Vcg ⊂ V the optimal set of mesh vertices, each of which is closest to {px }, that minimizes Esi,j . With Vcg , we define an energy function which represents the silhouette constraint as follows: Es (v), (4.24) Es = v∈Vcg

Es (v) = pv − p˜ v 2 ,

(4.25)

where pv and p˜ v denote the current position and the original position of vertex v ∈ Vcg . That is, p˜ v is the closest position to px which minimizes Ecg . 4.4.1.1.4 Smoothness Constraint The last constraint is the smoothness constraint defined for each neighboring vertex pair. The energy function for it is simply defined by the distance between the two vertices vi and vj as follows: Ec = Ec (vi , vj ), (4.26) vi ,vj ∈E

2 Ec (vi , vj ) = pvi − pvj 2 − p˜ vi − p˜ vj 2 ,

(4.27)

where vi , vj ∈ E denotes an edge in the mesh connecting vi and vj , |p˜ vi − p˜ vj |2 is the original distance of two vertices vi and vj in the initial state. This is a kind of elastic energy model where each vertex is connected by springs to its neighbors (Fig. 4.21).

4.4.1.1.5 Formalization as a Multi-labeling Problem Now we are ready to render the deformation as an iterative multi-labeling problem. The goal is to find a combination of vertex positions which minimizes E = E p + Es + Ec .

(4.28)


123

By assuming that each vertex of the initial mesh can move towards the opposite direction of its surface normal and that their movements are digitized with a certain unit distance, we can represent next possible positions of each vertex by K labels indexed with parameter k as follows: pv (k) = pv − μknv

(k = 0, . . . , K − 1),

(4.29)

where pv is the current position of v, nv is the normal vector of v at pv , μ is the unit distance or resolution, k is the unit count up to K − 1. Let v(kv ) denote the vertex v moved to the digitized position pv (kv ). Using this representation, we have Ep v(kv ) + Es v(kv ) + Ec vi (kvi ), vj (kvj ) (4.30) E= v∈V

v∈Vcg

vi ,vj ∈E

as the objective function of a multi-labeling problem. Since this function has only unary and binary terms, a regular loopy belief propagation can find an optimal solution efficiently. As discussed in Sect. 4.3.3.4, the initial result of optimizing Eq. (4.30) depends on the visibility defined by the visual hull. Therefore we conduct an iterative optimization of Eq. (4.30) while updating the state-based visibility based on the deformation result at the previous iteration stage until it converges.

4.4.1.2 Volumetric 3D Shape Reconstruction Algorithm As a practical volume-based 3D shape reconstruction algorithm, we here present a method based on Starck et al. [48], which employs a silhouette constraint defined in the intra-frame mesh-deformation algorithm as well as the following computational methods: • • • • •

voxel-based representation, the visual hull as the initial guess, state-based visibility, pair-wise ZNCC photo-consistency, and graph-cuts optimization.

Suppose we digitize the target space into voxels that are classified into three groups: voxels definitely outside the object surface and denoted by Vout , voxels definitely inside the object surface and denoted by Vin , and other voxels that may be carved out and denoted by Vmain . Vout is given by the voxels located outside the visual hull. Vmain is given by the voxels that are located inside the visual hull and such that their distances to the visual surface is less than a certain threshold. Vin is the ones in between Vout and Vmain (Fig. 4.22). The goal is to determine whether each voxel of Vmain is a part of the object or not. These per-voxel binary decisions can be rendered as a min-cut/max-flow problem of a graph as illustrated in Fig. 4.23. Voxels correspond to graph nodes, and are connected to neighbors (here we employed a 6-neighborhood). These inter-voxel

124


Fig. 4.22 Voxel classification (see text)

Fig. 4.23 Graph structure for Eq. (4.31). The blue and red circles are Vin and Vout nodes, respectively. Blue and red lines denote edges with infinite weight, i.e. edges cannot be cut

connections are called “n-links”. Each voxel has two additional links, one to the source and the other to the sink (called “s-link” and “t-link”). The s-link and t-link weights for nodes in Vin and Vout are set to infinity, respectively. In addition, an edge connecting a pair of Vin nodes or a pair of Vout nodes is given infinite weight as well. This configuration makes a cut of this graph to divide the Vmain nodes into either of the source side or the sink side while keeping Vin and Vout voxels in the source and the sink side, respectively. Here the question is how to assign the other edge weights so that the source side of the min-cut represents the photo hull. In this algorithm, the photo hull of the surface is given by finding a surface which minimizes the following objective function [47]: vi + vj , (4.31) λ+ ρ E =− 2 v∈V

vi ,vj ∈E

where ρ() is the ZNCC photo-consistency computed by the state-based visibility function with the visual hull. The first term λ (a positive weight as described in Sect. 4.3.3.1) is a per-voxel ballooning term and a shape prior. The second term


125

is a data term representing the photo-consistency measure at the point between vi and vj . Minimization of this objective function can be done by finding a min-cut of the graph with mapping the first term to the s-link weights and the second term to the n-link weights. Notice that smaller photo-consistency ρ should correspond to better score in this formalization. As well ρ ≥ 0 should hold because negative n-link weights make the min-cut very hard to find. Since ZNCC takes values within [−1, 1] and larger is better, we need to convert it to satisfy these requirements on ρ. As such a mapping function, Vogiatzis et al. [58] proposed the following function. This converts ZNCC value to [0, 1], where smaller is better, by π ρ = 1 − exp − tan2 (ZNCC − 1) σ2 , (4.32) 4 where σ is a parameter to control the hot range of the original ZNCC values. For example σ = 0.5 maps all ZNCC values within [−1, 0] to 1, and σ = 2 maps ZNCCs within [−1, −0.75] to 1. This means that a smaller σ discards a wider range of lower ZNCC values, as ρ = 1 (the worst photo-consistency), regardless of their original values. In addition, we embed silhouette constraints into the graph-cuts optimization process. Suppose the contour generators are given as a set of points Vcg as in Sect. 4.4.1.1.3. These points can serve as additional “definitely object-side points” as proposed by Tran and Davis [53]. This is simply done by assigning an infinite s-link weight for nodes close to a point in Vcg . Just as the voxel nodes in Vin , the nodes having infinite s-link weight will be included in the source side, i.e. the object side, since the min-cut cannot cut such links.

4.4.1.3 Performance Evaluations 4.4.1.3.1 Quantitative Performance Evaluations with Synthesized Data In this evaluation, we conducted the following experiments: Studio: Fig. 4.24 shows the studio with 15 XGA cameras, whose specifications are taken from the calibration data of Studio B in Table 2.3 and Fig. 2.4. 3D object: Generate a super-quadric [2] object of 1.2 [m] diameter by the following equations and place it at the center of the capture space in the studio: x(u, v) = ax cosn u cose v + bx , y(u, v) = ay cosn u sine v + by , z(u, v) = az sinn u + bz , −π/2 ≤ u ≤ π/2,

(4.33)

π ≤ v ≤ π,

where u and v are spherical coordinates, n and e denote parameters controlling roundness/squareness. ax , ay , az and bx , by , bz denote scale and translation

126


Fig. 4.24 Multi-view video capture studio with 15 synchronized XGA cameras @ 25fps in the same settings as Studio B in Table 2.3 and Fig. 2.4

factors for x, y, z, respectively. The object is generated with n = 1.0, e = 3.0, ax = 60.0, ay = 60.0, az = 50.0, bx = 0.0, by = 0.0, bz = 50.0 as shown in Fig. 4.25(a). It was designed to have heavy concavities and wide thin protrusions, which models complex 3D shapes of a dancing MAIKO. And moreover, their accurate 3D reconstruction is not easy in general. Multi-view images: The object surface is covered with random texture patterns, and a set of 15 multi-view images taken by the XGA cameras were generated as input data for the 3D shape reconstruction by the methods presented in Sects. 4.4.1.1 and 4.4.1.2. Figure 4.25 shows the results of the 3D shape reconstruction, where colors denote surface distance values between the original synthesized object and the reconstructed object surfaces: red regions indicate well reconstructed surface areas (areas close to the ground truth), whereas green indicate poor reconstructions. Figure 4.25(b) shows the visual hull indicating that the synthesized object has large concavities on its sides and that such concavities are not well reconstructed by shape from silhouette alone. Figure 4.25(c) shows the initial result of the mesh deformation using the visual hull as the initial shape. Figure 4.25(d) illustrates the result of the second iteration deformed from Fig. 4.25(c). Figure 4.25(e) shows the result by the volumetric reconstruction, while Fig. 4.25(f) illustrates another result of the volumetric reconstruction with a smaller ballooning term. Table 4.1 compares reconstruction accuracy and completeness quantitatively. Accuracy denotes the distance d (in cm) such that 90 % of the reconstructed surface is within d cm of the ground truth, and completeness measures the percentage of the reconstructed surface that are within 1 cm of the ground truth [45] (intuitively speaking, the percentage of “red” areas of Fig. 4.25). Max distance is the maximum distance from the reconstructed surface to the ground truth. From these results we can observe that: • The mesh-deformation algorithm performs better than the volumetric approach. • The iterative mesh optimization can contribute to refine the 3D surface geometry. • The volumetric reconstruction is sensitive to the ballooning factor. Reconstructions with smaller values tend to carve the volume too much in particular in regions where the shape is thin as shown in Fig. 4.25(f).


127

Fig. 4.25 Reconstructions of a synthesized object. (a) Ground truth, (b) visual hull, (c) mesh deformation (first iteration), (d) mesh deformation (second iteration), (e) volumetric reconstruction, (f) reconstruction failure by the volumetric reconstruction due to an inappropriate ballooning term. that the colors in (b) to (e) indicate surface distance values to the ground truth surface (red is closer). Note that the synthesized object (a) originally had a random texture over the surface, which was replaced by Gouraud shading for printing

Table 4.1 Reconstruction accuracy and completeness Visual hull Fig. 4.25(b)

Intra (first) Fig. 4.25(c)

Intra (second) Fig. 4.25(d)

Volumetric Fig. 4.25(e)

90 % accuracy

5.36 cm

1.27 cm

1.19 cm

3.44 cm

1 cm completeness

35.77 %

95.22 %

98.54 %

37.35 %

Max distance

14.31 cm

4.31 cm

2.37 cm

7.83 cm

The last point is due to the fact that the graph-cuts formalization (Eq. (4.31)) accumulates its cost over the entire object surface and volume. To avoid this problem, the following strategies can be introduced into the cost minimization process: • Find well photo-consistent partial surfaces, for which let the objective function accumulates smaller (ideally 0) photo-consistency costs over the surfaces as well as ballooning costs over the volumes enclosed by the surfaces.

128


Fig. 4.26 Cost aggregation around a “bottleneck” shape

• Find a “bottleneck” where the object (source-side nodes) can be cut into two groups by cutting a small area, that is, by cutting a small number of n-links (Fig. 4.26). In this case the objective function can discard the ballooning cost over the volume (the red area of Fig. 4.26) and the photo-consistency cost over the surface (the green line). Instead, it should accept the photo-consistency costs over the cutting area (the bold black line). Since it does not correspond to the real geometry, per-node photo-consistency costs can be large there. However, if the area size is small enough, then the sum of such photo-consistency costs can be smaller than the accumulated ballooning costs and photo-consistency costs over the volume and surface. Figure 4.25(f) shows an example of such “bottleneck” case. The silhouette constraint tries to preserve thin areas. However, as a result of balancing between the ballooning cost and object thickness, only the silhouette constraint points remain as a part of the object, and the generated shape has many holes around it. The reason is twofold: • The silhouette constraint defined in Sect. 4.4.1.1.3 is nothing but the silhouette “boundary” constraint. It helps to preserve the boundary, not the silhouette area inside it. • The volumetric formalization does not have an explicit local surface connectivity constraint. In the case of the mesh deformation, on the other hand, the mesh edges explicitly define a strict connectivity between vertices. This point allows the silhouette “boundary” constraint to serve as a silhouette “area” constraint even if its definition itself does not guaranty exact area constraint strictly.

4.4.1.3.2 Qualitative Performance Evaluations with Real Data To evaluate the performance of the mesh-deformation and the volumetric reconstruction methods for real world objects, we captured multi-view video data of a dancing MAIKO with FURISODE in Studio B, whose specifications and calibration accuracy are shown in Tables 2.3 and 2.5, and Fig. 2.4. Figure 4.27 illustrates a test set of captured multi-view image frames. It should be noted that her long thin sleeves and sash move widely in a complex manner as the MAIKO plays. Figure 4.28 shows (a) the visual hull and reconstructed 3D shapes by (b) the mesh-deformation and (c) the volume-based methods. In this evaluation, we used


129

Fig. 4.27 Multi-view image frames of a dancing MAIKO captured by the Studio B system

Fig. 4.28 (a) Visual hull and reconstructed 3D shapes by (b) the mesh-deformation and (c) the volume-based methods

5 mm mesh resolution for the mesh deformation, while the voxel resolution for the volumetric reconstruction was 10 mm. Both require approximately 1 GB memory of space in total. The experimental results demonstrate that the mesh-deformation algorithm provides better reconstruction qualitatively. The upper and lower rows of Fig. 4.29 show close-up images of the same 3D shapes from two different directions A and B as illustrated in Fig. 4.28, from which we can observe 1. The visual hull has some phantom protrusions and holes due to the complicated self-occlusions. 2. The mesh-deformation result clearly inherits the complicated mesh structure from the visual hull.

130


Fig. 4.29 Close-ups of the 3D shapes of Fig. 4.28. The directions for the upper and lower close-ups are shown in A and B in Fig. 4.28(b), respectively

3. The volumetric reconstruction successfully removed phantom volumes regardless of the complicated visual hull structure illustrated with the red circles in Fig. 4.29(c). 4.4.1.4 Discussions 4.4.1.4.1 Silhouette Constraint The 3D shape reconstruction methods presented in this section employ the silhouette boundary constraint. The silhouette areas inside the boundaries are used only to compute the visual hull. This means that the mesh is deformed to keep the silhouette boundaries, and the volumetric method may introduce holes even in areas covered by silhouette interiors as shown in Fig. 4.25(f). For a better use of the silhouette constraint, the following methods were developed. Furukawa and Ponce [11] proposed a patch-based algorithm with a silhouette “area” constraint. It explicitly assigns at least a single patch to each silhouette pixels to ensure that the projection of the estimated 3D shape exactly matches with original multi-view silhouettes. Sinha et al. [46] proposed an iterative shape reconstruction algorithm which estimates a 3D shape first, and then modifies it so that its 2D projections exactly match with original multi-view silhouettes. This algorithm iteratively utilizes shape from stereo and shape from silhouette to seek a 3D shape which satisfies both constraints. Cremers and Kolev [7] proposed a reconstruction framework based on convex optimization. The shape is represented by a set of voxels, and the silhouette area constraint is defined as a per-pixel inequality constraint which aggregates binary voxel occupancies along the visual ray passing through the pixel. This approach integrates both shape from stereo and silhouette constraints in a single optimization framework.


131

4.4.1.4.2 Topology In surface-based algorithms, simple-mesh deformations based on vertex translations do not change the global surface topology. This implies that the visual hull, or the initial shape for the deformation, must have the correct object topology. For example, if the object has a genus-1 shape like a torus, then the visual hull must have the same genus-1 topology. In contrast to this limitation, if we can assume that the visual hull and the real object have a same topology, then this simple deformation helps preserving the topology even for thin object areas. The above-mentioned limitation can be relaxed by implementing complex deformation operations [61] or changing the shape representation into volumetric or patch-based ones. One straightforward implementation is to employ a two-step approach. The first step utilizes the volume-based reconstruction with a relatively large (safer) volumetric cost λ, and then uses the resultant 3D shape as the input of the second step which utilizes the mesh-based reconstruction. However, designing the first step to cull phantom (or safe) volumes is not a simple task.

4.4.1.4.3 Ballooning To tackle the problem of uniform volumetric ballooning cost (Fig. 4.25(f)), Hernandez et al. proposed an “intelligent ballooning” scheme, which introduces nonuniform volumetric cost [14]. They defined the value of λ based on the result of multi-view 2.5D depth reconstruction. They first reconstruct multi-view 2.5D depth maps, and set a smaller λ if nodes are placed in front of the depth maps. This approach can be seen as an extension of traditional “depth fusion” techniques used to reconstruct a single 3D structure from multiple 2.5D depth maps obtained by a laser range finder [19].

4.4.2 Simultaneous 3D Shape and Motion Estimation from Multi-view Video Data by a Heterogeneous Inter-frame Mesh Deformation In this section we present a simultaneous 3D shape and motion estimation algorithm based on a heterogeneous inter-frame mesh deformation [39, 40]. Suppose we have reconstructed the 3D shape of frame t as M t by the intra-frame mesh deformation presented in Sect. 4.4.1.1. The goal of the algorithm is to find an inter-frame deformation which deforms M t to M t+1 based on the multi-view image frames at t + 1. The deformation should map each vertex of M t to a corresponding position in M t+1 while preserving the mesh structure: the number of vertices and their mutual connectivities. The key idea is to employ a heterogeneous deformation strategy where vertices with prominent textures lead the deformation of neighbors while poorly textured

132


vertices follow such leading neighbors. This is based on the following observations. If all vertices of M t had rich textures and could be uniquely localized/identified on the surface, we would be able to establish exact vertex correspondences over frames. For real world objects, however, such prominent textures are limited on some parts of their surfaces. In addition, for complex human actions like MAIKO dances, surface motion patterns vary a lot depending on their locations: some parts like a head follow rigid motions, while others like loose and soft sleeves and sashes show very complicated motion patterns. Since the motion estimation for the former can be well modeled, it may lead the estimation of the latter. Thus the heterogeneity employed in the presented algorithm is twofold: the mesh deformation is controlled depending on both texture prominence and motion rigidity. To implement the above-mentioned strategy, we employ the following constraints to deform M t to M t+1 . Photometric constraint: A patch in M t+1 should be placed so that its texture, which is computed by projecting the patch onto a captured image, should be consistent irrespectively of the multi-view images at both t and t + 1 it is projected on. This is an extension of the photo-consistency constraint used in the intra-frame deformation in Sect. 4.4.1.1, and we call this a spatio-temporal photo-consistency constraint. Silhouette constraint: When M t+1 is projected onto an image plane, its 2D silhouette should coincide with the observed object silhouette at frame t + 1 on that image plane. Smoothness constraint: M t+1 should be locally smooth and should not intersect with itself. Motion constraint: Vertices of M t should drift in the direction of the motion flow of its vicinity to reach those of M t+1 . The drifting is controlled by the rigidity of each vertex (see below). Collision constraint: While in motion, some parts of M t may collide with others and later get apart. To preserve the mesh structure even when such surface collisions occur, we introduce a new collision constraint, which prevents M t+1 from intruding inside the surface. (The definition will be given in detail later.) We implement these constraints as forces working on each vertex of M t , which deform M t to M t+1 so that M t+1 is consistent with the observed multi-view video frames at t + 1. In what follows, we present a heterogeneous mesh-deformation algorithm as the computational scheme to realize such a deformation.

4.4.2.1 Heterogeneous Mesh Deformation Algorithm As discussed above, two types of heterogeneity are employed in the algorithm: texture prominence and motion rigidity. Here we address their implementation methods.


133

Fig. 4.30 Roughly estimated motion flow patterns. (a) Two 3D surfaces of consecutive frames. (b) Estimated motion flow pattern. ©2008 IEICE [41]

4.4.2.1.1 Heterogeneous Motion Model As discussed above, complicated object actions like MAIKO dances cannot be modeled by rigid motions or their combinations. To represent such object actions, we introduce a heterogeneous motion model consisting of a mixture of surface warping and rigid motions. First we classify vertices of M t into warping or rigid parts by computing and clustering motion flow patterns from t to t + 1 frames (cf. Fig. 4.30, and the practical algorithm described later): Rigid part (Ca-1) A vertex representing an element of a rigid part of the object. It should move along with the others in the same part keeping their shape. Warping part (Ca-2) A vertex corresponding to a part of the object surface under free deformation. It should move smoothly along with its neighbors.

4.4.2.1.2 Vertex Identifiability In general, we cannot expect that all the points on the object surface have prominent texture and their 3D positions can be estimated by stereo matching. Hence not all the vertices of the mesh model are identifiable, and hence the photo-consistency constraint, which is used to estimate the vertex position on the 3D object surface, will not work well for such vertices. Thus the vertices should be classified into the following two types: Cb-1 A vertex with prominent texture. It can be stably identified over frames to obtain reliable local deformation, which its neighbors should follow. Cb-2 A vertex with less prominent or no texture. Its deformation should be led by its neighbors. We regard a vertex as identifiable if it has prominent texture and, as well, is photoconsistent. It is labeled as Cb-1 (identifiable), or otherwise as Cb-2.

134


4.4.2.1.3 Algorithm With the above-mentioned definitions of the two types of heterogeneity, the heterogeneous mesh-deformation algorithm is designed as follows: Step 1. Set the mesh M t at frame t as the initial estimation of M t+1 to be reconstructed. Note that M 0 is computed by the intra-frame mesh-deformation method described in Sect. 4.4.1.1. Step 2. Compute roughly estimated motion flow patterns between frames t and t + 1. Step 3. Categorize the vertices based on the motion flow patterns: Step 3.1. By clustering the estimated motion flow vectors, label the vertex with Ca-1 if it is an element of a rigid part, or Ca-2 otherwise. Step 3.2. Make the springs attached to vertices with label Ca-1 stiff. As will be shown in Fig. 4.32, the spring model is used to implement the smoothness constraint. Step 4. Deform the mesh iteratively: Step 4.1. Compute forces working at each vertex, respectively. Step 4.2. For a vertex having identifiability I (v) exceeding a certain threshold, that is, for a vertex with label Cb-1, let the force computed for it diffuse to its neighbors. Step 4.3. Move each vertex according to the force. Step 4.4. Terminate if all vertex motions are small enough. Otherwise go back to Step 4.1. Step 5. Take the final shape of the mesh model as the object shape at frame t + 1, M t+1 . Note that M t and M t+1 share the same mesh structure and the detailed motion vectors for all vertices, i.e. the vertex correspondences between M t and M t+1 , are also obtained. Note that for a vertex of type Ca-2 ∧ Cb-2, a vertex without prominent texture nor belonging to a part of a rigid part, the position at t + 1 is interpolated by the smoothness constraint. On the other hand, a vertex of type Ca-1 ∧ Cb-1, a vertex with prominent texture and belonging to a part of a rigid part, moves so as to lead the rigid part motion. A vertex of type Ca-2 ∧ Cb-1 moves freely to satisfy the photoconsistency, while a vertex of type Ca-1 ∧ Cb-2 moves under combined influences from the neighbors. In what follows, the practical implementation methods to compute the forces are given supposing M t = M t+1 = {V , E} where V and E denote the set of vertices and edges in M t as well as in M t+1 .

4.4.2.2 Rigid Part Estimation In Step 2 and Step 3.1 of the algorithm, we need to estimate a rough motion flow and then find rigid parts from it.


135

Fig. 4.31 Motion flow clustering. (a) Roughly estimated motion flow patterns based on inertia and nearest neighbor search between the current shape at t and the visual hull at t + 1. (b) Short flow removal. (c) Clustering by position. (d) Sub-clustering by motion direction. ©2008 IEICE [41]

The motion flow pattern is computed by a nearest point search from M t to the visual hull at t + 1, VHt+1 . Suppose each vertex is associated with a motion vector between M t−1 to M t to predict a motion vector from M t to M t+1 . Then, we estimate the vertex position at t + 1 by integrating both the surrounding motion flow pattern and motion inertia. Let VHt+1 denote the visual hull of frame t + 1 and pv the position of vertex v on M t . Then the motion flow of v is defined as follows. mv = pVH,v − pv ,

(4.34)

d pv where pVH,v denotes a vertex position on VHt+1 such that it is closest to pv + dt d and satisfies npv · npVH,v > 0. Here pv + dt pv denotes the predicted position of v at t + 1 based on its motion history and npVH,v the normal direction of VHt+1 at pVH,v . Then we estimate rigid parts by applying a clustering to mv . This is done by (1) removing short flows (Fig. 4.31(a) and (b)), (2) clustering mv based on the origin positions (Fig. 4.31(c)), and (3) sub-clustering based on the directions (Fig. 4.31(d)).

4.4.2.3 Constraints as Vertex Forces 4.4.2.3.1 Photometric Constraint Following the photometric constraint implementation of the intra-frame deformation in Sects. 4.4.1.1.2 and 4.4.1.1.5, the photo-consistent position of each vertex v ∈ V is searched along its normal direction. Then, the possible vertex position can be defined as follows: pv (k) = pv − μknv

(k = −K, . . . , K),

(4.35)

136


where pv denotes the current position of v. nv denotes the normal vector of v at pv , μ the unit distance or resolution, and k the unit count. Let v(kv ) denote the vertex v transformed to the digitized position pv (kv ). With this representation, the most photo-consistent position for v is defined by kp = arg max ρ v(k) , (4.36) k

where ρ(v(k)) evaluates the photo-consistency using state-based visibility with M t . Notice that the photo-consistency evaluation function ρ() evaluates multi-view textures of frame t captured at the position of pv , and frame t + 1 captured at pv (k) in order to measure the spatio-temporal consistency. Then, the texture prominence is defined by ρ(v(kp ))/ρ(v(kp )), where kp denotes the second best photo-consistent position. The force at v derived by the photometric constraint is controlled by this texture prominence measure as follows: pv (kp ) − pv (0) if ρ(v(kp ))/ρ(v(kp )) > threshold, (4.37) Fp (v) = 0 otherwise.

4.4.2.3.2 Silhouette Constraint First, the contour generator points Vcg on the visual hull at frame t + 1 is computed by the method described in Sect. 4.4.1.1.3. Then for each vertex v in M t , the existence of a point in Vcg is searched along the normal direction of v just in the same way as above. Let pv (k) (k = −K, . . . , K) denote the possible positions of v defined by Eq. (4.35) and pv (ks ) one of pv (k) (k = −K, . . . , K) closest to the contour generator points. Then, the force at v derived from the silhouette constraint is defined as follows: pv (ks ) − pv (0) if pv (ks ) − pcg,v 2 < threshold and npv · npcg,v > 0, Fs (v) = 0 otherwise, (4.38) where pcg,v denotes the point in Vcg closest to pv (ks ). npv and npcg,v denote the normal vector on M t at the original position of v and that on the visual hull of t + 1 at pcg,v , respectively. The force Fs (v) navigates the vertex v toward the contour generator if there exists pv (ks ) such that one of Vcg is close enough to pv (ks ) and has the similar normal direction to that of v.

4.4.2.3.3 Motion Constraint As we discussed earlier, not all the vertices have prominent textures. In the intraframe mesh deformation, we can simply assume that all vertices move toward the inside since the deformation starts from the visual hull. The inter-frame mesh deformation, on the other hand, cannot have such default motion.


137

Fig. 4.32 Extended spring model for the smoothness constraint. ©2008 IEICE [41]

The motion constraint in the inter-frame deformation is designed to provide a “default” motion direction for a vertex to deform. Similarly to the rigid part estimation, the force derived from the motion constraint for a vertex can be defined as follows: Fm (v) = pVH,v − pv ,

(4.39)

d pv where pVH,v denotes a vertex position on VHt+1 such that it is closest to pv + dt d and satisfies npv · npVH,v > 0. Here pv + dt pv denotes the predicted position of v at t + 1 based on its motion history and npVH,v the normal direction of VHt+1 at pVH,v . Notice that pVH,v is fixed throughout the iteration, but pv changes its position during the deformation.

4.4.2.3.4 Smoothness Constraint We model the smoothness constraint by using an extended spring model (Fig. 4.32). In this model, vertex v has two spring groups: • Structural springs connecting to the neighboring vertices of v, vj . • Flex springs connecting v to the vertices vˇj such that a line between v and vˇ j defines a diagonal of a quadrilateral obtained by merging a pair of neighboring triangle patches which share a pair of neighboring vertices of v. Structural springs control distances between neighboring vertices, while flex springs control mesh folding. With spring constants ks (v, vj ) and kf (v, vˇj ) for the structural and flex springs, the force at v derived from the smoothness constraint is defined as follows: F i (v) =

N j =1

N f i v, vj , ks (v, vj ) + f i v, vˇj , kf (v, vˇj ) − q˙ v ,

(4.40)

j =1

where vj denotes the j th neighboring vertex of v, vˇj the j th diagonally facing vertex of v, N the number of neighboring vertices, and q˙ v the damping force of spring which changes proportionally to the velocity of v. f i (·) is the Hooke spring

138


force given by f i (va , vb , k) = k

q va − q vb − l(va , vb ) (q va − q vb ), q va − q vb

(4.41)

where l(va , vb ) denotes the nominal length of the spring between va and vb . Note that the number of diagonally facing vertices is equal to the number of neighboring vertices N .

4.4.2.3.5 Collision Constraint The repulsive force Fr (v) works for vertices where surface collisions occur. It prevents collided surfaces from intruding into the surface interior. To implement this force, the parts of the surface that may collide with each other need to be identified first. While surface collisions could be computed with accurate shape and motion data, it would not be a good idea in our case since the accuracy of shape and motion is limited. We employ instead the visibility of each vertex for the collision detection. That is, for surface parts having faces close to others, no camera can observe them: the set of visible cameras Cv of v becomes empty. Note that the deformation of such invisible vertices as well as colliding ones should follow neighbors while preventing surface intersections. Suppose Fr (v) is initialized to 0 for all vertices and V∅ denotes a set of invisible vertices such that Cv = ∅. Then for each v ∈ V∅ , 1. Select the set of vertices v ∈ Vd (v) from V∅ \v such that the distance to v is closer than the shortest distance to the neighboring vertices of v. 2. Compute Fr (v) by p v − pv . (4.42) Fr (v) = − pv − pv 3 v ∈Vd (v)

While the general collision detection is known to be a time-consuming process, this algorithm can drastically reduce the number of vertices to be processed by reusing the visibility computation used for the photo-consistency evaluation.

4.4.2.4 Overall Vertex Force The overall vertex force is given by F (v) = ωp Fp (v) + ωs Fs (v) + ωi Fi (v) + ωm Fm (v) + ωr Fr (v),

(4.43)

where the ωs are weighting coefficients. Starting with M t , the 3D shape of frame t, we can compute this per-vertex force for the heterogeneous mesh deformation. Each of the constituent forces navigates a vertex of M t so that its corresponding constraint is satisfied. The heterogeneous deformation finds a vertex position where the forces balance.


139

Fig. 4.33 Camera setup for the synthesized data

4.4.2.5 Performance Evaluations 4.4.2.5.1 Synthesized Data Figure 4.34 shows the result of the inter-frame deformation for a synthesized object captured by the camera setup illustrated in Fig. 4.33. The left column shows a temporal sequence of the synthesized object, the second left their visual hulls, the second right the results of the intra-frame deformation, and the right a temporal sequence of the 3D shape computed by the inter-frame deformation. Figure 4.35 shows the quantitative performance evaluation results. Accuracy evaluates the distance d (in cm) such that 90 % of the reconstructed surface areas are within d cm of the ground truth, and completeness measures the percentage of the reconstructed surface areas that are within 1.0 cm of the ground truth [45]. From these results, we can observe that the inter-frame mesh-deformation algorithm can (1) improve the shape compared with the visual hull, and (2) preserve the

Fig. 4.34 Results of inter-frame deformation for the synthesized data

140


Fig. 4.35 Reconstruction accuracy and completeness

mesh structure even if some surface parts collide with others, but (3) shows lower quality measures compared with the intra-frame deformation as the time proceeds even though the former employs richer reconstruction cues, i.e., the temporal information. The reason of the limited performance of the inter-frame deformation is threefold. 1. The mesh structure (vertex connectivity) inherited from the initial frame mesh is not guaranteed to be optimal to describe the 3D shapes for the other frames. Keeping the initial mesh structure over time can work as an excessive constraint for the shape reconstruction. On the other hand, the shape optimization in the intra-frame deformation is free from such an additional constraint. 2. The intra-frame deformation can exploit a strong reconstruction cue given by the visual hull. That is, as long as the deformation starts from the visual hull, the real object surface is guaranteed to be found inside the visual hull. This allows the shape optimization process to seek a solution surface in a very limited direction. On the other hand, the inter-frame deformation does not have such a solid direction in its optimization. 3. The sequential deformation process of the inter-frame mesh deformation accumulates errors over time, which degrades the accuracy of the shape reconstruction at later frames. While we may be able to improve the performance of the inter-frame mesh deformation taking the above limitations into account, it would be a reasonable choice to employ frame-by-frame reconstruction methods to reconstruct a long sequence of 3D shapes in a good quality even if the mesh structure is not preserved over time. Thus the texture generation and 3D video applications (visualization, editing, kinematic structure analysis, and encoding) presented in the later chapters assume that the 3D shape reconstruction for a 3D video stream is conducted frame by frame. Hence the 3D mesh structure varies over time and no correspondence between a pair of consecutive frame mesh data is established.


141

Fig. 4.36 Frames of MAIKO video captured by one of the multi-view cameras. ©2008 IEICE [41]

Fig. 4.37 Sequence of 3D shapes reconstructed by the inter-frame mesh deformation

4.4.2.5.2 Real Data Figure 4.36 shows a temporal sequence of MAIKO video frames captured by one of the multi-view cameras. Figure 4.37 shows a sequence of 3D shapes reconstructed by the inter-frame mesh deformation and Fig. 4.38(c) the dense and longterm 3D motion flow patterns representing the vertex motion trajectories between Figs. 4.38(a) and (b). These results empirically prove that the inter-frame deformation can process dynamically complex object actions such as MAIKO dances. To evaluate the performance of the inter-frame deformation for complex object motions including surface collisions, multi-view video data of Yoga performance were analyzed. Figure 4.39 shows a temporal sequence of image frames captured by one of the multi-view cameras and Fig. 4.40 the result of the inter-frame de-

142


Fig. 4.38 Estimated long-term motion flow pattern. Each line (from blue to red) denotes a 3D motion flow

formation. In this sequence, the subject crosses her legs, which touch each other completely at frame 13 to 14. Figures 4.41 and 4.42 show the topological structures of the 3D shapes in Fig. 4.40 and the corresponding visual hulls. Here the topological structure of each 3D shape is computed by using Reeb-graph [54] (cf. the detailed computational algorithm in Chap. 8). We can observe that the topological structure is preserved over time with the inter-frame deformation, while those of the visual hulls drastically change due to the surface collisions at the legs. This point can be well observed by close-up figures of frame 14 shown in Fig. 4.43. From these results, we can conclude that the proposed inter-frame deformation can process object actions involving heavy surface collisions.

4.5 Conclusion This chapter addresses the 3D shape reconstruction from multi-view video data. We first reviewed Shape from X methods and concluded that shape from stereo and shape from silhouette are the most practical reconstruction methods among others, and the state-of-the-art technologies integrate them together since they can work complimentarily for accurate and robust reconstruction. As for 3D shape reconstruction for 3D video production, we categorized existing methods into three types: frame-wise 3D shape reconstruction, (2) simultaneous 3D shape + motion estimation, and (3) 3D shape sequence estimation. Then we introduced the three essential design factors for 3D shape reconstruction algorithms: (1) photo-consistency, (2) visibility evaluation, and (3) shape representation and associated computational model for optimization. Based on these design factors, we implemented several practical algorithms for frame-wise 3D shape reconstruction and simultaneous 3D shape and motion reconstruction, and evaluated their performance quantitatively with synthesized data as well as qualitatively with real world multi-view video data of MAIKO dance and Yoga performance. Important conclusions we obtained include: • According to the state-of-the-art technologies, the simultaneous 3D shape and motion reconstruction approach, i.e. the inter-frame mesh-deformation method, does not perform better than the frame-wise reconstruction methods in terms of the reconstructed 3D shape accuracy, while the former can manage complex object actions involving heavy surface collisions. This is because:

4.5 Conclusion

Fig. 4.39 Yoga video frames captured by one of the multi-view cameras. ©2008 IEICE [41]

Fig. 4.40 3D shapes reconstructed by the inter-frame deformation

143

144


Fig. 4.41 Topological structures computed from the 3D shapes in Fig. 4.40

– The mesh structure inherited from the initial frame mesh is not guaranteed to be optimal to represent 3D shapes over frames. Keeping the initial mesh structure over time can even work as an excessive constraint for the shape reconstruction especially when the object shape changes largely over time as MAIKO dances. – The intra-frame deformation can exploit a strong reconstruction cue given by the visual hull, which allows the shape optimization process to seek a solution in a very limited direction, while the inter-frame deformation does not have such a solid direction in its optimization. – The sequential computation process to deform the initial mesh structure frame by frame for a long period is prone to accumulate errors. • Consequently we employ the frame-wise reconstruction strategy in the following chapters. That is, each frame of a 3D video stream has a completely independent mesh structure while the object changes its shape continuously. In addition, this chapter has pointed out several open issues to be studied as future problems.

4.5 Conclusion

145

Fig. 4.42 Topological structures computed from the visual hulls

Fig. 4.43 Shapes and topological structures in a frame where parts of the object surface collide with each other. (a) 3D shape by the inter-frame deformation, (b) visual hull, (c) the topological structure of (a), (d) the topological structure of (b)

View-dependent 3D shape reconstruction: The introduction of a virtual camera/ viewpoint into the computational process of 3D shape reconstruction from multiview images can improve the reconstruction quality. The view-dependent 3D

146


shape reconstruction could weight multi-view images depending on the angles between the viewing directions of the virtual and actual cameras. New cues and sensors: The introduction of other cues and/or devices such as shape from (de)focus/specularities/polarization, active stereo, ToF cameras, etc. (Sect. 4.2.1) will enable us to capture specular and transparent object surfaces as well as improve the 3D shape accuracy. Problem formalization: Recent studies such as [7] showed that even the silhouette constraint can be better implemented with a new problem formalization (Sect. 4.4.1.4.1). Developing such new problem formalizations will improve the accuracy and robustness of dynamic 3D shape reconstruction from multi-view video data. Segmentation: In this book we assume that accurate multi-view object silhouettes are available. While silhouettes are strong reconstruction cues, their accurate extraction in real environments is a long lasting segmentation problem in computer vision. Further studies on simultaneous estimation of the 3D shape and 2D silhouettes (Sect. 4.2.2.2.1), or 3D shape reconstruction from incomplete silhouettes will help making the 3D video technology more practical. Limited visibility: Finally we should note that the limited visibility of the object surface (Sect. 4.3.2) is the essential source of the problem in 3D shape reconstruction from multi-view images. That is, even if we increase the number of cameras, some parts of the object surface cannot be observed due to self-occlusions, especially when an object performs complex actions like Yoga. To cope with such self-occlusions, the inter-frame mesh-deformation strategy can be augmented to estimate properties of occluded surface parts from data obtained when they become visible. Such global dynamic data processing schemes should be investigated.

References 1. Alexa, M., Behr, J., Cohen-Or, D., Fleishman, S., Levin, D., Silva, C.T.: Point set surfaces. In: The Conference on Visualization, pp. 21–28 (2001) 2. Barr, A.H.: Rigid Physically Based Superquadrics, pp. 137–159. Academic Press, San Diego (1992) 3. Baumgart, B.G.: Geometric modeling for computer vision. Technical Report AIM-249, Artificial Intelligence Laboratory, Stanford University (1974) 4. Baumgart, B.G.: A polyhedron representation for computer vision. In: Proceedings of the National Computer Conference and Exposition, AFIPS’75, pp. 589–596 (1975) 5. Campbell, N., Vogiatzis, G., Hernández, C., Cipolla, R.: Automatic 3D object segmentation in multiple views using volumetric graph-cuts. Image Vis. Comput. 28(1), 14–25 (2010) 6. Virtualizing Engine. Private communication with Profs. Takeo Kanade and Yaser Sheikh, Robotics Institute, Carnegie Mellon University, PA (2011) 7. Cremers, D., Kolev, K.: Multiview stereo and silhouette consistency via convex functionals over convex domains. IEEE Trans. Pattern Anal. Mach. Intell., 1161–1174 (2010) 8. Felzenszwalb, P., Huttenlocher, D.: Efficient belief propagation for early vision. Int. J. Comput. Vis. 70, 41–54 (2006)

References

147

9. Franco, J.-S., Boyer, E.: Efficient polyhedral modeling from silhouettes. IEEE Trans. Pattern Anal. Mach. Intell. 31(3), 414–427 (2009) 10. Fua, P., Leclerc, Y.G.: Using 3-dimensional meshes to combine image-based and geometrybased constraints. In: Proc. of European Conference on Computer Vision, pp. 281–291 (1994) 11. Furukawa, Y., Ponce, J.: Accurate, dense, and robust multi-view stereopsis. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007) 12. Goldlüecke, B., Magnor, M.: Space-time isosurface evolution for temporally coherent 3D reconstruction. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 350–355 (2004) 13. Habbecke, M., Kobbelt, L.: A surface-growing approach to multi-view stereo reconstruction. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007) 14. Hernandez, C., Vogiatzis, G., Cipolla, R.: Probabilistic visibility for multi-view stereo. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007) 15. Hernandez Esteban, C., Vogiatzis, G., Cipolla, R.: Multiview photometric stereo. IEEE Trans. Pattern Anal. Mach. Intell. 30, 548–554 (2008) 16. Horn, B.K.P., Brooks, M.J.: Shape from Shading. MIT Press, Cambridge (1989) 17. Hornung, A., Kobbelt, L.: Robust and efficient photo-consistency estimation for volumetric 3d reconstruction. In: Proc. of ECCV, pp. 179–190 (2006) 18. Ikeuchi, K.: Shape from regular patterns. Artif. Intell. 22(1), 49–75 (1984) 19. Ikeuchi, K., Oishi, T., Takamatsu, J., Sagawa, R., Nakazawa, A., Kurazume, R., Nishino, K., Kamakura, M., Okamoto, Y.: The great Buddha project: digitally archiving, restoring, and analyzing cultural heritage objects. Int. J. Comput. Vis. 75, 189–208 (2007) 20. Ishikawa, H.: Higher-order clique reduction in binary graph cut. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2993–3000 (2009) 21. Kanade, T., Rander, P., Narayanan, P.J.: Virtualized reality: constructing virtual worlds from real scenes. In: IEEE Multimedia, pp. 34–47 (1997) 22. Kang, S.B., Webb, J.A., Zitnick, C.L., Kanade, T.: A multibaseline stereo system with active illumination and real-time image acquisition. In: Proc. of International Conference on Computer Vision, pp. 88–93 (1995) 23. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: active contour models. Int. J. Comput. Vis. 1(4), 321–331 (1988) 24. Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Symposium on Geometry Processing, pp. 61–70 (2006) 25. Kolmogorov, V., Zabih, R.: What energy functions can be minimizedvia graph cuts? IEEE Trans. Pattern Anal. Mach. Intell. 26, 147–159 (2004) 26. Kutulakos, K.N., Seitz, S.M.: A theory of shape by space carving. In: Proc. of International Conference on Computer Vision, pp. 307–314 (1999) 27. Laurentini, A.: The visual hull concept for silhouette-based image understanding. IEEE Trans. Pattern Anal. Mach. Intell. 16(2), 150–162 (1994) 28. Lazebnik, S., Furukawa, Y., Ponce, J.: Projective visual hulls. Int. J. Comput. Vis. 74, 137–165 (2007) 29. Lempitsky, V., Boykov, Y., Ivanov, D., Ivanov, D.: Oriented visibility for multiview reconstruction. In: Proc. of European Conference on Computer Vision, pp. 226–238 (2006) 30. Marr, D.: Vision. W. H. Freeman & Co, New York (1982) 31. Martin, W.N., Aggarwal, J.K.: Volumetric description of objects from multiple views. IEEE Trans. Pattern Anal. Mach. Intell. 5(2), 150–158 (1983) 32. Matsuyama, T., Wu, X., Takai, T., Nobuhara, S.: Real-time 3D shape reconstruction, dynamic 3D mesh deformation and high fidelity visualization for 3D video. Comput. Vis. Image Underst. 96, 393–434 (2004) 33. Miller, G., Hilton, A.: Safe hulls. In: Proc. 4th European Conference on Visual Media Production, IET (2007) 34. Moezzi, S., Tai, L.-C., Gerard, P.: Virtual view generation for 3D digital video. In: IEEE Multimedia, pp. 18–26 (1997)

148


35. Nagel, H.H., Enkelmann, W.: An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 8, 565–593 (1986) 36. Nayar, S.K., Nakagawa, Y.: Shape from focus. IEEE Trans. Pattern Anal. Mach. Intell. 16, 824–831 (1994) 37. Nayar, S.K., Watanabe, M., Noguchi, M.: Real-time focus range sensor. IEEE Trans. Pattern Anal. Mach. Intell. 18, 1186–1198 (1996) 38. Nobuhara, S., Matsuyama, T.: Dynamic 3D shape from multi-viewpoint images using deformable mesh models. In: Proceedings of the 3rd International Symposium on Image and Signal Processing and Analysis, pp. 192–197 (2003) 39. Nobuhara, S., Matsuyama, T.: Heterogeneous deformation model for 3D shape and motion recovery from multi-viewpoint images. In: Proc. of International Symposium on 3D Data Processing, Visualization and Transmission, pp. 566–573 (2004) 40. Nobuhara, S., Matsuyama, T.: Deformable mesh model for complex multi-object 3D motion estimation from multi-viewpoint video. In: Proc. of International Symposium on 3D Data Processing, Visualization and Transmission, pp. 264–271 (2006) 41. Nobuhara, S., Matsuyama, T.: A 3D deformation model for complex 3D shape and motion estimation from multi-viewpoint video. IEICE Trans. Inf. Syst. J91-D(6), 1613–1624 (2008) (in Japanese) 42. Nobuhara, S., Tsuda, Y., Ohama, I., Matsuyama, T.: Multi-viewpoint silhouette extraction with 3D context-aware error detection, correction, and shadow suppression. IPSJ Trans. Comput. Vis. Appl. 1, 242–259 (2009) 43. Okutomi, M., Kanade, T.: A multiple-baseline stereo. IEEE Trans. Pattern Anal. Mach. Intell. 15(1), 353–363 (1993) 44. Seitz, S., Dyer, C.: Photorealistic scene reconstruction by voxel coloring. Int. J. Comput. Vis. 25(3), 151–173 (1999) 45. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 519–528 (2006) 46. Sinha, S.N., Mordohai, P., Pollefeys, M.: Multi-view stereo via graph cuts on the dual of an adaptive tetrahedral mesh. In: Proc. of International Conference on Computer Vision, pp. 1–8 (2007) 47. Starck, J., Hilton, A.: Surface capture for performance based animation. IEEE Comput. Graph. Appl. 27(3), 21–31 (2007) 48. Starck, J., Hilton, A., Miller, G.: Volumetric stereo with silhouette and feature constraints. In: Proc. of British Machine Vision Conference, pp. 1189–1198 (2006) 49. Starck, J., Maki, A., Nobuhara, S., Hilton, A., Matsuyama, T.: The multiple-camera 3-d production studio. IEEE Trans. Circuits Syst. Video Technol. 19(6), 856–869 (2009) 50. Subbarao, M., Surya, G.: Depth from defocus: a spatial domain approach. Int. J. Comput. Vis. 13, 271–294 (1994) 51. Szeliski, R.: Rapid octree construction from image sequences. CVGIP, Image Underst. 58(1), 23–32 (1993) 52. Tomasi, C., Kanade, T.: Shape and motion from image streams: a factorization method—full report on the orthographic case. Int. J. Comput. Vis. 9, 137–154 (1992) 53. Tran, S., Davis, L.: 3d surface reconstruction using graph cuts with surface constraints. In: Proc. of European Conference on Computer Vision, vol. 3952, pp. 219–231 (2006) 54. Tung, T., Schmitt, F.: The augmented multiresolution reeb graph approach for content-based retrieval of 3D shapes. Int. J. Shape Model. 11(1), 91–120 (2005) 55. Vedula, S., Baker, S., Seitz, S., Kanade, T.: Shape and motion carving in 6D. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition (2000) 56. Vogiatzis, G., Hernandez, C., Torr, P., Cipolla, R.: Multiview stereo via volumetric graphcuts and occlusion robust photo-consistency. IEEE Trans. Pattern Anal. Mach. Intell. 29(12), 2241–2246 (2007)

References

149

57. Vogiatzis, G., Torr, P., Seitz, S.M., Cipolla, R.: Reconstructing relief surfaces. In: Proc. of British Machine Vision Conference, pp. 117–126 (2004) 58. Vogiatzis, G., Torr, P., Cipolla, R.: Multi-view stereo via volumetric graph-cuts. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 391–398 (2005) 59. Wada, T., Ukida, H., Matsuyama, T.: Shape from shading with interreflections under a proximal light source: distortion-free copying of an unfolded book. Int. J. Comput. Vis. 24(2), 125–135 (1997) 60. Wu, X.: Parallel pipeline volume intersection for real-time 3D shape reconstruction on a PC cluster. PhD thesis, Kyoto University (March 2005) 61. Zaharescu, A., Boyer, E., Horaud, R.: Transformesh: a topology-adaptive mesh-based approach to surface evolution. In: Proc. of Asian Conference on Computer Vision, pp. 166–175 (2007) 62. Zeng, G., Quan, L.: Silhouette extraction from multiple images of an unknown background. In: Proc. of Asian Conference on Computer Vision, pp. 628–633 (2004)

Chapter 5

3D Surface Texture Generation

5.1 Introduction As discussed in the beginning of Part II, the complexity and limited availability of information in the computational model of 3D video production do not allow us to solve the problem of 3D video production from multi-view video data at once. Thus we take the following three-step solution method: 3D shape reconstruction, surface texture generation, and estimation of lighting environments, while introducing assumptions to overcome the complexity and make up for the incompleteness of input data. First, the 3D shape reconstruction methods described in the previous chapter produce a sequence of 3D meshes from multi-view video data assuming uniform directional lighting and Lambertian object surface. In the second step, the surface texture generation methods presented in this chapter compute surface texture patterns for a 3D mesh data sequence based on observed multi-view video data. The following are basic assumptions about the 3D mesh data: • Shape reconstruction and texture generation processes can work without any problem even for scenes with multiple objects, even though mutual occlusions can damage the quality of 3D video. In this chapter, however, we assume one single 3D object in the scene for the sake of simplicity. It should be noticed that even for one single object, the texture generation process should cope with selfocclusions. • We assume in this chapter a sequence of 3D mesh data reconstructed independently one frame to another. That is, the 3D mesh structure, i.e. number of vertices and their connectivity relations, changes over frames and no explicit correspondence is established between a pair of consecutive frame mesh data. With these assumptions, the problem of texture generation is defined as the problem of generating the surface texture of a 3D mesh frame from a set of multi-view video frames. Figure 5.1 illustrates the computational model for texture generation, where the gray and black arrows illustrate “generation” and “estimation” processes, respecT. Matsuyama et al., 3D Video and Its Applications, DOI 10.1007/978-1-4471-4120-4_5, © Springer-Verlag London 2012

151

152

5


Fig. 5.1 Computational model for texture generation. The gray and black arrows illustrate “generation” and “estimation” processes, respectively

tively. First, suppose 3D mesh data of an object at frame t , So , a set of multiview images, Ii (i = 1, 2, . . . , N), and calibration data of a group of cameras, Ci (i = 1, 2, . . . , N) are given. Then, the general computational model of 3D video production illustrated in Fig. II.2 in the beginning of this part is simplified into Fig. 5.1 assuming: Lambertian reflection: If required, the texture generation process can assume an object surface that follows Lambertian reflection, i.e. Ro = Lambertian, and conducts appearance-based matching between multi-view images to estimate appropriate surface texture patterns depending on local object surface position and shape. Thus photo-consistency evaluation methods similar to those employed in 3D shape reconstruction (Sect. 4.3.1) would be employed for texture generation. As described below, however, the texture generation methods presented in this chapter can simulate non-Lambertian surface reflections such as highlights while introducing a virtual camera. Non-attenuation of uniform directed lighting without interreflection: To simplify the complex light field phenomena, the same assumptions about lighting environments as for 3D shape reconstruction are introduced. That is L ≈ nonattenuation of uniform directed lighting. No background scene: Since the 3D shape of an object is segmented out from the background scene, we can eliminate the background scene from the computational model.

5.1 Introduction

153

The computational model for texture generation is best characterized by introducing a virtual camera, Cˆ. With this additional entity, view-dependent texture generation can be realized, which can render object surface textures with nonLambertian reflection properties such as highlights as well as partially specular shading and soft shadows induced by non-uniform lighting environments. These visual effects are crucial in rendering 3D video of MAIKO dances with shiny FURISODEs of silk embroidered with gold thread. The black arrows in Fig. 5.1 illustrate the computational flow of this view-dependent texture generation, which can simulate complex light field phenomena including highlight, shading, and shadow as well as unknown light sources by controlling the integration process of multi-view images based on a virtual camera. In what follows, the computational scheme of texture generation is discussed and compared to similar methods employed in computer graphics and 3D animation.

5.1.1 Texture Painting, Natural-Texture Mapping, and Texture Generation To develop computational methods of texture generation for 3D video production, we first have to clarify the existing differences with similar methods used in 3D CG animation, which we will refer to as texture painting. The left part of Fig. 5.2 illustrates the computational scheme of texture painting for 3D CG animation, while the right part illustrates texture generation for 3D video production. In the former, first, 3D mesh data of a (static) object and surface texture patterns are designed manually by artists. Then, to animate the object a dynamic sequence of 3D mesh data is generated by retargeting motion capture data and/or by manual motion design (Fig. 5.3 left). Since the designed surface texture patterns specify generic surface characteristics such as reflectance and roughness properties, i.e., diffuse, specular, transparency, and so on, 3D visual contents for 3D cinema, TV, and game can be rendered by computing object appearances including shading patterns, highlights, and shadows under specified lighting environments. This is the process of rendering texture patterns in visual CG contents, where all required information for computation is given. That is, all information on objects, light sources, and background scene as well as virtual camera in Fig. II.2 is prepared to generate the light field, which is then projected onto the image plane of the virtual camera. Since there is no computationability problem, performance is evaluated by “naturalness” of rendered contents and computation speed. Note that all 3D mesh data in 3D CG animation usually share the same mesh structure. This implies that once painted, the surface texture patterns are preserved over the sequence. More specifically speaking, the number of mesh faces (we will use “face” instead of “patch” in this chapter), their connectivity relations, and texture patterns are kept constant, while 3D coordinate values of mesh vertices change over time. While sophisticated 3D CG animation methods allow dynamic change of mesh structure and

154

5


Fig. 5.2 Computational schemes of texture painting and texture generation

Fig. 5.3 Data representations in 3D CG animation and 3D video

texture over time, complete information for texture synthesis is prepared a priori as well. To improve naturalness, natural-texture patterns captured by cameras are often mapped on 3D mesh data. This process is referred to as natural-texture mapping (in what follows, we will call it texture mapping for the sake of simplicity) and shares much with texture generation for 3D video production. The major problems

5.1 Introduction

155

Fig. 5.4 Computational scheme of natural-texture mapping

to be solved for texture mapping, which are shared as well by texture generation, are summarized as follows (see also Fig. 5.4). Geometric transformation between 3D mesh and 2D image Simple algebraic transformations cannot be employed to map 2D image data onto given 3D mesh data if the topological structures of 3D mesh and 2D image data are different. For example, consider how to map a bounded 2D planar image onto the unbounded surface of a sphere seamlessly. Since 3D mesh data representing complex human actions have very complex topological structures, which moreover vary over time, sophisticated geometric mapping methods should be devised for the 3D video texture generation process. This problem is known as mesh parameterization and has been actively studied in computer graphics [12, 18, 19]. Section 5.2 presents a fundamental geometric mapping scheme for texture mapping, which will be employed afterwards for texture generation. Estimation of generic surface reflectance properties from observed images The estimation of object surface reflectance properties from observed images has been a major issue in computer vision [5, 11, 14, 16, 17, 22]. The left half of Fig. 5.4 illustrates a computational model of surface reflection and image observation, which implies that the problem of surface reflectance estimation from observed images is ill-posed without appropriate assumptions. That is, since the 3D surface shape is known in the texture generation as well as in the texture mapping, information about lighting should be given or assumed for reflectance property estimation. While it is possible to estimate reflectance properties of a static object with well calibrated controlled lightings [2, 3], such methods can

156

5


only be used for texture mapping because lighting environments in 3D video capture studios cannot be controlled for calibration of an object that moves freely. In addition to these problems shared with texture mapping, the texture generation process has to solve several additional problems which are essential in 3D video production, and will be discussed next.

5.1.2 Problems in Texture Generation First, let us recall the assumptions introduced in the beginning of this chapter: Lambertian reflectance, non-attenuation of uniform directional lighting without interreflection, segmented 3D object, frame-by-frame 3D mesh reconstruction, and introduction of a virtual camera (Fig. 5.1). Under these assumptions, the texture generation methods presented in this chapter compute the object appearance viewed from the virtual camera using reconstructed 3D mesh and captured multi-view video frame data (Fig. 5.2 right). In this sense, we can call them appearance-based texture generation. As an approach toward generic-property-based texture generation, first a method of estimating dynamically changing lighting environments with a reference object having known 3D shape and reflectance properties will be presented in Chap. 6. Then, a method of estimating reflectance properties of object surface and rendering of visual contents under user-specified lighting environments will be presented in Sect. 6.7. Compared to generic-property-based texture generation, appearance-based texture generation is easier to realize. Its computation process has to solve the following technical issues for rendering high fidelity visual contents, which as well best characterize the differences from texture painting and texture mapping. Multiple observations of mesh face For each frame of video data, a face of 3D mesh is usually observed in multiple images, from which we have to generate a texture for that face. How to integrate such multiple observations without degrading texture quality is a problem. Errors in reconstructed 3D mesh data Since calibration and 3D shape reconstruction processes cannot be perfect, reconstructed 3D mesh data are corrupted with errors and noise. Moreover, as the accuracy of geometric mapping between 3D mesh and observed images is limited, observed images cannot be consistently aligned on 3D mesh. How to resolve such unknown misalignments is a crucial problem for rendering high fidelity visual contents. Time varying 3D mesh structures Although the inter-frame mesh deformation method presented in Sect. 4.4.2 produces a sequence of 3D mesh data preserving the mesh structure over time, many 3D shape reconstruction methods process frames independently one-by-one and generate temporally inconsistent sequence of 3D mesh data (see Fig. 5.3 right). Moreover, as the shape and calibration errors described above vary over time, it does not allow us to use motion information in texture generation. Consequently, the texture generation process should be designed to work independently frame-by-frame.

5.1 Introduction

157

Table 5.1 Texture generation for 3D video Virtual-view dependence

Surface property

Processing unit

Algorithm

Independent

Generic

face

Generic texture generation (Chap. 6)

Independent

Appearance

face

View-independent texture generation (Sect. 5.3)

Dependent

Appearance

vertex

View-dependent vertex-based texture generation (Sect. 5.4)

Dependent

Appearance

face

Harmonized texture generation (Sect. 5.5)

Interactive viewing-direction control The assumptions of uniform directional lighting and Lambertian object surface enable us to consider shading, highlight, and shadow as surface painted markers just in the same way as in 3D shape reconstruction. In texture generation, however, since viewing directions of 3D video are interactively specified and changed by a user, user-specified information, i.e. the properties of a virtual camera, can be employed to render realistic appearances of shiny object surfaces; highlights and shiny surface appearance changes according to viewing directions. That is, the texture generation process should compute the object appearance according to position and viewing direction of the virtual camera: fidelity factors of observed multi-view images should be evaluated based on user-specified viewing direction. With this viewdependent texture generation, object surface textures with non-Lambertian reflection properties such as highlights, as well as partially specular shading and soft shadows induced by non-uniform lighting environments, can be rendered even without knowing generic surface reflectance properties or accurate lighting environments. From a computational speed viewpoint, such texture generation process should compute object appearance from user-specified viewpoint in real time because the viewpoint is not known a priori and is changed dynamically. The remainder of this chapter is structured as follows (see Table 5.1). First, Sect. 5.2 introduces a basic concept of texture mapping focusing on data structures for representing geometric mapping relations between complex 3D mesh and 2D image data. Then Sect. 5.3 presents a naive view-independent texture generation method mainly to demonstrate how much the rendering of visual contents is degraded due to the problems described above. The effectiveness of using information about the virtual camera in texture generation is demonstrated with an improved view-dependent texture generation method described in Sect. 5.4. Finally, Sect. 5.5, which is the main part of this chapter, presents a sophisticated texture generation method named Harmonized texture generation developed to solve the problems described above. Experimental results have proved its performance in generating high fidelity visual contents comparable to originally observed video data. Section 5.6 concludes the chapter and discusses future problems.

158

5


Fig. 5.5 3D mesh and transformed 2D mesh data

5.2 Geometric Transformation Between a 3D Mesh and a 2D Image To map a texture image onto a 3D mesh, we need to know their geometric correspondence. In other words, we need to know how a 3D mesh surface is transformed onto a 2D image plane. The simplest method consists of decomposing a 3D mesh into discrete triangles and placing them uniformly on a 2D image plane. Figure 5.5(a) shows a 3D mesh of MAIKO observed from a certain viewpoint, and Fig. 5.5(b) a uniform decomposition of a 2D rectangular image plane: the correspondences between the 3D mesh triangles and 2D triangles can be established by assigning unique IDs for corresponding 3D and 2D triangles and defining a geometric transformation between a pair of the triangles with the same ID. This simple decomposition, however, has the following drawbacks that degrade the quality of visualized 3D video: Shape distortion: Original shapes of 3D mesh triangles are not preserved. Uniform size: Original size differences among 3D mesh triangles are not preserved. Discontinuity: Connectivity relations among 3D mesh triangles are not preserved. While the former two degrade texture quality, the latter introduces jitters, i.e. visible noisy discontinuities, along triangular face boundaries. To solve these problems, sophisticated methods for geometric transformation between a 3D mesh and a 2D image have been developed, and are referred to as mesh parameterization. Although mainly used for texture mapping, they can be applied to other applications: mesh editing which converts complex 3D algorithms to simpler 2D computations and mesh coding which encodes 3D mesh vertex coordinates (x, y, z) into RGB pixel values of a 2D image. The latter application, which is known as geometry image [9], will be discussed in detail in Chap. 10. Concerning mesh parameterization for texture generation, we basically use the smart uv project implemented in Blender [1]. The algorithm is defined as follows: 1. Cluster faces of a 3D mesh into several groups based on their normal vectors. Note that the clustering does not rely on the connectivity between faces and only processes normal vectors.

5.3 Appearance-Based View-Independent Texture Generation

159

2. For each group, compute the average normal vector of member faces, and derive the orthogonal projection to transform faces from 3D space to 2D plane. Weighting factors proportional to face area sizes are employed to average the normal vectors. Hence normal vectors of larger faces should be respected. This process enables us to transform faces from 3D to 2D while preserving shapes and sizes. 3. For each group, merge neighboring member faces to generated islands on the 2D plane. 4. Tile all generated islands in a specified 2D rectangular area as compactly as possible to obtain a uv unwrapped mesh. Here ‘uv’ denotes the 2D coordinate system for the 2D rectangular area. This method enables us to transform a 3D mesh into a 2D image that almost preserves shape, size, and connectivity relations among 3D mesh triangles. Figure 5.5(c) shows the result of the uv unwrapping of MAIKO with the smart uv project. We refer to the process as uv unwrapping of a 3D mesh. ‘Unwrapping’ denotes the process of the geometric transformation from a 3D mesh surface to a 2D image plane. ‘uv’ may be referred as texture coordinates since surface texture patterns are recorded in the 2D rectangular area specified by the uv coordinate system. As for the practical data structure, a face in a 3D mesh is represented by its constituent: three vertices recording 3D coordinate values as well as texture coordinates. Note that texture coordinates of some 3D vertices may be recorded differently depending on which faces they are considered to belong to. When all faces in a uv unwrapped 2D mesh are painted and/or include texture patterns, we refer to it as texture image. The uv unwrapping is also employed in 3D CG animation, since it enables artists to paint texture patterns using 2D image painting software without painting directly on a 3D mesh surface. The view-independent texture generation methods for 3D video employ uv unwrapping to generate and store surface texture data, while the view-dependent texture generation methods exploit augmented mesh parameterizations for interactive 3D video visualization. The next section presents appearance-based viewindependent texture generation, where a uv unwrapped mesh image is textured based on observed multi-view images.

5.3 Appearance-Based View-Independent Texture Generation In this section we present an appearance-based view-independent texture generation method for 3D video. We demonstrate how significantly rendered 3D video contents are degraded if we neglect the inaccuracy of the camera calibration and errors of the shape reconstruction, and stick to view-independent methods similar to those in computer graphics without introducing a virtual camera into the texture generation process. Algorithm 1 describes the overall computational process. (The notation we employ will be explained in Sect. 5.3.1.) Namely, we first generate a partial texture image where the uv unwrapped mesh image is partially textured by

160

5


Algorithm 1: Appearance-based view-independent texture generation foreach camera C in C do TC ← create a partial texture image (Sect. 5.3.2). T ← combine partial texture images with a specified method (Sect. 5.3.3).

a captured image, and then combine them with a certain criterion. This texture generation method does not take into account the position and direction of a virtual viewpoint, and hence the generated texture is static: it is generated before rendering and never changes during interactive visualization. In what follows, we first introduce the notation and the studio configuration, and then describe the partial texture generation process followed by several methods of combining partial texture images into a full texture image of a given 3D mesh. Object images rendered from several viewpoints are presented to demonstrate how artifacts look like. We discuss reasons for their poor image quality, which leads us to view-dependent texture generation methods.

5.3.1 Notation and Studio Configuration We first summarize the notations we use in this chapter as follows: • Ci (i = 1, 2, . . . , nc ), nc = number of cameras: ith camera placed in the multiview video capture studio. ICi denotes an image captured by Ci . • V Ci : the viewing-direction vector of Ci . • C: a set of the cameras. • M: a reconstructed 3D mesh of an object in motion at t which comprises vertices, vi ∈ R3 (i = 1, 2, . . . , nv ), nv = number of vertices, and faces, fj (j = 1, 2, . . . , nf ), nf = number of faces. Note that we do not denote t explicitly since all the presented texture generation methods process one video frame at a time independently of the others. We use a mesh with triangular faces.1 • N f : a normal vector of face f . • pI f : 2D positions of three constituent vertices of face f on I . • T : a generated texture image. Figure 5.6 shows the configuration of the studio (Studio B in Table 2.3) including cameras and a MAIKO. Note that the checker pattern of the floor was generated just for illustration and the real studio floor is uniformly painted gray as the wall. The 1 As described in Chap. 4, the resolution of reconstructed 3D mesh data is approximately 5 mm in the length between a pair of neighboring vertices. Except for the vertex-based method described in Sect. 5.4, we will use a decimated mesh in this chapter. As a matter of fact, since the other texture generation methods compute texture patterns of mesh faces, the fidelity of rendered object images can be realized with high-resolution texture image data.


161

Fig. 5.6 Studio configuration

studio space is a dodecagonal prism: its diameter and height are 6 m and 2.4 m, respectively. It is equipped with 15 cameras (Sony XCD-710CR; XGA, 25 fps) that have the same lenses except for three cameras: the frontal (Camera 13) and the ceiling cameras (Cameras 14 and 15) are designed to capture zoom-up images of an object. In the figure, the positions, viewing directions, and fields-of-view of the cameras are illustrated with the quadrilateral pyramids. A set of captured multi-view images is shown in Fig. 5.7. At this frame, the MAIKO was out of the field-of-view of Camera 15. Note that most of the images include partial views of the object, to which the intra-frame mesh deformation method presented in Sect. 4.4.1.1 was applied to reconstruct the full 3D object shape. The input and output data for the texture generation method described in this section are the following, assuming uv unwrapping is applied to a 3D mesh before texture generation. Input: Camera parameters, multi-view images, and a 3D mesh with uv texture coordinates.2 Intermediate output: A single texture image generated from multiple partial texture images. Output: A rendered image of 3D video.

2 The original mesh consists of 142,946 vertices and 285,912 faces, and is decimated to 1,419 vertices and 2,857 faces for the face-based methods.

162

5


Fig. 5.7 A set of captured multi-view images

5.3.2 Generating Partial Texture Images A partial texture image is a uv unwrapped mesh image where only surface areas observable from a camera are textured with image data observed by that camera. To generate a partial texture image, we need to know two transformations: (1) projection from a 3D mesh to the image plane of a camera and (2) geometric transformation from a 3D mesh to an unwrapped mesh image. The former is given by camera calibration and the latter is computed by the uv unwrapping process as described in Sect. 5.2. The partial texture image generation process includes the following two steps: (1) generating a depth map to efficiently compute the visibility of 3D mesh surfaces from a specified camera, and (2) generating a texture image from the image captured by the camera. The detailed computational process at each step is described below.

5.3.2.1 Depth Map To create a partial texture image from a captured image, we need to know which parts of the 3D object surface can be observed from a camera. More precisely, we need to know to which surface point of the 3D shape each pixel of the captured image corresponds to. We can compute the correspondence by casting a ray from the projection center of a camera to a pixel on the image plane and computing the point on the 3D mesh surface closest to the intersecting point between the ray and the surface. In practice, we create a depth map to find the correspondences efficiently, instead of computing ray casting every time we need.


163

Algorithm 2: Generating depth map DCi ← create a map with the same geometry of ICi and initialize it to the infinite value. foreach face f in M do if N f · V Ci < 0 then DCi

pf

← compute 2D positions of the vertices of face f on DCi . DC

foreach pixel p inside the triangle defined by pf i do P ← compute the 3D position on M corresponding to p. d ← compute the depth between P and the projection center of Ci . if d < DCi [p] then DCi [p] ← d

Fig. 5.8 Depth maps. Darker pixels are closer to the camera

The depth map is an image that contains depth values instead of colors, where depth denotes the distance from the projection center of a camera to a surface point of the 3D mesh. It is also called depth buffer, Z-buffer, or W-buffer, and often utilized in 3D computer graphics [8, 15]. Algorithm 2 shows a procedure to generate the depth map, DCi for camera Ci . Figure 5.8 shows depth maps visualized as grayscale images, where a darker pixel is closer to a camera. Note that the object is not observable from Camera 15 and thus no depth map is generated for the camera.

164

5


Algorithm 3: Partial texture generation TCi ← create a texture image and initialize the pixels to uncolored. foreach face f in M do if N f · V Ci < 0 then TCi

pf

← compute 2D positions of the vertices of f on TCi . TC

foreach pixel p inside the triangle defined by pf i do P ← compute the 3D position on M corresponding to p . d ← compute the depth between P and the projection center of Ci . p ← compute the 2D pixel position on ICi corresponding to P . if d − DCi [p] < threshold then TCi [p ] ← ICi [p].

Once a depth map is created, we can efficiently find the correspondence between image pixels and surface points by comparing values in the depth map with computed distance values between 3D points on the surface and the projection center of a camera.

5.3.2.2 Partial Texture Image Generation Algorithm 3 shows the procedure of generating the partial texture image, TCi , for camera Ci using the depth map DCi . Note that the geometry (width and height) of the texture image, i.e. the uv unwrapped mesh image, can be specified arbitrarily, but the power of two is effective for graphics processors. Figure 5.9 shows partial texture images generated for the cameras. We do not have the partial texture image for Camera 15 for the same reason as for the depth-map generation.

5.3.3 Combining Partial Texture Images After obtaining a set of partial texture images, we combine them with a certain criterion to create a complete texture image for a 3D mesh. Since the images have the same geometry, the combination process just computes a color value for each pixel from the partial texture images.3 3 We ignore the uncolored pixels in this process. Namely, the average and the median processes described later in this section calculate colors only from colored pixels, and the face normal and the face size processes also choose colors only from colored pixels.


165

Fig. 5.9 Partial texture images. The uncolored pixels are shown in white in the images

We developed the four combination processes described below, namely two image-based and two geometry-based. The former includes the average and median processes and the latter the face normal and face size processes. Note that the view-dependent texture generation method described in Sect. 5.4 uses the average process in its computational process, the harmonized texture generation method described in Sect. 5.5 uses the average, face normal, and face size processes implicitly, and the generic texture generation method described in Chap. 6 uses the median process. Characteristics of each combination process are summarized as follows: Average This process creates a full texture image by averaging color values of the pixels at the same position in all partial texture images. (The uncolored pixels are ignored.) It can create a smooth texture image but introduces blurring artifacts in regions with strong specular reflections, i.e. highlights, because captured colors around such regions vary a lot with viewing directions of cameras. Ghosting, as well as blurring artifacts appear due to the inaccuracy of camera calibration and errors in the reconstructed 3D shape. Median This process is similar to the average process but can reduce blurring artifacts caused by strong highlights, because highlights appear in a narrow range of the reflection angle and can be reduced by taking the median among color values in multi-view images. In this sense, this method can be used for estimating

166

5


the diffuse reflection parameter of the surface. In Chap. 6, actually, the texture image is generated with this method for estimating surface reflectance properties and computing lighting effects on a 3D object surface. However, the generated texture image is noisier than the one generated by the average process in general. Face normal This process extracts a texture image for each face of a 3D mesh from the image captured by the best face-observing camera, i.e. the one most directly oriented toward the face. Since this method does not blend color values, it can create sharp and clear texture images, while textural discontinuities appear at face boundaries where the best face-observing cameras are switched from one to another. Face size Although this process is similar to the face normal process, it takes into account the face size instead of the normal vector. The pros and cons are the same as for the face normal process. This process works well when the size of an object in observed multi-view images varies a lot. The highest resolution, i.e. largest texture image, for a face is then extracted to generate the texture image. Figure 5.10 shows the complete texture images generated by processes mentioned above. We first examine the results of the image-based processes. The average process demonstrates that it can generate a smooth texture image but introduces blurring artifacts (Fig. 5.10(a)). On the other hand, the median process demonstrates that it can generate a sharp and clear texture image. In particular, it reduces ghosting artifacts (doubly projected patterns) on the sleeves, the body, etc., which are introduced by the average method (Fig. 5.10(b)). On the other hand, some noisy areas are introduced at the sash on the object’s back, sleeves, etc. These points will be observed more clearly in the rendering results in the next section. Secondly, the geometry-based processes can generate a sharper and clearer texture image than the image-based ones. We can see very clear patterns on the clothes and the sash in Fig. 5.10(c) and Fig. 5.10(d). However, textural discontinuities between mesh faces appear largely in the images. Sources of the discontinuities can be classified into two types: color mismatch and geometric misalignment. The color mismatch is caused by the inaccuracy of the color calibration of the cameras, as well as the deviation of reflectance properties from the assumed Lambertian. The geometric misalignment is caused by the inaccuracy of the geometric calibration of the cameras and errors in the shape reconstruction. The face size process has more artifacts around the object’s face, sleeves, etc. than the face normal process, because distorted texture images are selected when the sizes of their projected areas on captured images are larger than the others. Consequently, this makes the combined texture distorted. These points will be also clearly shown in the next section.

5.3.4 Discussions Figure 5.11 shows rendering of 3D object images from different viewpoints based on the combined texture images shown in Fig. 5.10. As described in the previous


167

Fig. 5.10 Combined texture images with different criteria

section, each combination process has pros and cons, which can be observed more clearly in the close-up images in the figure. The image-based processes can generate smooth images although blurring and ghosting artifacts are introduced, especially by the average process. This is because the 3D shape and the camera calibration are not perfectly accurate. In the experiment, in fact, the partial texture image generated from Camera 14 (Ceiling camera) is not consistent with the other images (see Fig. 5.9). Consequently, the average process introduces many artifacts into the texture image. The median process can remove such artifacts by excluding outliers even though its blurring factor is larger than in the geometry-based processes.

168

5


Fig. 5.11 Object images rendered with different texture images

On the other hand, the geometry-based processes can generate very sharp and clear images. Nevertheless, although clearer patterns on the sash, the sleeve, and so on can be generated, discontinuity artifacts on the surface, especially on the sash, are introduced. As described in the previous section, the reflectance of FURISODE is not perfectly Lambertian since it is made of silk and some patterns are embroidered. As readers may have noticed, the process of combining partial texture images described here has much to do with the photo-consistency evaluation of the 3D shape reconstruction described in Sect. 4.3.1. Even though their objectives are different, as the former is a combination and the latter an evaluation, multi-view images are compared with each other with similar criteria in both cases. Moreover, both come up with the same problem: how to cope with non-Lambertian surfaces. In addition to this problem, texture generation for 3D video should comply with contradicting requirements, namely smoothness and sharpness under inaccurate camera calibration and erroneous shape reconstruction. This section has proven that a simple view-independent texture generation method based on the texture mapping in CG cannot generate high quality texture for 3D video. To solve these problems, the next section will introduce a view-dependent texture generation method, which uses a virtual viewing point for 3D video visualization into the texture generation process.

5.4 View-Dependent Vertex-Based Texture Generation

169

5.4 View-Dependent Vertex-Based Texture Generation As discussed in the previous section, we need to solve the following contradictory problems in addition to the non-Lambertian surface problem to generate high quality texture for 3D video. Smoothness: If blending is simply applied to smooth texture, then blurring and ghosting artifacts are created. Sharpness: If the best face-observing image is simply selected and copied to avoid blurring and ghosting, then texture discontinuity artifacts are created. If perfectly accurate camera parameters and 3D shape of an object could be obtained, like in 3D CG animation, these problems would never be introduced. In practice, however, the errors are inevitable. Thus, we have to solve the problems by augmenting texture generation algorithms. In this section, we introduce two ideas to cope with these problems: viewdependent and vertex-based texture generation: View-dependent The view-dependent texture generation dynamically updates a texture image generated from multi-view images based on a virtual viewpoint of a 3D object [4]. Namely, it blends multi-view images by controlling weighting factors computed using angular differences between viewing directions of a virtual camera and the real cameras. This method can reduce both blurring and discontinuity artifacts by smoothly blending the multi-view images as well as dynamically changing the best face-observing camera: larger weighting factors are given to cameras having viewing directions closer to the virtual camera. This method ensures that exactly the same object appearance as in the captured image is rendered when the viewing direction of a virtual camera coincides with one of the real cameras. Moreover, the non-Lambertian surface problem can be solved by using the following method; the texture generation can simulate the viewing-direction dependency of non-Lambertian surface appearances. When the viewing direction of a virtual camera is close to be aligned with the light ray direction that is specularly reflected on a non-Lambertian surface, the view-dependent texture generation method generates a texture image representing well such specular reflection. On the other hand, when the viewing direction of a virtual camera is far away from the specularly reflected light, a texture image without specular reflections is generated. Such dynamic appearance changes of a surface depending on its viewing angles make the surface to be perceived as made of a shiny material. It should be noticed that this simulation is valid only under the lighting environments in which the multi-view video data are captured. In other words, the view-dependent texture generation encodes the lighting environments into generated texture images as well as the 3D object shape and its surface texture. Hence we cannot modify lighting environments freely as in computer graphics. In this sense, the viewdependent texture generation is just as one of the appearance-based texture generation methods under fixed lighting environments.

170

5


Vertex-based The vertex-based texture generation method generates a texture image by interpolating vertex colors. This method never generates discontinuity artifacts because neighboring faces share the same vertices. However, one disadvantage of this method is that a very high-resolution 3D mesh is required to represent fine details of the surface texture pattern. As for the vertex-based 3D shape representation and surface texture generation, see the point-cloud-based technologies in [7]. In what follows, we present the algorithm of the view-dependent vertex-based texture generation,4 and then discuss the quality of rendered images and the problems of this method.

5.4.1 Algorithm The input data is the same as in Sect. 5.3.1 except for the mesh resolution. We use the original mesh with 5 mm resolution rather than the decimated one because the vertex-based method requires a dense mesh. We introduce the following notations in addition to those defined in Sect. 5.3.1. v: a vertex of M. N v : a normal vector of vertex v, which is computed by averaging the normals of the faces that share vertex v. ˆ V Cˆ : a viewing direction of the virtual camera, C. wCi : a weighting factor of camera Ci to be used for combining multi-view images. c[v]: a vertex color of v. c˚ [v, Ci ]: a vertex color of v extracted from the image, ICi , captured by Ci . Algorithm 4 shows the computational process. We first compute color values of the vertices visible from each camera as an offline processing. This vertex visibility test is done using multi-view depth maps as described in Sect. 5.3.2.1, while color values of the invisible vertices are specified as uncolored. We then dynamically compute weighting factors and color values of vertices viewed from the virtual camera in real time. The weighting factors are computed as the dot products of the viewing directions between the virtual camera and the real cameras. When the viewing directions between the virtual camera and a real camera get closer, the weighting factor for the real camera becomes almost 1.0 whereas the others become almost 0.0.5 Thus, the generated texture image will be equivalent to the image captured by the camera. After obtaining the color values of all visible vertices from the virtual camera, the texture of each face is generated by interpolating the vertex colors as follows: 4 An

earlier version was published in [13].

5 The

sharpness of this weighting factor attenuation is controlled by the γ value in Algorithm 4. We heuristically set γ = 50 in the following experiments.

5.4 View-Dependent Vertex-Based Texture Generation

171

Algorithm 4: View-dependent vertex-based texture generation /* OFFLINE: /* Extract vertex colors from each camera. foreach vertex v in M do foreach camera Ci in C do c˚ [v, Ci ] ← extract a color value of v if it is visible from Ci .

*/ */

/* ONLINE: /* Compute vertex color values for the virtual viewpoint. V Cˆ ← specify the viewing direction of the virtual viewpoint. foreach camera Ci in C do V Cˆ · V Ci + 1 γ wCi ← , 2

*/ */

(5.1)

where γ denotes a control parameter for weighting. foreach vertex v in M do if N v · V Cˆ < 0 then /* Update weighting factors. foreach camera Ci in C do if c˚ [v, Ci ] is uncolored then w¯ Ci ← 0 else w¯ Ci ← wCi /* Compute vertex color. c[v] ←

nc i

w¯ C nc i c˚ [v, Ci ]. ¯ Cj j w

*/

*/

(5.2)

foreach face f in M do Generate texture of f by interpolating colors, c[v], of all vertices of f (see text).

• All vertices of a face have color values: generate texture for the face by bilinearinterpolation of all the values. • Two vertices of a face have color values: generate texture for the face by linear interpolation of the two values. • One vertex of a face has a color value: paint the face with that value. • No vertex of a face has a color value: no texture is generated (paint the face with a default color such as gray).

172

5


Fig. 5.12 Images rendered by the view-dependent vertex-based texture generation

5.4.2 Discussions Figure 5.12 shows 3D object images rendered by the view-dependent vertex-based texture generation. The viewpoints for generating these images are the same as those used for generating Fig. 5.11. As shown in the figure, it can generate sharper and clearer images than the image-based processes, and smoother images than the geometry-based processes. For example, we can clearly see the face of MAIKO and the colorful patterns on the sash and the sleeve, and cannot see any strong discontinuity artifacts on the surface. Figure 5.13 demonstrates the effect of simulated non-Lambertian surface appearances by the view-dependent method. With the view-dependent method, the shiny appearance of the sash changes depending on the viewing direction of the virtual camera, while the appearance is fixed with the view-independent method. These rendered images demonstrate the effectiveness of the view-dependent vertex-based method for 3D video texture generation. As a conclusion of this section, we have to mention that the quality of the viewdependent vertex-based texture generation remains lower than the quality of the captured images, because the rendered images are still blurred by the vertex-based texture interpolation. Although we can generate 3D mesh data of finer resolution by dividing faces of the original mesh, the improvement of rendered image quality is limited due to calibration and shape reconstruction errors [13].

Fig. 5.13 Simulated appearances of the non-Lambertian surface, the sash of MAIKO. Upper: the view-dependent vertex-based method. Lower: the view-independent surface-based method with the surface normal process

5.5 Harmonized Texture Generation

173

Moreover, this method does not solve the issues related to the inaccuracy of the camera calibration and errors in the shape reconstruction directly, even though it can reduce the artifacts up to a certain level. In the next section, we introduce the best texture generation method for 3D video to our knowledge, namely, harmonized texture generation, which can cope with these problems and generate non-blurred texture by dynamically deforming the geometry of the texture as well as the color patterns.

5.5 Harmonized Texture Generation 5.5.1 Background As discussed in the previous section, while the view-dependent vertex-based texture generation enables us to visualize 3D video in better image quality than the view-independent method, there still remain possibilities of further improvements. In this section, we present the harmonized texture generation method.6 Its most distinguishing characteristics are that the inaccuracy of the camera calibration and errors in the shape reconstruction are explicitly computed and managed in the texture generation process, and surface texture patterns can be generated in almost the same quality as originally captured images. The key ideas of the harmonized texture generation are (1) view-dependent deformation of local surface textures and geometric shapes, and (2) mesh optimization that works together with the deformation. Idea (1) enables us to render sharp and smooth images by dynamically deforming captured multi-view local images so that they become consistent with each other on the reconstructed 3D object surface. Idea (2) leads us to the mesh optimization to facilitate the texture deformation and rendering. These ideas enable the harmonized texture generation to generate high quality texture for 3D video even if the reconstructed 3D mesh deviates from the actual 3D object shape due to the inaccuracy of the camera calibration and errors in the shape reconstruction. Incidentally, Eisemann et al. proposed floating textures [6], which reduce blurring and ghosting artifacts by dynamically warping local texture patterns depending on a virtual viewpoint so that the textures are consistent to each other. They assume that the camera parameters and the reconstructed 3D shape are almost accurate but not perfect, and thus small pixel-wise deformations are sufficient to compensate texture inconsistency. Although the basic concept of the floating textures is similar to the harmonized texture generation, differences are (1) the former processes pixels while the latter mesh data, and (2) the latter uses 3D mesh optimization for texture deformation that enables more effective deformations of local texture patterns. Distinguishing characteristics of the harmonized texture generation which the floating textures do not share are listed as follows: 6 An

earlier version was published in [20].

174

5


Adaptive texture deformation: The degree of inconsistency among the captured multi-view images on the reconstructed 3D mesh, which is incurred by errors in the calibration and the shape reconstruction, varies depending on local surface areas. Errors in concave areas are often larger than those in convex areas. In 3D video of complex object shape and motion, especially like MAIKO dances, accurate reconstruction of heavy concavities with a limited number of cameras are very difficult. To cope with the uneven shape error distribution, the harmonized texture generation conducts a coarse-to-fine 3D mesh optimization by evaluating degrees of consistency in both local surface shape and texture. With this function, it can adaptively control where and how much local textures should be deformed. Nonlinear texture deformation: Coupled with the mesh optimization, the locally linear face-based texture deformation can realize globally nonlinear texture deformation, which enables the harmonized texture generation to cope with larger errors in the calibration and the shape reconstruction. Real-time rendering: The harmonized texture generation can render images in real time (≥30 fps) with 15 viewpoint images, since it performs the 3D mesh optimization as an offline process before rendering.

5.5.2 Algorithm Overview The key idea of the harmonized texture generation, i.e. the view-dependent deformation of multi-view local surface textures coupled with the 3D mesh optimization, enables us to render non-blurred and high quality images even with large local errors in the calibration and the shape reconstruction. It admits the errors and harmonizes multi-view images based on a specified viewpoint. The following notations are used to describe the computational process of the harmonized texture generation in addition to those defined in Sects. 5.3.1 and 5.4.1. Mˆ Ci (i = 1, 2, . . . , nc ): the mesh projected onto the image plane of camera Ci . We refer to this mesh as a projected mesh. Mˆ Ci

pv

∈ R2 : the 2D position of a projected vertex of Mˆ Ci . This value is valid only when vertex v is visible from Ci .

Mˆ Ci

If

: the triangular image area, i.e. a group of pixels, in ICi defined by the projected face f of Mˆ Ci . We refer to this image as a face-image segment.

Mˆ Ci ←Mˆ Cj

If

Mˆ Cj

: a transformed version of If

, which is transformed so that its geMˆ Ci

ometry as a group of pixels is aligned with that of If Mˆ C If j

Mˆ C If i

. With this transforma-

and can be computed by a tion, the texture discrepancy between simple pixel-based similarity evaluation. We refer to this image as a transformed face-image segment.

5.5 Harmonized Texture Generation Mˆ Ci

Iv

: a polygonal image region on the image plane of camera Ci consisting of a Mˆ Ci

set of neighboring If image segment.

Mˆ Ci ←Mˆ Cj

Iv

175

s that share vertex v. We refer to this image as a vertexMˆ Cj

: a transformed version of Iv

, which is transformed so that its geMˆ Ci

ometry as a group of pixels is aligned with that of Iv Mˆ C Iv j

. With this transforma-

Mˆ C Iv i

and can be computed by a tion, the texture discrepancy between simple pixel-based similarity evaluation. We refer to this image as a transformed vertex-image segment. T¯ : the harmonized texture image that is defined on the image plane of the userspecified virtual viewpoint. The geometry of the image, i.e. width, height, and resolution, is also specified by the user. T¯f denotes a texture segment of face f in T¯ . As shown in Fig. 5.14, the computational flow of the harmonized texture generation is divided into two stages, i.e. the view-independent mesh optimization (offline) and the view-dependent texture deformation (online): 1. Mesh simplification: In the offline stage, we optimize the 3D mesh reconstructed from multi-view images so that the texture deformation in the succeeding stage can be facilitated. The mesh optimization is done based on a coarse-tofine strategy. That is, we first reduce the number of faces of the 3D mesh so that the size of each face is large enough to include sufficient feature points for the later matching. 2. Mesh refinement and deformation vector computation: We then refine the mesh by subdividing faces for a precise texture deformation. In other words, the mesh optimization restructures the 3D mesh with respect to texture consistency as well as shape preservation. To assess surface texture misalignments among multi-view images incurred by shape reconstruction errors, we first compute deformation vectors for each projected vertex of the mesh on each image plane. A deformation vector for vertex v on the image plane of camera Ci represents a misalignment between its projected images in a pair of observed images ICi and ICj (cf. Fig. 5.17). The detailed computation algorithm is given below. With the deformation vectors, the degree of texture misalignments is computed for each face. Then, faces with large misalignments are recursively divided into sub-faces until the misalignment degree or the face size becomes small enough. Note that we do not try to reconstruct a more accurate 3D shape in the refinement process because: (1) we have already optimized the shape by the 3D shape reconstruction process described in the previous chapter, and (2) it is hard to estimate the shape more accurately with inaccurate camera parameters. The deformation vectors are also used as key information for the view-dependent texture deformation in the succeeding stage. 3. Harmonized position computation: In the online stage, we first compute the harmonized position of each projected vertex on each image plane with the deformation vectors and the user-specified virtual viewpoint. Note that the harmo-

176

5


Fig. 5.14 Processing flow of the harmonized texture generation

nized position is changed depending on the viewing direction. The harmonized positions of three vertices of a face specify from which areas in each multiview image the texture image of that face is extracted for view-dependent texture blending. 4. Texture extraction: Texture images are extracted for each face of the mesh using the harmonized positions from the multi-view images. 5. Texture transformation and blending: Extracted textures are transformed into harmonized texture images, in which the extracted textures can be compiled. We finally generate a non-blurred and high quality texture for the 3D video object by blending the harmonized texture images depending on the virtual viewpoint. In summary, the harmonized texture generation implicitly realizes a nonlinear texture deformation depending on the user-specified viewpoint as well as a mesh optimization for an effective deformation. It enables us to generate non-blurred and high quality texture images even with inaccurate camera parameters and errors in the shape reconstruction. In the following sections, we describe the algorithms of the processes above in detail.


177

5.5.3 Mesh Optimization As described in the previous section, the mesh optimization consists of two processes: mesh simplification and refinement. In the mesh simplification, we first reduce the number of faces of the 3D mesh by edge collapse operations with shape preservation [10]. Then, we collapse edges of faces whose textures extracted from multi-view images are inconsistent with each other. An ordinary mesh simplification method generally collapses redundant edges alone and does not include the process of merging texture-inconsistent faces, because such additional operation usually increases the discrepancy between the original and reduced meshes. The harmonized texture generation, on the other hand, eliminates such textureinconsistent faces and represents them by larger faces. Although it may sound strange, this is the point of the mesh simplification of the harmonized texture generation. The basic idea is described in what follows. In surface areas with large texture inconsistency, 3D shape is not accurately reconstructed mainly due to heavy surface concavities. This is because it is difficult to accurately reconstruct concave areas by silhouette-based and wide-baseline stereo-based 3D shape reconstruction methods. Furthermore, inaccurate camera parameters can produce large texture inconsistencies as multi-view images may be projected onto the 3D mesh jumping over sharp surface concavities. In order to extract consistent textures for such areas, we need to deform images more dynamically than in texture-consistent areas. To realize large dynamic image deformation, larger faces are needed and therefore such texture-inconsistent faces are eliminated and represented by larger faces. Consequently, the mesh simplification method employed in the harmonized texture generation enables us to adaptively reduce the 3D mesh with respect to both shape preservation and texture inconsistency. After the mesh simplification, we refine the mesh by subdividing faces to include sufficient deformation vectors where textures are inconsistent.

5.5.3.1 Mesh Simplification Algorithm 5 shows the procedure of the mesh simplification. In the first step, the number of faces is reduced in order to make faces large enough to include sufficient image features. Then, the face-image-segment discrepancy of each face is computed, which is given by Algorithms 6 and 7 (see also Fig. 5.15), to evaluate texture inconsistency of the faces. After obtaining a set of face-image-segment discrepancies, the face with the maximum value of discrepancy is selected. If its area is smaller than a certain threshold, then it is merged with one of its neighboring faces. By iterating this process until no face is merged, we can sufficiently simplify the mesh to realize the dynamic image deformation.

178

5


Fig. 5.15 Computing the texture discrepancy of a face

Algorithm 5: Mesh simplification repeat Mesh Simplification by edge collapse with shape preservation. until average size of faces > threshold. continue ← true. repeat foreach face f in M do Df ← compute face-image-segment discrepancy of f (see Algorithm 6). f ← find a face that has the maximum Df and its area < threshold. if f is empty then continue ← false else Find one of the three edges of f which, after elimination, preserves shape the most, and collapse it. until continue is false.

5.5.3.2 Mesh Refinement and Deformation Vector Computation Following the mesh simplification, deformation vectors and mesh refinement are computed by subdividing faces to include sufficient deformation vectors. Algorithm 8 shows the overall procedure of the mesh refinement and the deformation vector computation. First, the deformation vectors by vertex-image-segment matching is computed (described later). Then, for each face, the discrepancy of the deformed face-image segment that is generated using the deformation vectors is computed. This discrepancy indicates how much the texture consistency of a face can be improved using the deformation vectors. A large value of the discrepancy implies that the deforma-


179

Algorithm 6: Computing face-image-segment discrepancy of face f foreach face f in M do foreach camera Ci in C do /* Compute a camera index that evaluates how well a camera captures a face directly from its front as well as in an image resolution. */ eCi ← compute camera index of face f by Algorithm 7. C ← find the best face-observing camera that has the largest camera index. Compute the face-image-segment discrepancy of face f by n

C Mˆ C 1 Mˆ C ←Mˆ C , Df = eCi If i − If i nC − 1

(5.3)

i,i=

where C and nC denote a set of cameras that can observe face f and the Mˆ Ci

number of such cameras, respectively. If the difference between

Mˆ C If i

and

Mˆ Ci ←Mˆ C

− If

Mˆ C ←Mˆ C If i ,

Mˆ Ci Mˆ C ←Mˆ C = 1 I − If i f 3np p

which is given by

Mˆ Ci I [p, c] f

c∈{R,G,B}

Mˆ Ci ←Mˆ C

− If Mˆ C

computes

[p, c],

(5.4)

Mˆ C ←Mˆ C

[p, c] denote the intensity of color where If i [p, c] and If i band c at pixel p in the triangular area of each face-image segment, respectively, and np the number of pixels in the triangular area. As

Mˆ C

Mˆ C ←Mˆ C

share the described in the beginning of Sect. 5.5.2, If i and If i same pixel-based representation. Note that the difference values are weighted with the camera indices. Hence, difference values computed for cameras with larger camera indices affect largely texture discrepancy.

tion vectors are not sufficient to generate consistent texture and face subdivision is required for more dynamic deformation. Furthermore, faces without textures can exist due to self-occlusion, as shown in Fig. 5.16(a), because the texture is generated only for a face where all of its vertices are visible at least from one camera. Thus, faces without textures are also

180

5


Algorithm 7: Computing camera index for face f Compute camera index eCi defined by eCi = α d¯Ci + (1 − α)¯sCi ,

(5.5)

where dC d¯Ci = nc i , j dCj

−V Ci · N f + 1 2

γ1

sC s¯Ci = nc i , j sCj (5.6) • sCi : area of the face projected on the image plane of Ci , and • α, γ1 : weighting coefficient. Values of α and γ1 are heuristically defined as 0.3 and 10, respectively. dCi =

,

Algorithm 8: Mesh refinement and deformation vector computation repeat a set of deformation vectors ← compute vertex-image-segment matching (see Algorithm 9). a set of discrepancies ← compute the deformed face-image-segment discrepancy of each face with the deformation vectors (see Fig. 5.18 and text). continue ← evaluate discrepancies and self-occlusion, and then subdivide faces if required (see Algorithm 10). until continue is false.

Fig. 5.16 Subdivision of partially occluded faces. The gray colored faces are partially occluded by blue faces. Some of such faces are subdivided as shown in (b) to be textured. The red circles and lines denote newly introduced vertices and faces. ©2009 ITE [21]

marked to be subdivided for generating textures with as large as possible surface areas (Fig. 5.16(b)). In the third step, faces are subdivided considering the discrepancy and the selfocclusion at the same time. By iteratively conducting the above processes, a refined mesh and a set of the deformation vectors are finally obtained. In the following, we describe the detailed procedures of the algorithm.


181

Fig. 5.17 Computing deformation vectors by the vertex-image-segment matching

Computing deformation vectors by vertex-image-segment matching Figure 5.17 illustrates the computation of deformation vector by vertex-imagesegment matching. As shown in the figure, a vertex is projected onto each image plane with the camera parameters as illustrated by green circles and dashed lines. Here the focus is on image ICj . A vertex-image segment of projected Mˆ C

Mˆ C

vertex pv j , Iv j , is extracted from ICj as represented by the red polygonal region. Then, it is transformed onto the image plane of camera Ci , which Mˆ Ci ←Mˆ Cj

is denoted by a transformed vertex-image segment, Iv

as shown in the

Mˆ C ←Mˆ Cj Mˆ C figure. The best match position of Iv i around p v i on ICi is sought Mˆ C ←Mˆ Cj where the image appearances of Iv i and ICj are the most similar to Mˆ C ←Mˆ Cj each other. The best match position is denoted by pv i , which is illusMˆ Ci trated by a filled circle near the projected vertex p v on ICi . If the 3D shape Mˆ C ←Mˆ Cj and the camera parameters were perfectly accurate, pv i would coincide Mˆ Ci with p v , i.e. the projected position of vertex v on the image plane of Ci . In Mˆ C ←Mˆ Cj Mˆ C practice, however, pv i is located away from p v i due to errors in the calMˆ C ←Mˆ Cj Mˆ C ibration and shape reconstruction. The displacement from p v i to p v i is Mˆ Ci ←Mˆ Cj Mˆ Ci ←Mˆ Cj Mˆ Ci denoted by the deformation vector v v = pv − p v . Note that

a set of deformation vectors are computed for each projected vertex on each image; the displacements change depending on which projected vertex is used as key reference point. Algorithm 9 shows the procedure described above in detail. Computing the deformed-face-image-segment discrepancy Figure 5.18 illustrates the procedure of computing the discrepancy value of a deformed face-

182

5


Algorithm 9: Computing deformation vectors by the vertex-image-segment matching ˆ = {Mˆ Ci |i = 1, 2, . . . , nc } ← compute projected meshes with M and a set of M cameras C. ˆ do foreach projected mesh Mˆ Ci in M foreach vertex v in M do ˆ do foreach projected mesh Mˆ Cj in M if vertex v is not visible from Ci then Mˆ C ←Mˆ C

j vv i ← undefined. else if vertex v is not visible from Cj or Cj = Ci then

Mˆ Ci ←Mˆ Cj

vv else

Mˆ Cj

Iv

←0

← extract the vertex-image segment from ICj .

Mˆ C ←Mˆ Cj Iv i

Mˆ C

← transform Iv j from the image plane of Cj to that of Ci . /* Search the best matched position. */ Mˆ Ci ←Mˆ Cj

pv

← search the corresponding position in ICi Mˆ C ←Mˆ C

j which matches best with Iv i . /* Compute the deformation vector.

Mˆ C ←Mˆ Cj vv i

Mˆ C ←Mˆ Cj ← pv i

*/

Mˆ C − pv i .

Note that the deformation vector denotes a 2D vector on the image plane of camera Ci .

image segment. The computation process is similar to the evaluation process of the texture difference of a face-image segment described in Algorithm 6. For each face f , (1) first determine its best face-observing camera C . (2) For each camera Ci , (2-1) compute the best match positions on IC for three verMˆ C

Mˆ C ←Mˆ C

i , k = 1, 2, 3. tices of face-image segment If i on ICi , respectively: pvk (2-2) Extract the triangular area from IC defined by the three best match po-

ˆ

MC ,C sitions. This area is referred as a deformed face-image segment of f , Iˆf i . (2-3) Transform the deformed face-image segment so that its geometry as a group Mˆ Ci

of pixels coincides with If

. (2-4) Compute the discrepancy value between the ˆ

ˆ

ˆ

MC ←MCi MC transformed deformed face-image, Iˆf , and If i by the image differ-


183

Fig. 5.18 Computing the deformed-face-image-segment discrepancy

ence computation used in Algorithm 6. (3) Just as in Algorithm 6, compute the deformed-face-image-segment discrepancy of f , Dˆ f , as the weighted average of the discrepancy values computed for all cameras that can observe f . Evaluation of discrepancies, self-occlusion, and subdivision of faces With Dˆ f , we can evaluate the effectiveness of the deformation vectors computed for f . A small value of Dˆ f implies that the deformation vectors are good enough to compensate the image inconsistency on the face. Usually most faces have small values because the reconstructed 3D shape and camera parameters are almost accurate. In contrast, a large average value of Dˆ f implies the texture patterns on f have large mutual inconsistencies, which leads to subdividing f to localize such inconsistencies. Moreover, self-occlusion is inspected as described at the beginning of this section to find texture-less faces, which then are also subdivided for possible texture generation for subdivided faces. The face subdivision is performed by adding vertices at the center of each edge of a face. To preserve the topology of the 3D mesh, first, faces that are required to be subdivided are found, and then a type (Type 0 to 4) is assigned to all the faces based on the state of the adjacent faces as shown in Fig. 5.19. Finally, the faces are subdivided at the same time after the assignment is completed. See Algorithms 10 and 11 for the detailed procedure.

5.5.4 View-Dependent Texture Deformation With the optimized mesh and the deformation vectors, textures are dynamically deformed depending on the user-specified viewpoint. Figure 5.20 illustrates the processes of the view-dependent texture deformation including the harmonized position computation, extraction of deformed texture segment, transformation and blending of texture segments.

184

5


Fig. 5.19 Subdivision patterns of a face. The central face is subdivided depending on the number of the adjacent faces that are supposed to be subdivided. Type 0 illustrates the face that will not be subdivided, and Types 1–3 illustrate how the face is subdivided depending on the states of its adjacent faces. Type 4 is different from the others, that is, the face is marked to be subdivided by the discrepancy evaluation step and/or the occlusion test. Red circles denote the vertices introduced by the subdivision process. ©2009 ITE [21]

Algorithm 10: Evaluation and subdivision of faces Compute the average of the deformed-face-image-segment discrepancies by 1 ˆ Dˆ avg = Df . nf

(5.7)

f

if Dˆ avg > threshold or subdivision is required due to the occlusion then Subdivide faces (see Algorithm 11). return true else return false

Algorithm 11: Subdivision of faces foreach face f in M do if Dˆ f > threshold or subdivision of f is required due to the occlusion then typeOfFacef ← 4 foreach face f in M do if typeOfFacef = 4 then typeOfFacef ← 0 foreach adjacent face fa of f do if typeOfFacefa = 4 then typeOfFacef ← typeOfFacef + 1 foreach face f in M do if typeOfFacef > 0 then Subdivide face f as shown in Fig. 5.19 by typeOfFacef .


185

Fig. 5.20 Generating harmonized texture from the deformed textures

Harmonized position computation When viewing direction VCˆ of virtual camera Cˆ is specified, weighting factors for multi-view images can be computed, respectively, and then harmonized positions for extracting deformed texture segments as well. Algorithm 12 shows the procedure of computing harmonized positions with the deformation vectors. In the algorithm, the weighting factor, w, is computed by the dot product between the real and virtual camera’s viewing directions, i.e. a larger value means that their directions are closer. The weighting factor controls how much a texture generated from a camera has to be deformed: a larger weighting factor realizes a smaller texture deformation. Namely, a texture generated from a camera with a larger value is not so much deformed and hence the image captured by that camera is preserved. This enables us to generate texture patterns without any deformation for such camera having similar viewing direction as the virtual one.7 Mˆ C

Texture extraction The texture segment of face f on image ICi , Tf i , is defined as a triangular image area on ICi specified by the harmonized positions of three Mˆ Ci

vertices of f . Tf

is the deformed version of the face-image-segment of f on

Mˆ C If i ,

ICi , which is illustrated as a green triangle in Fig. 5.20. With this deformation, the inconsistency among multi-view texture patterns can be reduced. Texture transformation After obtaining the texture segments of f from multiview images, they are transformed onto the same image plane of the harmonized texture image. As described in the beginning of Sect. 5.5.2, the harmonized tex7 We

use a coefficient γ2 for controlling the weighting factor which is heuristically defined as γ2 = 50. With this value, the normalized weighting factor (in Eq. (5.9)) for the camera that has the same viewing direction as the virtual one becomes almost 1.0 and the others 0.0.

186

5


Algorithm 12: Computing the harmonized position of vertex v on each image plane ˆ V ˆ ← specify the viewing direction of virtual camera C. C

foreach camera Ci in C do Compute weighting factor by wCi =

V Cˆ · V Ci + 1 2

γ2 (5.8)

,

where γ2 denotes a weighting coefficient. foreach camera Ci in C do foreach visible vertex v in M from Ci do

Mˆ Ci

Compute the harmonized position of v on image ICi , p¯ v weighting factors and the deformation vectors by Mˆ C p¯ v i

Mˆ C = pv i

+

nc j

wCj Mˆ C ←Mˆ Cj vv i . C k wCk

, with the

(5.9)

ture image is defined on the image plane of the virtual camera, and the geometry is specified by the user. See the first half of Algorithm 13 for details. Texture blending We finally blend the transformed textures with the weighting factors computed in Algorithm 12, and generate the harmonized texture image for the 3D video object. See the second half of Algorithm 13 for details.

5.5.5 Experimental Results We evaluate the performance of the harmonized texture generation with two 3D video data streams of MAIKOs and one of Kung-fu. Data characteristics are as follows: MAIKO 1: a woman wearing a green FURISODE (Fig. 5.21). Both sides of her body and a part of the broad sash have large concavities that could not be reconstructed accurately. The sash on her back is reconstructed thicker than the actual, and thus the texture patterns on it that are extracted from captured multi-view images are not consistent with each other. MAIKO 2: a woman wearing a red FURISODE, as used in the previous sections. Kung-fu: a man performing Kung-fu. Clothes are simpler than MAIKOs’. All data were captured in Studio B, as described in Sect. 5.3.1.


187

Algorithm 13: Texture extraction and blending foreach face f in M visible from virtual camera Cˆ do /* Texture extraction and transformation. foreach camera Ci in C do if f is observable from Ci then

*/

Mˆ C

Tf i ← extract the texture segment of f from ICi using the harmonized positions. ˆ

ˆ

ˆ

M ˆ ←MCi MC ˆ Note this ← transform Tf i from Ci to C. Tˆf C Mˆ Ci

transformation makes Tf face-image segment

Mˆ ˆ If C

align pixel-wisely with the

of f on the image plane of Cˆ

/* Texture blending. foreach pixel p in T¯f do foreach camera Ci in C do

*/

ˆ

ˆ

M ˆ ←MCi if Tˆf C exists then w¯ Ci ← wCi else w¯ Ci ← 0

Compute color at p by T¯f [p] =

nc i

w¯ C Mˆ ˆ ←Mˆ Ci nc i Tˆf C [p], ¯ Cj j w

(5.10)

where Tf [p] denotes the color value at pixel p in face-image Mˆ Cˆ

segment If ˆ

ˆ Note that T¯f and of f on the image plane of C.

ˆ

M ˆ ←MCi Tˆf C share the same pixel-wise shape representation.

5.5.5.1 Mesh Optimization Figure 5.22 shows the 3D mesh data of MAIKO 1 generated during the mesh optimization process. Figure 5.22(a) shows the original mesh with 107,971 vertices and 215,946 faces, and Fig. 5.22(b) the mesh simplified for each face to include sufficient image features for matching. Figure 5.22(e) illustrates the simplified mesh colored with the texture discrepancy computed by Algorithm 6, where high discrepancy value is painted in red-yellow, medium in green, and low in lightdark blue. We can observe that large texture discrepancy appears on the side of the body, sash, sleeves and hem of the FURISODE. Texture inconsistencies of

188

5


Fig. 5.21 Captured images of MAIKO 1. ©2009 ITE [21]

the side and sash are caused by concavities. Others are mainly due to the occlusions from the ceiling cameras. The latter are not easy to compensate by the local texture deformation alone, because errors of the calibration and shape reconstruction produce large texture misalignments. In such occluded areas, very different multi-view texture patterns are generated on corresponding mesh surface faces. Actually, the texture of a head area observed from the ceiling camera is projected onto a part of the hem (a lower part of the mesh, which shows a larger difference value than the other areas). This causes an artifact of texture switching when the virtual camera moves between the ceiling cameras and the others. Figure 5.22(c) shows the mesh after the second step of mesh simplification based on the texture consistency. As shown in Fig. 5.22(f), faces with inconsistent texture are merged to occupy larger surface areas on the mesh. Figures 5.22(d) and (g) show the final result of the mesh optimization. The mesh refinement process partitions and localizes faces with inconsistent texture that were enlarged by the previous process. This result demonstrates that the original 3D mesh is adaptively optimized with regard to texture consistency and shape preservation.

5.5.5.2 Quantitative Performance Evaluation We align a virtual camera with the same parameters of the real camera Ci , and render an image IC i , without using ICi , the image captured by Ci . We then evaluate the performance of the harmonized texture generation by computing PSNR between ICi and IC i . For comparison, we also evaluated PSNRs of the view-independent


189

Fig. 5.22 Mesh optimization results. ©2009 ITE [21]. Images (a)–(d) show the original and resulting mesh data generated by the mesh optimization process. ‘Simplified-S’ and ‘Simplified-T’ denote the simplified mesh data with respect to shape preservation and texture consistency, respectively. Images (e)–(g) illustrate the spatial distributions of texture difference values for the mesh data in (b), (c), and (d), respectively, where red implies large texture difference values, green medium, and blue small

average method (described in Sect. 5.3) and the view-dependent vertex-based method (described in Sect. 5.4). We performed the evaluation for all cameras in the studio except the frontal and ceiling cameras, i.e., 12 cameras from the top-left in Fig. 5.21. Figure 5.23 illustrates the result of the performance evaluation, which demonstrates the harmonized texture generation can generate much higher fidelity images than the other methods.

190

5


Fig. 5.23 Quantitative performance evaluation of three texture generation methods. ©2009 ITE [21]

5.5.5.3 Qualitative Image Quality Evaluation Figure 5.24 shows close-up captured images and corresponding images generated by the view-independent average method, the view-dependent vertex-based method, and the harmonized texture generation method. Although the first method generates textures with almost the same sharpness as the captured images, blurring and ghosting artifacts appear on the broad sash and the sleeves due to errors of the camera calibration and the shape reconstruction. The images generated by the view-dependent vertex-based method also contain blurring artifacts due to the interpolation of vertex colors as well as the errors. On the other hand, the harmonized texture generation method can obviously reduce such artifacts and render high quality images.

Fig. 5.24 Images for qualitative image quality evaluation (MAIKO 1). ©2009 ITE [21]


191

Fig. 5.25 Images for qualitative image quality evaluation (MAIKO 2)

Figure 5.25 illustrates another set of rendered images, by the above mentioned three methods and the view-independent face normal method (described in Sect. 5.3). As shown in the figure, the harmonized texture generation method demonstrates its effectiveness in reducing blurring, ghosting, and discontinuity artifacts drastically. Note also that the view-independent average method cannot render shiny areas on the sash correctly, whereas the other three can generate highlights with high fidelity.

5.5.5.4 Real-Time Rendering Lastly, Table 5.2 shows the computational time of each process of the harmonized texture generation for rendering one frame. The numbers of faces of optimized MAIKO 1 and Kung-fu data are approximately 5,000. The specification of the PC that we utilized for the evaluation is CPU: Core2Duo 2.4 GHz, Memory: 4 GB, GPU: GeForce 8800 GTX and VRAM: 750 MB, and the software is implemented in C# and Managed DirectX with Pixel shader 2.0. Figure 5.26 shows rendered images of the Kung-fu stream. The harmonized texture generation method can render high quality images in real time (≥30 fps) using a 15-viewpoint video stream, which

192

5


Fig. 5.26 Rendered 3D video stream of Kung-fu. ©2009 ITE [21] Table 5.2 Computational time for rendering one frame (msec). ©2009 ITE [21]

MAIKO 1

Kung-fu

Weighting factors (Eq. (5.8))

1), and then iteratively deriving the graphs at lower resolution until r = 1 (cf. Fig. 8.5). At resolution r = 0, the graph consists of one unique root node. A Reeb graph at resolution level R is constructed by: 1. Partitioning the range of μN , i.e. [0, 1], into 2R regular intervals by iterative subdivisions, and assigning interval labels to surface points (i.e. mesh vertices) according to their μN values. 2. Creating a graph node for each surface region consisting of mutually connected surface points with the same interval label. 3. Linking the nodes that have their corresponding regions connected on the surface. In practice, when the surface is represented by a triangular mesh, at each resolution each node corresponds to a set of connected triangles (and is placed at its centroid). The nodes created in the step 2 above stand for the Reeb graph nodes, and the links created in the step 3 stand for the Reeb graph edges. Reeb graphs at lower resolutions r < R are obtained by first merging the intervals of μN values two by two using a hierarchical procedure. Then, a parent node, a node in the lower resolution Reeb graph, is assigned to each group of nodes in the higher resolution graph whose corresponding surface regions are connected and share the same merged μN value interval label. Hence, each node at resolution r > 1 has a unique parent node belonging to a Reeb graph at resolution r − 1 [8]. Note that the object surface is partitioned into regions with 2r interval labels at the resolution level r.

8.3 Topology Description Using Reeb Graph

267

As defined the multi-resolution Reeb graph captures the topological structure of 3D surface models at different levels of resolution. To obtain finer shape description of 3D models, Reeb graphs are enriched with geometrical and topological features. They are then called augmented Multi-Resolution Reeb Graph (aMRG). aMRG was initially introduced in [35] to perform fine shape matching and retrieval in dataset of 3D art objects. This approach was first applied to 3D video data stream in [36] for 3D video compression, and later in [32] for 3D video description while introducing the topology dictionary. In an aMRG, each node embeds specific attributes that characterize the surface region it is assigned to (as described above). 3D model shape similarity can then be estimated by comparing the following node attributes. • Relative area: area of the region with respect to the overall model surface, • μ range: min and max values of μ in the region the node is assigned to, • local curvature statistic: histogram of Koenderink shape indices computed at each point of the surface region [15], • cord length statistic: histogram of Euclidean distances measured between each point of the region to the region center of mass, namely cord lengths [22], • cord angle statistics: histogram of cord angles with respect to the principal 1st and 2nd axes of the region [22], • hough 3D descriptor: 3D histogram of normal orientations (azimuth θ ∈ [−π, π] and elevation φ ∈ [0, π]) and distances to node center (r ≥ 0) computed at each point of the surface region [40]. Note that some attributes are invariant to rotation, translation, and scale transformations, whereas some attributes are computed with respect to principal axes (e.g. cord angle statistics). The axes can be either the world coordinate axes if the object is already oriented, or obtained by PCA [25]. Details concerning aMRG matching are given in Sect. 8.4.

8.3.3 Robustness As 3D models in a 3D video data stream usually contain reconstruction artifacts, the Reeb graph extraction has to be particularly robust to surface noise. Fortunately, the normalized Morse function introduced in Eqs. (8.8) and (8.9) is robust to local surface noise thanks to the integral formulation (as well as being invariant to rotation, translation and scale transformation). To evaluate the stability of the Reeb graph regarding surface noise, we tested the Reeb graph extraction on 3D models of different resolution (high and low). We observed that extra nodes might appear occasionally, especially at the extremities of the graphs. These are due to surface sampling implementation artifacts, as geodesic distances between vertices of a surface mesh are computed using the mesh edges. Hence geodesic distances on two meshes having different connectivity may differ. However, as defined Eq. (8.8) can usually cope with all kind of surface noise (including mesh connectivity change) thanks to the integral formulation which smoothes local variations.

268

8

Behavior Unit Model for Content-Based Representation and Edition

Fig. 8.6 Reeb graph robustness against surface noise incurred by the mesh simplification. Reeb graphs are constructed for 3D models of different mesh resolution (high and low): (a) 40th frame from the Tony sequence with 17,701 vertices and 1,335 vertices, (b) 25th frame from the Free sequence of [31] with 142,382 vertices and 6,500 vertices. The resolution of the Reeb graphs is r = 3. Despite some extra nodes at some extremities of the graphs, their overall structure and topology are well preserved. ©2012 IEEE [34]

Figure 8.6 shows (a) the 40th frame from the Tony sequence, captured and reconstructed in our laboratory, with 17,701 vertices and simplified to 1,335 vertices, and (b) the 25th frame from the Free sequence of [31] with 142,382 vertices and simplified to 6,500 vertices. The Reeb graphs are shown at the resolution r = 3. The simplifications were performed by edge collapsing in order to affect the geodesic measurements as much as possible. Despite some extra nodes at some extremities of the Reeb graphs, we can observe that their overall structure and topology are preserved.

8.3.4 Advantage Structure extraction from arbitrary shape is usually performed by fitting a 3D skeleton to the shape surface model, such as in [3]. When successful, this kind of approach is powerful because the kinematic structure of the object can be extracted, and the structure joints can be tracked while the object is in motion. This topic will be discussed in detail in the next chapter. However, fitting a skeleton requires prior knowledge on the shape to be described: the skeleton has to be defined beforehand and cannot be fitted to any arbitrary shapes [12]. On the other hand, the Reeb graph overcomes these limitations as it can characterize topology and shape of arbitrary 3D models. No a priori knowledge on the model shape and topology is required, and no initial pose is required.

8.3 Topology Description Using Reeb Graph

269

Fig. 8.7 Comparison of structure extraction techniques. The Reeb graph (in green) has several advantages compared to a skeleton fitting approach (in red) [3]. It extracts a consistent structure from arbitrary shapes without any prior knowledge, even though (a) limbs are not visible, and regardless of the model (b) topology, (c) orientation, and (d) complexity. ©2012 IEEE [34]

Figure 8.7 illustrates the advantage of using an automatic topological structure extraction method such as the Reeb graph, as opposed to a skeleton fitting technique (such as [3]). As can be observed, the Reeb graph can extract a consistent structure from arbitrary shapes without any prior knowledge, even though: (a) limbs are not visible, and regardless of the model (b) topology, (c) orientation, and (d) complexity.1 Note that other approaches, such as the curve-skeleton [2, 4, 17], can be used to extract a graph with homotopy property. Nevertheless, as shown in the next section, our approach is the most suitable for shape matching as it features a hierarchical multi-resolution structure. Otherwise graph matching computation in a huge dataset can quickly become intractable.

1 Raptor

model provided courtesy of INRIA by the AIM@SHAPE Shape Repository.

270

8


8.4 Behavior Unit Model This section presents the techniques we developed for behavior unit-based 3D video data representation, i.e. a practical implementation of the topology dictionary introduced in Sect. 8.2. A behavior unit is a novel data representation method for contentbased representation and edition of 3D video. The behavior unit model represents a set of atomic actions. It is defined as a partial interval of a 3D video data stream in which an object performs a simple action such as stand up, sit down, hands up, rotate, and so on, while preserving its overall topological structure. The partition of a 3D video stream results from the extraction and clustering of aMRG graph feature vectors, and the encoding is achieved by indexing the feature vectors using a topology dictionary model (cf. Sect. 8.2). In what follows we present: feature vector representation as an abstraction of aMRG graph, feature vector similarity computation, retrieval performance in 3D video, and behavior unitbased encoding and decoding of 3D video data stream. Note that the behavior unit model is defined based on atomic actions which are represented by a continuous sequence of cluster (or pattern) indices, while a cluster is a set of feature vectors. As presented in Sect. 8.2.1, the clustering constrains the selected feature vectors to be similar in object topology as well as in object motion. The clustering is performed on a distance matrix whose elements are obtained using an integration over time using a time window (cf. Eq. (8.3)). Past, present and future 3D video frames of an atomic action are taken into account when performing similarity computation. Hence clusters are bound to contain ‘continuous’ feature vectors and suits well to represent behavior units.

8.4.1 Feature Vector Representation As introduced in Sect. 8.3, aMRG graphs capture topology and shape of 3D models. The aMRG representation consists of a set of Reeb graphs at several levels of resolution that embed geometrical and topological attributes in each node. aMRG feature vector representation consists of storing graph structures and node attributes in a unique (binary) file. Attributes of each node are stored as tables of floats, and multi-resolution graph structures are stored by indexing all edges as in a standard mesh coding method (e.g. OFF format). Table 8.1 shows the data format for aMRG. The size of a feature vector is then 2 + 4 ∗ 2 + res + i nbNodes[i] ∗ nbAttr + nbEdges ∗ 2 bytes. Note that the data structure above considers the simple case of one attribute per node encoded by one float. In practice, each node embeds the attributes presented in Sect. 8.3. Table 8.2 shows the data format for an aMRG node. In our implementation the attributes of each node are encoded in 784 bytes. The nature of each attribute, as well as the parameters (e.g. number of histogram bins) are set using heuristics [33].

8.4 Behavior Unit Model

271

Table 8.1 Data format for aMRG Size in bytes

ID

Description

sizeof(char) = 1

res

aMRG highest resolution level R

sizeof(char) = 1

nbAttr

Number of attributes in each node

(res + 1) ∗ sizeof(int) = (res + 1) ∗ 4

nbNodes[] List containing the number of graph nodes at each resolution level

sizeof(int) = 4 i nbNodes[i] ∗ nbAttr ∗ sizeof(float) = i nbNodes[i] ∗ nbAttr ∗ 4

nbEdges

2 ∗ nbEdges ∗ sizeof(int) = 2 ∗ nbEdges ∗ 4

Total number of graph edges All graph node attributes All graph edges (pairs of node indices)

Table 8.2 Data format for aMRG node Size in bytes

ID

Description

sizeof(float) = 4

a

Relative area

sizeof(float) = 4

l

Length of μN interval

sizeof(int) = 4

i

Index of μN interval (i ∈ [0, 2r − 1])

16*sizeof(float) = 64

curv[]

Histogram of Koenderink shape indices


cordL[]

Histogram of cord lengths


cord1[]

Histogram of cord angles w.r.t. 1st axis


cord2[]

Histogram of cord angles w.r.t. 2nd axis

4*8*4*sizeof(float) = 512

hough[]

Hough 3D descriptor

At resolution level r, the interval in which μN belongs to (i.e. [0, 1]) is partitioned into 2r intervals. Each interval carries an interval index i ∈ [0, 2r − 1], where i = 0 correspond to the regions having the minimal values of μn . The length of an interval of μN is defined as the difference between the maximum and minimum values of μN in the interval. Once the number and size of attributes per node are fixed, the file size will vary linearly with respect to the number graph nodes. The size of a feature vector can therefore be quite large, especially when dealing with high resolution aMRG graphs, and similarity computation can become quickly intractable in huge dataset as graph matching complexity is NP-complete. However, the coarse-to-fine hierarchical matching strategy described in the next sections is a solution to overcome this issue.

8.4.2 Feature Vector Similarity Computation Similarity Evaluation Assuming aMRG graphs M and N extracted from two 3D models, we denote their corresponding feature vectors equally by M and N ,

272

8


respectively, as they are abstractions of the same objects. Feature vector similarity computation, as introduced in Sect. 8.2.1, consists of calculating Eq. (8.3) and the SIM function described below: 1 1+R R

SIM(M, N ) =

sim(m, n),

(8.10)

r=0 {(m,n)∈Cr }

where Cr ⊂ M ×N contains all the pairs of topologically consistent nodes at the resolution level r ∈ [0, R], whose definition will be given below, and sim : M × N → [0, 1] evaluates the similarity between two nodes m and n. sim returns a higher contribution when nodes are similar: sim(m, n) =

nbAttr-1

λk δk fk (m), fk (n) ,

(8.11)

k=0

where nbAttr is the number of attributes embedded in each node, λk (with λ = 1) is a weighting factor for the attribute fk , and δk is a function that k k compares the attributes fk depending on the types of attribute listed in Table 8.2: a(m) a(n) • if fk ≡ a, then δk (fk (m), fk (n)) = min( a(M) , a(N ) ), where a(m) and a(n) are the areas of the surface regions associated with m and n, respectively, and a(M) and a(N) are the total areas of the surfaces associated with M and N , respectively. Note that a(m) and a(n) rely on the resolution level r, whereas a(M) and a(N ) are independent of r. l(m) l(n) • if fk ≡ l, then δk (fk (m), fk (n)) = min( l(M) , l(N ) ), where l(m) and l(n) are the lengths of the intervals of μN defined on the surface regions associated with m and n, respectively, and l(M) and l(N) are the sum of the lengths of μN intervals associated with all the nodes of M and N , respectively. Note that l(m) and l(n) rely on the resolution level r, whereas l(M) and l(N ) are independent of r. • if fk ≡ curv, cordL, cord1, cord2, hough or more generally a histogram, then δk is the histogram intersection:

B−1 min fk (m)[i], fk (n)[i] , δk fk (m), fk (n) =

(8.12)

i=0

where B is the number of histogram bins and fk is a normalized histogram satisfying the following equation: ∀m ∈ M,

B−1 i=0

fk (m)[i] =

a(m) . a(M)

(8.13)

Thus, if M = N then SIM(M, M) = 1. SIM computes the similarity between two feature vectors by summation of the similarity scores obtained for each pair of matching nodes by sim at every level of resolution from r = 0 to R (the matching process is described below). Each similarity evaluation of a pair of nodes by


273

sim returns a (positive) contribution to the global similarity score given by SIM. As defined above, δk returns bigger contributions when nodes are similar. The descriptors a and l serve to characterize the global shape of the object. On the other hand, the local descriptors (namely, here, the histograms) serve to characterize details and variations on the surface of the objects. These descriptors were first introduced for shape matching and retrieval in database of 3D art objects in [35]. Note that sim in Eq. (8.11) is positive, reflexive, symmetric and transitive. However, SIM in Eq. (8.10) is positive, reflexive, symmetric but not transitive as M and N may not have the same node structure. In our implementation the weights {λk } are determined heuristically: λa = 0.2, λl = 0.3, λcurv = 0.1, λcordL = 0.1, λcord1 = 0.1, λcord2 = 0.1, and λhough = 0.1 and correspond to selected attributes {fk } as described above. Topology Matching Topology matching is the process that consists of matching two aMRG graphs M and N based on their topological structure. The algorithm described below returns all the pairs of topologically consistent nodes between M and N . By definition [8, 35], two nodes m and n at resolution r, belonging to M and N , respectively, are topologically consistent if: 1. The parents m and n of m and n respectively have been matched together at the level of resolution r − 1. 2. m and n have an equal interval index of μN (as defined in Table 8.2). 3. If m and n belong to a graph branch,2 they must have the same label (if they have any) to be matched: when two nodes are matched, a label (e.g. α) is identically assigned to both of them if they do no have one yet. Then, the label is propagated to their connected neighbors3 in both graphs M and N following the two monotonic directions with increasing and decreasing values of μN . Note that label propagation is performed only for branches, and not for branching nodes. 4. The parents of the neighbors of m and n, if they have ones, have been matched together at the level of resolution r − 1. In addition, the node matching procedure allows the matching between a node m and a set of nodes {n} when m is topologically consistent to all nodes of {n}. This alleviates some possible boundary issues (especially located at branch junctions) after segmentation of the object surface into regions caused by discretized values of μN (as μ is computed on the mesh vertices and using the mesh edges). Furthermore, a cost function loss is introduced to discriminate nodes or set of nodes that are all topologically consistent. loss is minimal when two candidates are similar and is used to find the best topologically consistent candidate. The function is defined in the spirit of Eq. (8.11), but involves only the node global descriptors a 2A

graph branch is a set of successive nodes linked two by two by a single edge. Two branches match together when all the nodes belonging to them match together. 3 A neighbor is a node belonging to an adjacent surface region. Neighboring nodes are connected by a Reeb graph edge at the same resolution level.

274

and l:

8


l(m)

a(m) a(n)

l(n)

+ − − loss(m, n) = (1 − )

l(M) l(N ) , a(M) a(N )

(8.14)

where = 0.5. When evaluating loss and sim (cf. Eq. (8.11)) for a set of nodes, the features representing a set can be obtained by a simple addition of the attributes embedded in each node of the set. In practice the calculation of loss includes as well the neighboring nodes in order to increase the discrimination power. The matching process involves a coarse-to-fine strategy where topology consistencies between nodes of M and N are evaluated hierarchically from the root node at resolution r = 0 to the nodes at the finest level of resolution r = R. At each resolution level r, the nodes are sorted following their embedded μN values in an ascending order and visited one by one. If two nodes m and n at resolution level r and belonging to M and N , respectively, are found topologically consistent, then the matching pair (m, n) is inserted in a list Cr , and the matching process is repeated with their children nodes at resolution level r + 1. On the other hand, if two nodes m and n are not topologically consistent, then the matching process is aborted for these two nodes. The topology matching process terminates when all the pairs of topology consistent nodes are found, meaning that no matching process is running anymore. The set of pairs of consistent nodes {C0 , . . . , CR } is then returned and used for similarity computation as described in Eq. (8.10). Figure 8.8 illustrates topology matching for multi-resolution Reeb graph. Matching algorithm implementation details are as follows: 1. Let M and N denote two aMRG graphs to be compared. At the lowest resolution level r = 0, M and N are represented by one single node which are matched by default. 2. For each resolution level r > 0, the nodes {m ∈ M} are visited one by one following the interval index of μN they belong to, from 0 to 2r−1 . 3. For each visited node m ∈ M, all the nodes {n ∈ N } at the same resolution r are taken into consideration regarding the topology consistency rules defined above. The candidates (node n or group of nodes {n}) that are topologically consistent to m are further discriminated using the cost function loss (cf. Eq. (8.14)). 4. The best candidate that returns the smallest cost with loss is paired with m and inserted in a list Cr ⊂ M × N . 5. As all nodes {m ∈ M} at resolution level r are visited, Cr should contain all topologically consistent pairs (m, n) between M and N at resolution r. 6. Topology matching (i.e. steps 2 to 5) is repeated at r +1 with all children nodes of nodes that have been matched at r, and iteratively until the highest resolution r = R is reached. In Fig. 8.8, at resolution r, several nodes have similar topology. The loss function is then evaluated between all the candidates to find the best matches. Nodes that are matched at resolution r are represented with identical colors. In addition, blue links with arrows show examples of matching between nodes at resolution r (m and n ), as well as between their children nodes (m and n) and group of nodes at resolution level r + 1. Parent to child links are represented by yellow dashed arrows. Note that m and n belong to branches of M and N ,


275

Fig. 8.8 Topology matching of multi-resolution Reeb graph. Nodes that are matched at resolution r are represented with identical colors. In addition, blue links with arrows show examples of matching between nodes at resolution r (m and n ), as well as between their children nodes (m and n) and group of nodes at resolution level r + 1. Parent to child links are represented by yellow dashed arrows. Note that m and n belong to branches of M and N , respectively. A label (here α) is propagated in both branches as the pair (m, n) is formed. Consequently, the corresponding branches of M and N with label α are matched together

respectively. A label (here α) is propagated in both branches as the pair (m, n) is formed. Consequently, the corresponding branches of M and N with label α are matched together. Note that, by definition of a branch, the label is not propagated to branching nodes. 7. Finally, aMRG similarity score SIM(M, N ) between M and N is obtained by adding node similarity values of all topology consistent pairs {(m, n) ∈ Cr } at every level of resolution r = 0 . . . R using the sim function which takes into account embedded node attributes (cf. Eqs. (8.10) and (8.11)). Multi-resolution Strategy The coarse-to-fine multi-resolution matching strategy has two major advantages. First, it is crucial for computation tractability when dealing with large databases. It avoids indeed the NP-complete complexity issue of large graph matching by aborting the similarity evaluation process, as soon as no consistent match is found between the graphs, starting from the lowest graph resolution (which have only few nodes). Second, it provides a judicious matching scheme as

276

8


global shape and topology (e.g. body postures) are privileged over fine details (e.g. arm positions, fingers). The nodes are matched hierarchically using the topology consistency rules and similarity functions (cf. sim function described above). The coarse-to-fine recursion is spread through node children (starting from the root at r = 0) up to the highest resolution (r = R). Note that irrelevant nodes are discarded in the graph matching process as they would return weak matching scores (e.g. nodes resulting from surface noise due to 3D reconstruction artifact). Further reading concerning the matching process can be found in [8, 35].

8.4.3 Performance Evaluation The performance of the 3D shape similarity computation method using aMRG presented above is evaluated against various shape similarity metrics for 3D video sequences of people with unknown temporal correspondence [9]. Performances of similarity measures are compared by evaluating Receiver Operator Characteristics (ROC) for classification against ground truth of a comprehensive dataset of synthetic 3D video sequences consisting of animations of several people performing different motions (Fig. 8.9). The synthetic dataset is created using 14 articulated character models, each of which animated using 28 motion capture sequences. Animated models are people with different gender, body shape and clothing, and height between 1.6 m and 1.9 m. Models are reconstructed using multiple view images, as a single connected surface mesh with 1 K vertices and 2 K triangles. Recognition performances are evaluated using ROC curve, showing the truepositive (TPR) or sensitivity in correctly defining similarity against the false-positive rate (FPR) or one-specificity where similarity is incorrect: TPR =

ts ts + td

and

FPR =

fs , fs + td

(8.15)

where ts denotes the number of true-similarity predictions, fs the false similar, td the true dissimilar and fd the false dissimilar in comparing the predicted similarity between two frames to the ground-truth similarity. For each similarity measure, dissimilarity values between all pairs of 3D shape data in the database are computed in the distance matrix S (cf. Sect. 8.2.1), whose element sij denotes the distance (or dissimilarity) between ith and j th 3D shape data. Distances are normalized to the range sij ∈ [0, 1]: sij =

sij − smin , smax − smin

(8.16)

where smin = 0 and smax is the maximal distance over all sij ∈ S of the whole database. A binary classification matrix for the shape descriptor C(α) = {cij (α)}ij is defined: 1 if sij < α cij (α) = (8.17) 0 otherwise


277

Fig. 8.9 Synthetic dataset. aMRG robustness and accuracy is evaluated using ground-truth data. (a) Illustration of 14 human models. (b) Six of the 28 motions

Fig. 8.10 ROC curves for the shape classification of 3D video sequences. The aMRG is one of the top performers for shape classification in 3D video. The comparison includes Shape Histograms (SHvr), Multi-Dimension Scaling (MDS), Spin Image (SI), Shape Distribution (SD) and Spherical Harmonics Representation (SHR). ©2012 IEEE [34]

The classification cij (α) for a given α is then compared to a ground-truth similarity classification. The number of true and false similarity classifications, ts(α), td(α), fs(α), fd(α) are then counted. The ROC performance for a given shape similarity measure is then obtained by varying the threshold α ∈ [0, 1] to obtain the true TPR(α) and false FPR(α) positive rates according to Eq. (8.15). The evaluations with the generated 3D video sequences demonstrate that the aMRG-based method is one of the top performers in the task of finding similar poses of the same person in 3D video compared to the state-of-the-art shape matching techniques (Fig. 8.10 and [10]). The comparison includes Shape Histograms (SHvr), Multi-Dimension Scaling (MDS), Spin Image (SI), Shape Distribution (SD) and Spherical Harmonics Representation (SHR) (see [9] for additional details). As the aMRG-based method is particularly sensitive to topology changes compared to other approaches, it is our first choice as a topology-based shape descriptor.

278

8


8.4.4 Data Stream Encoding Here we present a linear process to compactly encode a 3D video data stream based on behavior units. As described in the previous sections, the behavior units are obtained by first clustering feature vectors characterizing topological structures, indexing clusters within a dictionary, and then partitioning the original data stream into a sequence of temporal intervals using the cluster indices. This method is a practical implementation of the topology dictionary (Fig. 8.1). Assuming the data stream S containing T frames, S = {s1 , . . . , sT }, a data structure representing a behavior unit ki is created for every of the N clusters ci ⊂ S , where 1 ≤ i ≤ N . Each ki encompasses a specific pose of the object and j j all its variations within each temporal interval ci that forms ci = j {ci }. j

j

Let simin and simax denote the frames having, respectively, the smallest and j biggest indices in the j th interval ci ⊂ ci . ki contains one textured mesh (i.e. one mesh and one texture map), one graph structure (namely one Reeb graph at a chosen resolution), a table of node position offsets corresponding to all the node trajectories j to transit from any simin to ki , and a table of node position offsets corresponding to j all the node trajectories to transit from ki to any simax . The node trajectories are j obtained by tracking the node positions in each ci , which is a trivial task as long as the graphs are consistent to each other (as it should be in each cluster). In practice, noisy nodes and edges are removed so that any Reeb graph constructed from a frame in ci is topologically consistent to ki . Practically, ki should be chosen at the center of the cluster ci : ki = arg min SIM(Mk , M), (8.18) Mk ∈ci

M∈ci

where Mk and M are meshes in ci , and SIM is the similarity function which computes aMRG feature vector similarity as described in Sect. 8.4.2 (Eq. (8.10)). The encoding process consists of sequentially substituting each frame in a 3D video data stream by a cluster (or pattern) reference index. If a frame st does not belong to the same cluster ci−1 as the previous frame st−1 , then a new data structure representing a behavior unit ki is created. If st belongs to the same cluster ci−1 as st−1 , then only position offsets of the graph nodes at t are stored into ki−1 . It is then possible to recover the node trajectories between consecutive frames and reconstruct the mesh sequence by the skinning operation, which will be presented in the next section. Let sm be the size of an encoded mesh plus a Reeb graph structure, and sg be the size of an encoded set of node position offsets, the total data size σ of the encoded sequence is then σ ≤ sm ∗ N + sg ∗ T .

8.4.5 Data Stream Decoding Model animation can be performed using several sophisticated techniques and CG software, where the model deformation process is usually guided by a skeleton


279

Fig. 8.11 Sequence reconstruction. 3D video sequences are reconstructed from encoded data stream by mesh skinning. (a) Textured mesh from initial 3D video data. (b) Reeb graphs extracted at resolution r = 4. (c) Reconstructed surfaces by mesh skinning. (d) Reconstructed surfaces with texture to be compared with (a). Here, one unique texture map (view-independent vertex-based texture) is used for the whole reconstructed sequence. (e) The overlay of both surfaces (initial surface in blue and reconstructed surface in red) shows that the overall surface shape is well reconstructed. Note that this 3D video was produced from multi-view video captured in our oldest studio rather than Studios A, B, or C in Chap. 2. ©2012 IEEE [34]

(e.g. Blender, Maya [3, 14]). First, a skeleton of the model is created interactively and subparts of the model are attached to bones (rigging process). Then, skeleton joints are manipulated and deformations are cast sequentially to the mesh (animation

280

8


process). Additional post-processing steps are usually applied to smooth discontinuities between submesh boundaries (skinning process), as well as mesh edition techniques to improve surface rendering [23, 30]. Skinning is a popular method for performing character and object deformation in 3D games and animation movies. In our framework, 3D video reconstruction from encoded frames is obtained using a mesh skinning method where surface deformations are driven by Reeb graphs. During the skinning process, the graph is bound to a single mesh object, and the mesh is deformed as the graph nodes move. Recall that the data structure representing a behavior unit ki contains the motion trajectory data of each node in the graph. As node coordinates change, transformation matrices associated with the vertices of the mesh cause them to be deformed in a weighted manner. A weight defines how much a specific node influences vertices in the deformation process (e.g. 1.0 for rigid skinning, and less than 1.0 for smooth skinning). It is usual to set smoother skinning for vertices belonging to a joint area on a surface mesh [11]. The data stream reconstruction is performed for each cluster ci by considering j j each temporal interval ci independently, where ci = j {ci }, in order to avoid surface topology change issues. A unique data structure ki , as introduced in the previous section, describes behavior units represented by the cluster ci : all of the 3D video frames whose corresponding feature vectors belong to ci are reconstructed j by deforming ki according to encoded node coordinates (for each ci ) using a mesh skinning method as described above. Figure 8.11 illustrates a sample of 3D video data reconstructed from an encoded data stream. Note that even though the implemented mesh skinning method is not optimal, no major reconstruction artifacts would be noticed at video frame rate (25 fps). We measured 3D position distortions in a 400 × 400 × 400 voxel grid (corresponding to a 2 m × 2 m × 2 m volume having 5 mm resolution), and we obtained the mean squared error MSE ∼ 0.005 and the peak signal-to-noise ratio PSNR ∼ 75 dB when computing surface distances between original and decoded mesh data using Hausdorff distance as metric.

8.5 Applications This section presents several applications of the behavior unit model, in particular, for 3D video data editing and semantic description. As described in the previous sections, it provides an abstraction of the structure of data stream. Hence long poses, repeated actions, and slow motions can be identified in video sequences of human in motion, and compactly encoded and manipulated.

8.5.1 Behavior Unit Edition Let us assume G = (C, E) is a weighted directed graph, where C are the vertices and E are the edges. C = {c1 , . . . , cN } denotes the set of N clusters obtained by

8.5 Applications

281

topology-based clustering of T frames of the 3D video sequence S = {s1 , . . . , sT }. As presented in Sect. 8.2.2 and Sect. 8.4, G is a motion graph representing the structure of the data stream S by its states and transitions in a probabilistic framework, and C is a set of identified behavior units. Behavior unit-based edition is performed by interacting with the motion graph G. In particular, estimations of path probability with Eq. (8.6) allow users to create new sequences of actions while preserving scenario realism. Our educated guess suggests that new sequences can be created by picking two behavior units cs and ct in C, and concatenating all the behavior units that correspond to the clusters belonging to the (most) probable paths linking cs and ct . To achieve high quality rendering, transitions between behavior units should be particularly well managed, as the transition between two surfaces with very different topology can be challenging [39]. Although this was not fully investigated yet, we believe that implementing an ad-hoc surface skinning method should pay off. In what follows we present an unsupervised scheme to skim 3D video sequences by processing behavior units of human performances. The goal is to automatically produce shorter sequences while preserving scenario consistency. The motion graph is used to identify isolated and non-relevant patterns and progressively remove them. First, the set C is sorted with respect to the cluster weights P (ci ) in order to identify the frames belonging to clusters having the highest and lowest probabilities: • If P (ci ) 0, then ci contains either: (1) a long sequence of successive frames belonging to ci , or (2) a recurrent pose identified by frames belonging to ci scattered in the sequence. In the case (1), long poses (e.g. low variations such as between frames #370 and #430 in Fig. 8.2) are compressed by encoding intermediate 3D video frames as described in the previous section. In the case (2), ci is represented as a cycle (or loop) junction node in the motion graph G. The strategy is therefore to gradually remove the small and non-relevant cycles. Let L denote a cycle in the motion graph. Then compute: P (c) , (8.19) S(L ) = c∈L card{c ∈ L } {eij ∈L } P (cj |ci )P (ci ) P (L ) = , (8.20) card{eij ∈ L } where the size S(L ) is the average weight of the cycle L , and the relevance P (L ) is defined as the probability of the cycle L under Markov assumption. The following weight is used to sort the cycles: W (L ) = λ.S(L ) + (1 − λ).P (L ),

(8.21)

where λ = 0.5 ∈ [0, 1]. Practically, the skimming process consists of removing redundant frames from video sequences where small cycles with low weight values are selected as first candidates for skimming. As will be discussed in Sect. 8.6, several other skimming strategies can be adopted.

282

8


Fig. 8.12 Behavior units with semantic labels. Arbitrary poses with annotations can be learned and indexed in the topology dictionary. ©2012 IEEE [34]

• If P (ci ) ∼ 0 then ci contains few frames. Identified isolated patterns are reclassified into adjacent clusters: e.g. in the sequence {ci , ci , cj , ci , ci }, the frames {s ∈ cj } are reclassified into ci . In practice, graph structures are modified (by node filtering) to be consistent with ci and avoid reconstruction artifacts. Finally, summarization can be processed iteratively up to some user-defined constraints such as a limitation on the sequence size or compression ratio (cf. Sect. 8.6).

8.5.2 Semantic Description Semantic description of data stream is obtained by specifying semantic labels for identified behavior units. The semantic labels are obtained by first analyzing training datasets to prepare a set of prototype behavior units and then giving to each behavior unit a semantic label such as “stand up, hands on hips”, “stand up, hands joined over the head, head looking the hands”, etc. In practice, any behavior unit with a semantic label can be added as learning and indexing can be performed on any arbitrary data. For example as shown in Fig. 8.12, models from various sources can be annotated for action recognition application. With this training process, a semantically labeled topology dictionary is constructed. Then, given a new 3D video stream, each video frame is compared with the behavior units with semantic labels and is classified to the most similar behavior unit to convert the 3D video stream into a sequence of behavior unit indices and/or semantic labels.


283

In addition, as can be observed in Fig. 8.12, labels can be specified based on shape categorization in addition to topology. The Homer, woman and alien models have similar topology structure as they all stand up, but different shape features as the limbs have different lengths, and body build is different. The variations are captured by the graph node attributes defined in Table 8.2. Thus, the topology dictionary can perform classification and description based on shape as well as topology. Furthermore, as the behavior units we are considering are based only on shape and topology, there is no knowledge about content importance. However, a semantic weight describing the content importance can be added along with labels (or annotations) to behavior units obtained from training datasets. Hence, behavior units with lower importance weight could be removed first in the skimming process described in Sect. 8.5.1.

8.6 Performance Evaluations To assess the performance of the behavior unit model, several experiments were performed on various 3D video sequences. The Yoga sequence in Fig. 8.2 and Tony sequence in Fig. 8.5 are interesting as they contain many human poses. These sequences are useful to set the parameter τ for sequences of humanoid objects. The MAIKO dataset is challenging for shape description as the subject wears a loose FURISODE which covers arms and legs. Fortunately the Reeb graph is an effective tool to characterize arbitrary shapes. The Capoeira sequence represents quick moves of martial art. In these experiments, MAIKO, Yoga, Capoeira and Tony sequences contain, respectively, 201, 7500, 300 and 250 frames from the original sequences (cf. Fig. 2.13). Every frame contains one 3D mesh consisting of about 30 K triangles with texture information. One uncompressed frame encoded in standard OFF format requires 1.5 MB, which means 11.25 GB for 7500 frames. Feature vectors were computed on a Core2Duo 3.0 GHz 4 GB RAM, nevertheless the process requires less than 512 MB RAM. A feature vector up to resolution level R = 5 is generated in 15 s with the current implementation (cf. [33] for binaries). The similarity computation between two aMRG models takes 10 ms. Other efficient computation of Reeb graphs can be found in the literature (e.g. [24]). All the independent steps can be run sequentially using a script file.

8.6.1 Topology Dictionary Stability The core of the topology dictionary model relies on its ability to discriminate shape topology. The definition of the Morse function is therefore crucial (cf. Sect. 8.3). The ability of the dictionary to extract and classify patterns has been evaluated against different Morse functions and resolution levels R of Reeb graphs: the curves in

284

8


Fig. 8.13 Clustering of 500 frames of Yoga with respect to τ . Curves are obtained with different Morse function (geodesic integral defined in Eq. (8.8) and height function), and at different levels of resolution. ©2012 IEEE [34]

Fig. 8.13 named geodesic R = 5, geodesic R = 4, and geodesic R = 3 were obtained when using the geodesic integral as Morse function as defined in Eq. (8.8), and by computing similarities up to the resolution levels R = 5, R = 4, and R = 3, respectively. The curve named geodesic r = 4 was obtained with the geodesic integral as Morse function, but without summation of coarse resolution levels when computing the similarity (i.e. only the contributions at level r = 4 were used in Eq. (8.10)). The curve named height R = 4 was obtained when using the height function μ(v) = z as Morse function, and by computing similarities up to the resolution level R = 4. The clustering performance is then evaluated with respect to the threshold τ . In Figs. 8.13 and 8.14, the sequences contain, respectively, 500 and 7500 frames of a Yoga session. They consist of a succession of various (complex) poses. The clustering behavior was analyzed with different values of τ and parameter setting. Finally, it turned out that the integral geodesic functions with R = 4 and τ = 0.1, and R = 3 and τ = 0.08, give the best trade-offs for clustering performance and computation time for humanoid model sequences in comparison to a hand-made clustering. The full Yoga sequence (7500 frames) contains 1749 clusters: 44 long poses or repeated actions (>1 s), 115 short actions (40 ms to 1 s) and 1590 transition states (

3D Video and Its Applications

Video Segmentation and Its Applications

Video Segmentation and Its Applications

Video Segmentation and Its Applications

3D and HD Broadband Video Networking

Photoshop CS3 Extended Video and 3D Bible

Photoshop CS3 Extended Video and 3D Bible

3D and HD Broadband Video Networking

Interactive Video: Methods and Applications

3D Imaging, Analysis and Applications

Geometry and its applications

Geometry and its Applications

Functional Analysis and its Applications

Fourier Analysis and Its Applications

Discrete Mathematics and Its Applications

Hopf bifurcation and its applications

Superanalysis (Mathematics and Its Applications)

Credit scoring and its applications

Local cohomology and its applications

Linear programming and its applications

Net Theory and Its Applications

Linear algebra and its applications

Linear algebra and its applications

Fourier analysis and its applications

Set Theory and its Applications

Credit scoring and its applications

Weak Convergence and Its Applications

Discrete Mathematics and its Applications

Fourier Analysis and Its Applications

3D Video and Its Applications

Video Segmentation and Its Applications

Video Segmentation and Its Applications

Video Segmentation and Its Applications

3D and HD Broadband Video Networking

Photoshop CS3 Extended Video and 3D Bible

Photoshop CS3 Extended Video and 3D Bible

3D and HD Broadband Video Networking

Interactive Video: Methods and Applications

3D Imaging, Analysis and Applications

Geometry and its applications

Geometry and its Applications

Functional Analysis and its Applications

Fourier Analysis and Its Applications

Discrete Mathematics and Its Applications

Hopf bifurcation and its applications

Superanalysis (Mathematics and Its Applications)

Credit scoring and its applications

Local cohomology and its applications

Linear programming and its applications

Net Theory and Its Applications

Linear algebra and its applications

Linear algebra and its applications

Fourier analysis and its applications

Set Theory and its Applications

Credit scoring and its applications

Weak Convergence and Its Applications

Discrete Mathematics and its Applications

Fourier Analysis and Its Applications

Recommend Documents