Advances in Artificial Reality and Tele-Existence: 16th International Conference on Artificial Reality and Telexistence, ICAT 2006, Hangzhou, China, November

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris ...

Author: Ronghua Liang | Zhigeng Pan | Adrian Cheok | Michael Haller | Rynson W.H. Lau | Hideo Saito

10 downloads 1224 Views 65MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Moshe Y. Vardi Rice University, Houston, TX, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

4282

Zhigeng Pan Adrian Cheok Michael Haller Rynson W.H. Lau Hideo Saito Ronghua Liang (Eds.)

Advances in Artificial Reality and Tele-Existence 16th International Conference on Artificial Reality and Telexistence, ICAT 2006 Hangzhou, China, November 29 – December 1, 2006 Proceedings

13

Volume Editors Zhigeng Pan Zhejiang University, Hangzhou, China E-mail: [email protected] Adrian Cheok National University of Singapore, Singapore E-mail: [email protected] Michael Haller Upper Austria University of Applied Sciences, Hagenberg, Austria E-mail: [email protected] Rynson W.H. Lau University of Durham, Durham DH1 3LE, UK E-mail: [email protected] Hideo Saito Keio University, Yokohama, Japan E-mail: [email protected] Ronghua Liang Zhejiang University, Hangzhou, China E-mail: [email protected] Library of Congress Control Number: 2006936898 CR Subject Classification (1998): I.2, H.5, H.4, H.3, I.3, I.4, J.3, J.5 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/Web and HCI ISSN ISBN-10 ISBN-13

0302-9743 3-540-49776-5 Springer Berlin Heidelberg New York 978-3-540-49776-9 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springer.com © Springer-Verlag Berlin Heidelberg 2006 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 11941354 06/3142 543210

Preface

ICAT is the oldest international conference on virtual reality and tele-existence. ICAT 2006 not only looked for innovations in the technology itself, but also explored novel ways to transfer and express information and creative ideas to the society and people. The 16th International Conference on Artificial Reality and Telexistence was held at the Zhejiang University of Technology, Hangzhou, P.R. China from November 29 to December 1, 2006. The main purpose of the conference is to provide opportunities for researchers and practitioners to present their research findings and exchange opinions on the development and use of such systems. The conference included plenary invited talks, workshops, tutorials, and paper presentation tracks. The main conference received 523 submissions in total from 21 different countries, including China (mainland, Hong Kong, Taiwan), USA, UK, Germany, Austria, France, Australia, Canada, Korea, Japan, Malaysia, Mexico, etc., of which 138 papers were accepted for this volume and 11 papers were invited to submit extended versions for a special issue of International Journal of Virtual Reality (IJVR, 5(4)). The papers in this volume cover the topics including: Artificial Reality, VR, Telexistence, AR, Mixed Reality, Ubiquitous/Wearable Computing, Visual and Auditory Displays, Rendering Techniques, Architecture for VR, Immersive Projection Technology, Techniques for Modeling VR Systems, Virtual Heritage, Motion Tracking, Innovative Applications of VR, Haptics, Evaluation of VR Techniques and Systems, VR Interaction and Navigation Techniques, Artificial Life, Distributed and Collaborative VR Systems, VR Input and Output Devices, Virtual Medicine and Health Science, Human Factors of VR, Interactive Art and Entertainment. The wide range of questions, ideas, concepts and applications discussed in the contributions of this volume reflect the vitality and engagement of the artificial reality and tele-existence communities and their neighboring disciplines. The current research situation in these areas demands interdisciplinary cooperation and mutual stimulation. With the strong support of Springer, the proceedings of ICAT this year were published in the Lecture Notes in Computer Science (LNCS) series. November 2006

Zhigeng Pan Adrian Cheok Michael Haller Rynson Lau Hideo Saito Ronghua Liang

Organization Committee Listings General Honorary Co-chairs Deren Li (Wuhan University, China) Susumu Tachi (University of Tokyo, Japan)

General Co-chairs Zhigeng Pan (Zhejiang University, China) Hideo Saito (Keio University, Japan)

Program Co-chairs Adrain Cheok (Mixed Reality Lab, Singapore) Michael Haller (University of Applied Sciences, Hagenberg, Austria) Rynson W.H. Lau (University of Durham, UK)

Local Chair Qingzhang Chen (Zhejiang University of Technology)

Local Vice-Chairs Limin Men (Zhejiang Provincial Key Lab of Optic Fiber Communication Technology) Huageng Wan (Zhejiang University) Decai Huang (Zhejiang University of Technology) Ronghua Liang (Zhejiang University of Technology)

Publicity Co-chairs Ruigang Yang (The University of Kentucky) Xiaohong Jiang (Zhejiang University)

Publication Chair Li Li (Hangzhou Dianzi University)

Workshop Chair Takuya Nojima (Japan Aerospace Exploration Agency)

Demo Chair Masahiko Inami (the University of Electro-Communications)

Financial Co-chairs Hui Gu (Zhejiang University of Technology) Mingmin Zhang (Zhejiang University)

VIII

Organization

International Program Committee Mark Billinghurst, New Zealand Adrain Cheok, Singpore Yukio Fukui, Japan Michael Haller, Austria Naoki Hashimoto, Japan Larry Hodges, USA Masa Inakage, Japan Michael Jenkin, Canada Fumio Kishino, Japan Kiyoshi Kiyokawa, Japan Frederick Li , UK Tetsuro Ogi, Japan Christian Reimann, Germany Hideo Saito, Japan Yoichi Sato, Japan Daniel Thalmann, Switzerland Toshio Yamada, Japan Ruigang Yang, USA Shunsuke Yoshida, Japan

Tony Brooks, Denmark Gordon Clapworthy, UK Kenji Funahashi, Japan Soonhung Han, Korea Wataru Hashimoto, Japan Nicholas Holliman, UK Shuichi Ino, Japan Hiroyuki Kajimoto, Japan Itaru Kitahara, Japan Rynson W.H. Lau, UK Robert Lindeman, USA Kenichi Okada, Japan Jeha Ryu, Korea Kosuke Sato, Japan Shamus Smith, UK Ulrich Nlrichmann, USA Hiroyuki Yamamoto, Japan Hiroaki Yano, Japan Hongbin Zha, China

Andreas Butz, Germany Michael Cohen, Japan Norihiro Hagita, Japan Harold Thwaites, Malaysia Michitaka Hirose, Japan Yasushi Ikei, Japan Yutaka Ishibashi, Japan Myoung-Hee, Korea Yoshifumi Kitamura, Japan Gun Lee, Korea Takeshi Naemura, Japan Zhigeng Pan, China Amela Sadagic, USA Makoto Sato, Japan Wookho Son, Korea Greg Welch, USA Ungyeon Yang, Korea Naokazu Yokoya, Japan

Reviewer Listings Song Aiguo Congchen Chen Adrain Cheok Weilong Ding Xianyong Fang Kenji Funahashi Soonhung Han Hanwu He Decai Huang Yutaka Ishibashi Xiaohong Jiang Fumio Kishino Rynson W.H. Lau Hua Li Ronghua Liang Yongkui Liu Takeshi Naemura Yanjun Peng Amela Sadagic Yoichi Sato Chengfang Song Zheng Tan Jianhua Tao Huageng Wan Qiang Wang Yongtian Wang Haihong Wu

Qing Xu Bailing Yang Hiroaki Yano Zhengsheng Yu Jiawan Zhang Wei Zhang Bingfeng Zhou Mark Billinghurst Dingfang Chen Gordon Clapworthy Xiumin Fan Jieqing Feng Hui Gu Naoki Hashimoto Michitaka Hirose Yasushi Ikei Michael Jenkin Xiaogang Jin Itaru Kitahara Gun Lee Li Li Robert Lindeman Jianfeng Lu Tetsuro Ogi Xujia Qin Hideo Saito Shamus Smith

Jizhou Sun Bing Tang Daniel Thalmann Guoping Wang Ruchuan Wang Greg Welch Lingda Wu Xiaogang Xu Ungyeon Yang Zhiqiang Yao Ruwei Yun Maojun Zhang Nailiang Zhao Jiejie Zhu Tony Brooks Ling Chen Michael Cohen Jinglong Fang Jun Feng Norihiro Hagita Wataru Hashimoto Larry Hodges Masa Inakage Qingge Ji Hiroyuki Kajimoto Yoshifumi Kitamura Fengxia Li

Zili Li Yushen Liu Xiaolan Luo Kenichi Okada Christian Reimann Kosuke Sato Wookho Son Shusen Sun Jie Tang Harold Thwaites Yangsheng Wang Yigang Wang Enhua Wu Hui Xiang Toshio Yamada Ruigang Yang Naokazu Yokoya Hongbin Zha Mingmin Zhang XiangJun Zhao Andreas Butz Wei Chen Guofu Ding Tongzhu Fang Yukio Fukui Michael Haller Gaoqi He

Organization Nicholas Holliman Shuichi Ino Jinyuan Jia Myoung-Hee Kim Kiyoshi Kiyokawa Xuelong Li Frederick Li

Jianfei Mao Zhigeng Pan Jeha Ryu Makoto Sato Mingli Song Zhengxing Sun Yong Tang

Ulrich Neumann Jianming Wang Zhaoqi Wang Fuli Wu Yongliang Xiao Hiroyuki Yamamoto Hongwei Yang

IX

Shunsuke Yoshida Yongzhao Zhan Qiong Zhang Wenting Zheng

Acknowledgements and Sponsoring Institutions ICAT 2006 was sponsored by VRSJ (Virtual Reality Society in Japan) and was organized by： z z

Zhejiang University of Technology VR Committee, China Society of Image and Graphics

However, ICAT 2006 owes its big success to the financial and organizational support of various institutions including: z z z z

DEARC, Zhejiang University Computer Network and Multimedia Lab, Zhejiang University of Technology International Journal of Virtual Reality (IJVR) National Science Foundation of China

We would like to thank all of them for offering the opportunity to organize ICAT 2006 in a way that provided a diversified scientific and social program. Especially, we would like to thank all members of the International Program Committee and Organizing Committee for their great job in defining conference topics, reviewing the large number of submitted papers, and managing to put all the material together for this great event.

Table of Contents

Anthropomorphic Intelligent Robotics, Artificial Life Gesture Recognition Based on Context Awareness for Human-Robot Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Seok-Ju Hong, Nurul Arif Setiawan, Song-Gook Kim, Chil-Woo Lee

1

Evolving Creatures in Virtual Ecosystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nicolas Lassabe, Herve Luga, Yves Duthen

11

The Application of Affective Virtual Human in Intelligent Web Tutoring System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Huiling Peng

21

Extracting Behavior Knowledge and Modeling Based on Virtual Agricultural Mobile Robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xiangjun Zou, Jun Lu, Lufeng Luo, Xiwen Luo, Yanqiong Zhou

28

Human Motion Interpolation Using Space-Time Registration . . . . . . . . . . . Zhihua Chen, Lizhuang Ma, Yan Gao, Xiaomao Wu

38

Multi-stream Based Rendering Resource Interception and Reconstruction in D3DPR for High-Resolution Display . . . . . . . . . . . . . Zhen Liu, Jiaoying Shi, Hua Xiong, Aihong Qin

48

VirtualPhonepad: A Text Input Tool for Virtual Environments . . . . . . . . . . Jaewoo Ahn, Kyungha Min

58

An Animation System for Imitation of Object Grasping in Virtual Reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Matthias Weber, Guido Heumer, Heni Ben Amor, Bernhard Jung

65

Generating Different Realistic Humanoid Motion . . . . . . . . . . . . . . . . . . . . . . Zhenbo Li, Yu Deng, Hua Li

77

Handheld AR for Collaborative Edutainment . . . . . . . . . . . . . . . . . . . . . . . . . Daniel Wagner, Dieter Schmalstieg, Mark Billinghurst

85

XII

Table of Contents

VR-Based Simulation for the Learning of Gynaecological Examination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Liliane dos Santos Machado, Ronei Marcos de Moraes

97

Visual Perception Modeling for Intelligent Avatars . . . . . . . . . . . . . . . . . . . . 105 Ronghua Liang, Tinan Huang, Meleagros Krokos, Jinglong Fan Steering Behavior Model of Visitor NPCs in Virtual Exhibition . . . . . . . . . 113 Kyungkoo Jun, Meeyoung Sung, Byoungjo Choi

Augmented Reality/Mixed Reality Scalable Architecture and Content Description Language for Mobile Mixed Reality Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 Fumihisa Shibata, Takashi Hashimoto, Koki Furuno, Asako Kimura, Hideyuki Tamura A Novel Sound Localization Experiment for Mobile Audio Augmented Reality Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 Nick Mariette A Tangible User Interface for Remote Collaboration System Using Mixed Reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Yuichi Bannai, Hidekazu Tamaki, Yuji Suzuki, Hiroshi Shigeno, Kenichi Okada Multi-sensor Data Fusion Based on Fuzzy Integral in AR System . . . . . . . . 155 Yan Feng, Yimin Chen, Minghui Wang AR Baseball Presentation System with Integrating Multiple Planar Markers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Yuko Uematsu, Hideo Saito An Efficient 3D Registration Method Using Markerless Image in AR-Based CNC Machining Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 Z.Y. Ma, Y.P. Chen, C.M. Yuan, Z.D. Zhou Shared Design Space: Sketching Ideas Using Digital Pens and a Large Augmented Tabletop Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 Michael Haller, Peter Brandl, Daniel Leithinger, Jakob Leitner, Thomas Seifried, Mark Billinghurst AR Table Tennis: A Video-Based Augmented Reality Sports Game . . . . . 197 Jong-Seung Park, TaeYong Kim, Jong-Hyun Yoon

Table of Contents

XIII

Enhancing Immersiveness in AR-Based Product Design . . . . . . . . . . . . . . . . 207 Taejin Ha, Yeongmi Kim, Jeha Ryu, Woontack Woo Personal Information Annotation on Wearable Computer Users with Hybrid Peer-to-Peer Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 Koji Makita, Masayuki Kanbara, Naokazu Yokoya An Initialization Tool for Installing Visual Markers in Wearable Augmented Reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 Yusuke Nakazato, Masayuki Kanbara, Naokazu Yokoya Increasing Camera Pose Estimation Accuracy Using Multiple Markers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 Jong-Hyun Yoon, Jong-Seung Park, Chungkyue Kim Exploring Human-Robot Interaction Through Telepresence Board Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 Min Xin, Ehud Sharlin Projected Reality – Content Delivery Right onto Objects of Daily Life . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262 Jochen Ehnes, Michitaka Hirose An Evaluation of an Augmented Reality Multimodal Interface Using Speech and Paddle Gestures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 Sylvia Irawati, Scott Green, Mark Billinghurst, Andreas Duenser, Heedong Ko An Augmented Reality System for Computer Maintenance . . . . . . . . . . . . . 284 Bo Kang, Peng Ren, Changzhi Ke Multi-view Video Composition with Computer Graphics . . . . . . . . . . . . . . . 292 Artem Ignatov, Manbae Kim Floating Virtual Mirrors: Visualization of the Scene Behind a Vehicle . . . . 302 Toru Miyamoto, Itaru Kitahara, Yoshinari Kameda, Yuichi Ohta

Distributed and Collaborative VR System A Scalable Framework for Distributed Virtual Reality Using Heterogeneous Processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314 Qishi Wu, Jinzhu Gao, Mengxia Zhu A Collaborative Virtual Reality Environment for Molecular Modeling . . . . 324 Sungjun Park, Jun Lee, Jee-In Kim

XIV

Table of Contents

An Interest Filtering Mechanism Based on LoI . . . . . . . . . . . . . . . . . . . . . . . . 334 Sheng Wei, Zhong Zhou, Wei Wu Study on Data Collaboration Service for Collaborative Virtual Geographic Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344 Jun Zhu, Jianhua Gong, Hua Qi, Tao Song VCS: A Virtual Collaborative Space Based on Immersive Teleconferencing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354 Weidong Bao, Maojun Zhang, Wang Chen A Scalable HLA-Based Distributed Simulation Framework for VR Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361 Zonghui Wang, Jiaoying Shi, Xiaohong Jiang Location-Based Device Ensemble Architecture for a Spatial Reality Enhancement of Distributed Multimedia Presentation . . . . . . . . . . . . . . . . . 372 Doo-Hyun Kim, Lila Kim, Hwasun Kwon, Dongwoon Jeon, Songah Chae

Haptics, Human Factors of VR Air-Jet Button Effects in AR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384 Yeongmi Kim, Sehun Kim, Taejin Ha, Ian Oakley, Woontack Woo, Jeha Ryu Perceptualizing a “Haptic Edge” with Varying Stiffness Based on Force Constancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392 Jaeyoung Cheon, Seungmoon Choi Multi-sensory Perception of Roughness: Empirical Study on Effects of Vibrotactile Feedback and Auditory Feedback in Texture Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406 Ki-Uk Kyung, Dong-Soo Kwon The Research of Spiritual Consolation Object with the Function of Affective Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416 Yiping Tang, Hongjie Sun, Xiaokai Gu, Tao Deng Real Time Dynamics and Control of a Digital Human Arm for Reaching Motion Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425 Huijie Yu, Ray P.S. Han

Table of Contents

XV

Emotion Recognition Using Physiological Signals . . . . . . . . . . . . . . . . . . . . . . 437 Lan Li, Ji-hua Chen

Innovative Applications of VR Animation Preserving Simplification for MPEG Based Scalable Virtual Human . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447 HyungSeok Kim A Virtual Reality Training System for Robot Assisted Neurosurgery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457 Da Liu, Tianmiao Wang The Experiment of Improving Students’ Spatial Ability by Using VGLS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467 Ruwei Yun, Haixu Xi, Yi Li Sketch Based 3D Animation Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474 Difei Lu, Xiuzi Ye Extracting Space Welding Seam for Virtual Teach-In of Robotic Welding System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486 Wusheng Chou, Tianmiao Wang Primary-Color-Based Spatial Layout Features and New Image Matching Algorithm Based on Dual Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494 De-cai Huang, Jia Hu, Yuan Yuan Ridge-Valley Lines Smoothing and Optimizing . . . . . . . . . . . . . . . . . . . . . . . . 502 Hao Jing, Weixiang Zhang, Bingfeng Zhou Mynews: Personalization of Web Contents Transcoding for Mobile Device Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512 Teuk-Seob Song, Jin-Sang Lee, Yoon-Chul Choy, Soon-Bum Lim Visualization Application in Clothing Biomechanical Design . . . . . . . . . . . . 522 Ruomei Wang, Yi Li, Xiaonan Luo, Xin Zhang

Motion Tracking Robot Position Estimation and Tracking Using the Particle Filter and SOM in Robotic Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530 TaeSeok Jin, JangMyung Lee

XVI

Table of Contents

Robust Motion Tracking in Video Sequences Using Particle Filter . . . . . . . 540 Guixi Liu, Chunyu Fan, Enke Gao Fast Motion Estimation Using Spatio-temporal Correlations . . . . . . . . . . . . 548 Hyo Sun Yoon, Jae Myeong Yoo, Toan Nguyen Dinh, Hwa Jeong Son, Mi Seon Park, Guee Sang Lee Bi-directional Passenger Counting on Crowded Situation Based on Sequence Color Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557 Ning Liu, Chengying Gao Moving Object Detection Based on a New Level Set Algorithm Using Directional Speed Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565 Dong-Gyu Sim An Integrated Robot Vision System for Multiple Human Tracking and Silhouette Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575 Jung-Ho Ahn, Sooyeong Kwak, Cheolmin Choi, Kilcheon Kim, Hyeran Byun Disparity Weighted Histogram-Based Object Tracking for Mobile Robot Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584 Cheolmin Choi, Jungho Ahn, Seungwon Lee, Hyeran Byun Towards Robust 3D Reconstruction of Human Motion from Monocular Video . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594 Cheng Chen, Yueting Zhuang, Jun Xiao Wavelet Scalable Video Coding with Minimal Average Temporal Distance for Motion Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604 Changhoon Yim, Hojin Ha, Bae-Keun Lee, Woo-Jin Han Single Camera Remote Eye Gaze Tracking Under Natural Head Movements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614 Ruian Liu, Shijiu Jin, Xiaorong Wu

Real Time Computer Simulation High-Quality Shear-Warp Volume Rendering Using Efficient Supersampling and Pre-integration Technique . . . . . . . . . . . . . . . . . . . . . . . . 624 Heewon Kye, Kyoungsu Oh

Table of Contents

XVII

A Simple, Efficient Method for Real-Time Simulation of Smoke Shadow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633 Minghao Yang, Guangzheng Fei, Minyong Shi, Yongsong Zhan Reducing Interrupt Latency at Linux Kernel Level for Real-Time Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643 Dae Sung Lee, Jeong Seob Kim, Jaehyun Park, Ki Chang Kim Investigation of Surface Integrity in the Case of Chemical Mechanical Polishing Silicon Wafer by Molecular Dynamics Simulation Method . . . . . 651 Xuesong Han Independent Component Analysis Approach for the Neurobiological Recording Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 660 Minghai Yao, Jing Hu, Qinglong Gu Real-Time Virtual Surgery Simulation Employing MM-Model and Adaptive Spatial Hash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 666 Shaoting Zhang, Lixu Gu, Weiming Liang, Jingsi Zhang, Feng Qian Creating Dynamic Panorama Using Particle Swarm Optimization . . . . . . . 676 Yan Zhang, Zhengxing Sun, Wenhui Li Real-Time and Realistic Simulation of Large-Scale Deep Ocean Surface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 686 Xin Zhao, FengXia Li, ShouYi Zhan Exploiting Frame-to-Frame Coherence for Rendering Terrain Using Continuous LOD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 695 Jing Zhang, Guojun Chen Light Field Modeling and Its Application to Remote Sensing Image Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 705 Mingxiang Huang, Jianhua Gong, Zhou Shi, Lihui Zhang Simple and Fast Terrain Rendering Using Graphics Hardware . . . . . . . . . . 715 Hanli Zhao, Xiaogang Jin, Jianbing Shen Animating Grass in Real-Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 724 Linqiang Yao, Liyu Tang, Chongcheng Chen, Jingjing Sun Gaussian Mixture Model in Improved HLS Color Space for Human Silhouette Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 732 Nurul Arif Setiawan, Seok-Ju Hong, Jang-Woon Kim, Chil-Woo Lee

XVIII

Table of Contents

Geometry Compression Using Spherical Wavelet . . . . . . . . . . . . . . . . . . . . . . 742 Guojun Peng, Jiawan Tan, Zhigeng Pan, Yicheng Jin

Tools and Technique for Modeling VR Systems Dynamic Radiosity on Multi-resolution Models . . . . . . . . . . . . . . . . . . . . . . . 753 Hui Xiao, Gary Tam, Frederick Li, Rynson W.H. Lau Mesh Simplification Algorithm Based on N-Edge Mesh Collapse . . . . . . . . 764 Hua-hong Chen, Xiao-nan Luo, Ruo-tian Ling 3-D Personalized Face Reconstruction Based on Multi-layer and Multi-region with RBFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 775 Yongzhao Zhan, Rongrong Shen, Jianming Zhang Recognition and Location of Fruit Objects Based on Machine Vision . . . . 785 Hui Gu, Yaya Lu, Jilin Lou, Weitong Zhang A New Algorithm of Gray Watermark Embedding . . . . . . . . . . . . . . . . . . . . 796 Yong-zeng Shen, Min-jie zhang, Feng Liu A Semiautomatic Nonrigid Registration in Liver Cancer Treatment with an Open MR System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 802 Songyuan Tang, Yen-wei Chen, Rui Xu, Yongtian Wang, Shigehiro Morikawa, Yoshimasa Kurumi Example-Based Realistic Terrain Generation . . . . . . . . . . . . . . . . . . . . . . . . . 811 Qicheng Li, Guoping Wang, Feng Zhou, Xiaohui Tang, Kun Yang Bayesian Mesh Reconstruction from Noisy Point Data . . . . . . . . . . . . . . . . . 819 Guiping Qian, Ruofeng Tong, Wen Peng, Jinxing Dong Efficient Retrieval of 3D Models Based on Integer Medial Axis Skeleton . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 830 Yong Tang, Xuefan Zhang Implicit Surface Boolean Operations Based Cut-and-Paste Algorithm for Mesh Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 839 Xujia Qin, Xinhua Yang, Hongbo Zheng Meshes Simplification Based on Reverse Subdivision . . . . . . . . . . . . . . . . . . . 849 Jian-ping Ma, Xiao-nan Luo, Shi-jia Ma, Guifeng Zheng

Table of Contents

XIX

Kd-Tree Based OLS in Implicit Surface Reconstruction with Radial Basis Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 861 Peizhi Wen, Xiaojun Wu, Tao Gao, Chengke Wu Retrieving 3D CAD Model with Regional Entropy Distributions . . . . . . . . 871 Liu Wei, He Yuanjun Virtual Tang-Style Timber-Frame Building Complex . . . . . . . . . . . . . . . . . . 880 Deren Li, Yixuan Zhu, Zhiqiang Du, Tao Hong An Accelerating Rendering Method of Hybrid Point and Polygon for Complex Three-Dimensional Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 889 Aimin Hao, Guifen Tian, Qinping Zhao, Zhide Li User Mediated Hypermedia Presentation Generation on the Semantic Web Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 901 Jayan C. Kurian, Payam M. Barnaghi, Michael Ian Hartley Progressive Transmission of Vector Map Data Based on Polygonal Chain Simplification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 908 Haisheng Zhan, Guangxin Li Image-Based Fast Small Triangle Rasterization . . . . . . . . . . . . . . . . . . . . . . . 918 Jim X. Chen, Harry Wechsler, Jian Cui On Volume Distribution Features Based 3D Model Retrieval . . . . . . . . . . . . 928 Mingyong Pang, Wenjun Dai, Gangshan Wu, Fuyan Zhang Personalized Model Deformation Based on Slice Space Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 938 Lifeng Ren, Mingmin Zhang, Yunlong Xu, Zhigeng Pan Flexible Camera Setup for Visual Based Registration on 2D Interaction Surface with Undefined Geometry Using Neural Network . . . . . . . . . . . . . . . 948 Ary Setijadi Prihatmanto, Michael Haller, Roland Wagner Painterly Rendering with Vector Field Based Feature Extraction . . . . . . . . 960 Chen Pang, Mingli Song, Jiajun Bu, Chun Chen, Dong Wang

Ubiquitous/Wearable Computing Reducing Time Cost of Distributed Run-Time Infrastructure . . . . . . . . . . . 969 Zhong Zhou, Qinping Zhao

XX

Table of Contents

Wearable Multimodal Interface for Helping Visually Handicapped Persons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 980 Byeong-Seok Shin, Honam Ahn, Gye Young Kim Personalized Primary Port: Analysis of User View for the Smart Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 989 Wei-jin Park, Junhwan Kim, Doo-Hwan Bae Mobile Information Presentation Schemes for Supra-adaptive Logistics Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 998 Bj¨ orn Schwerdtfeger, Troels Frimor, Daniel Pustka, Gudrun Klinker Interactive Collaborative Scene Assembly Using AR on Mobile Phones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1008 Miroslav Andel, Alexander Petrovski, Anders Henrysson, Mark Ollila Design of the Multimedia Communication Protocol and System for Wireless LAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1018 Qingzhang Chen, Jianghong Han, Keji Mao

Virtual Heritage, Virtual Medicine and Health Science Representation of Color in Ancient Noh Masks . . . . . . . . . . . . . . . . . . . . . . . 1026 Xin Yin, Yasuo Miichi, Hiromi T. Tanaka Modeling and Immersive Display of Cultural Heritage . . . . . . . . . . . . . . . . . 1035 Lei Zhao, Duan-qing Xu, Yi Li, Yi-min Yu Personalized Content Presentation for Virtual Gallery . . . . . . . . . . . . . . . . . 1045 Wonil Kim, Hanku Lee, Kyoungro Yoon, Hyungseok Kim, Changduk Jung Accessible Information from Radiated Sound of Extracorporeal Shock Wave Lithotriptor for Improving Fragmentation Efficiency . . . . . . . . . . . . . 1055 Yun Seok Jang, Jin Ho Choi Visualization of Segmented Color Volume Data Using GPU . . . . . . . . . . . . 1062 Koo-Joo Kwon, Byeong-Seok Shin Research on Segmentation Algorithm of 3d Medical Data . . . . . . . . . . . . . . 1070 Yanjun Peng, Dandan Zhang, Weidong Zhao, Jiaoying Shi, Yongguo Zheng

Table of Contents

XXI

Kidney Diseases Diagnosis System for Sensation Type Using Physiological Signal Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1078 Bong-hyun Kim, Se-hwan Lee, Dong-uk Cho A Computer-Aided Driving Posture Prediction System Based on Driver Comfort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1088 Qun Wu, Shiwu Luo, Shouqian Sun Height-Based Deformation and Ray Supersampling for Colon Unfolding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1098 Honam Ahn, Byeong-Seok Shin Content-Based Retrieval of Cultural Relic Images in Digital Museum . . . . 1108 Tongwei Ren, Gangshan Wu Virtual Biology Modeling Method Based on Validity Confirmation Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1118 Ying Liang, Wen-Yong Wang, Shao-Chun Zhong, Qing-Rong Zhang, Yao Wang, Xiao-Lin Quan, Qian Xu

Virtual Reality Virtual Reconstruction of Two Types of Traffic Accident by the Tire Marks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1128 Xiaoyun Zhang, Xianlong Jin, Jie Shen A Seamless Visualizaton Model of the Global Terrain Based on the QTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1136 Xuesheng Zhao, Jianjun Bai, Jun Chen, Zhilin Li Photographs Alignment and High Dynamic Range Image Composition Based on Varying Exposure Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1146 Shungang Hua, Lidan Wang Elastic Algorithm: A New Path Planning Algorithm About Auto-navigation in 3D Virtual Scene . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1156 Yong Chen, Ke-qin Wu, Dong Wang, Ge Chen Mental Representation of Spatial Structure in a Virtual Room-Sized Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1166 Zhiqiang Luo, Henry Been-Lirn Duh A Novel Planar Walking Algorithm for Virtual Walking Machine . . . . . . . . 1175 Jangwoo Park, Jungwon Yoon, Yo-An Lim, Jeha Ryu

XXII

Table of Contents

An Interactive System of Virtual Face Pose Estimation . . . . . . . . . . . . . . . . 1186 Zheng-yi Wu, Jian Shi, Xiaoming Tong, Zheng Tan An Efficient Manipulation of Game Contents on Heterogeneous Platforms Using MR Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1193 Gi Sook Jung, Seung-Dae Jeong, Kyung Ho Jang, Soon Ki Jung Manipulation of Field of View for Hand-Held Virtual Reality . . . . . . . . . . . 1204 Jane Hwang, Jaehoon Jung, Gerard Jounghyun Kim Hierarchical 3D Data Rendering System Synchronizing with HTML . . . . . 1212 Yousuke Kimura, Tomohiro Mashita, Atsushi Nakazawa, Takashi Machida, Kiyoshi Kiyokawa, Haruo Tamekura Fragment Indexing of XML-Based Metadata for Digital TV Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1223 Hyoseop Shin, Minsoo Lee, Seokhyun Son Using a MyoKinetic Synthesizer to Control of Virtual Instruments . . . . . . 1233 Duk Shin, Atsushi Katayama, Kyoungsik Kim, Hiroyuki Kambara, Makoto Sato, Yasuharu Koike

VR Interaction and Navigation Techniques View-Dependent Simplification of Complex Urban Scenes Using Weighted Quadtrees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1243 Bum-Jong Lee, Jong-Seung Park, Mee Young Sung Indoor Environment Modeling for Interactive VR – Based Robot Security Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1253 Sangwoo Jo, Yong-Moo Kwon, Hanseok Ko Research on Mass Terrain Data Storage and Scheduling Based on Grid GIS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1263 Jing Zhang, Hai Huang Multi-touch Interaction for Table-Top Display . . . . . . . . . . . . . . . . . . . . . . . . 1273 Song-Gook Kim, Jang-Woon Kim, Ki-Tae Bae, Chil-Woo Lee Reliable Omnidirectional Depth Map Generation for Indoor Mobile Robot Navigation Via a Single Perspective Camera . . . . . . . . . . . . . . . . . . . . 1283 Chuanjiang Luo, Feng Zhu, Zelin Shi

Table of Contents

XXIII

Enhanced 3D Interaction with Augmented Visual Information . . . . . . . . . . 1293 Dongwook Lee, Jinah Park Research and Application of Virtual Reality in the Field of Web-Based Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1305 Chengling Zhao, Zhaohua Zhang Indoor/Outdoor Pedestrian Navigation with an Embedded GPS/RFID/Self-contained Sensor System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1310 Masakatsu Kourogi, Nobuchika Sakata, Takashi Okuma, Takeshi Kurata A Communication Model of Scents Mediated by Sense-Descriptive Adjectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1322 Yuichi Bannai, Masayuki Ishizawa, Hiroshi Shigeno, Kenichi Okada Hand Shape Recognition Using a Mean-Shift Embedded Active Contour (MEAC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1333 Eun Yi Kim Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1343

Gesture Recognition Based on Context Awareness for Human-Robot Interaction Seok-Ju Hong, Nurul Arif Setiawan, Song-Gook Kim, and Chil-Woo Lee Intelligent Media Lab, Department of Computer Engineering, Chonnam National University, Gwangju, Korea Tel.: 82-62-530-1803 [email protected], [email protected], [email protected], [email protected]

Abstract. In this paper, we describe an algorithm which can naturally communicate with human and robot for Human-Robot Interaction by utilizing vision. We propose a state transition model using attentive features for gesture recognition. This method defines the recognition procedure as five different states; NULL, OBJECT, POSE, Local Gesture and Global Gesture. We first infer the situation of the system by estimating the transition of the state model and then apply different recognition algorithms according to the system state for robust recognition. And we propose Active Plane Model (APM) that can represent 3D and 2D information of gesture simultaneously. This method is constructing a gesture space by analyzing the statistical information of training images with PCA and the symbolized images are recognized with HMM as one of model gestures. Therefore, proposed algorithm can be used for real world application efficiently such as controlling intelligent home appliance and humanoid robot. Keywords: Gesture Recognition, Context Awareness, PCA, HMM.

1

Introduction

Recently, Human gesture analysis has received more attentions in computer vision field with applications in areas such as human friendly interfaces, intelligent surveillance, virtual reality, and so on. Especially, natural communication method between human and computer is becoming a key issue as intelligent interface techniques can be applied to home appliances. Human acquire the information over 80% from their eyes. And, we understand the human behavior using the temporal sequence of movement of human body automatically, and then to estimate some meanings of the temporal movement from previously trained and arranged motion data. Also, human uses non-linguistic method for their communication and it is better to make more natural communication with each other than using only linguistic method. However, it is very difficult to analyze temporal changes by computer, namely historical meaning of bodily motion

Corresponding author.

Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1–10, 2006. c Springer-Verlag Berlin Heidelberg 2006

2

S.-J. Hong et al.

automatically because a human body is a three-dimensional object with very complicated structure and flexibility. In early works, many researchers tried to measure the configuration of the human body with sensors attached to the joints of limbs and to recognize gesture by analyzing the variation of joint angles [3][4]. But, this requires the user to put on irritating devices, and it usually needs long cables connection from the devices to computers. So, it hinders the easy usage and embarrasses the user with unnatural dullness. Also, other algorithms use various markers attaching to the joints of a moving articulated body. With this reason, commercial products adopting the algorithm become very expensive and real-time processing can not be allowed. To overcome these problems, video-based recognition techniques are proposed and it uses a set of video cameras and computer vision techniques to interpret gestures. One of the famous algorithms using global motion feature information is Motion History Image (MHI) method. In this method, the accumulation of object silhouettes is used to analyze the human motion [5]. Many other algorithms which use feature points or silhouette of body are proposed [6], but the methods have the generalization problems for recognition results. These systems have no problems in the case of recognizing the pre-trained gestures, but if the gestures are not defined previously, systems can not recognize or misunderstand the gestures. Moreover, because the human gestures can not be trained for all cases, it is easy to develop the restricted and system dependent algorithms. To address these problems, we propose the gesture recognition algorithm using attentive features; probability of state, global and local motion and positional information. In this algorithm, the system classifies the situation of behaviors before recognizing the individual meaning of gestures. So, proposed algorithm can analyze the human gestures more precisely and it has a feature that can expand the application of recognition system. And we describe the algorithm which can automatically recognize human gesture without such constraints by utilizing three-dimensional features extracted from stereo images. The paper structured as follows: Section 2 and 3 present a probabilistic state transition model and gesture interface system using the state transition model. And in Section 4.1, we explain Active Plane Model(APM) that the bodily region and features can be extracted from the stereo images. And in section 4.2, explain how the gesture model space is constructed by analyzing the statistical information of training images with the Principal Component Analysis (PCA) method. And then, section 4.3 presents how the sequential poses are classified into model gesture with Hidden Markov Model (HMM) algorithm. Finally, the paper is closed with mentioning the conclusion and further works in Section 5.

2

Overview

Before taking up the main subject, we need to define the terminology ”context” clearly. According to the Cambridge dictionary, ”context” is defined as the situation within which something exists or happens, and it also means that variation of circumstance or locational information of robot in engineering fields. In this paper, we define the ”context” as intention of behaviors between human

Gesture Recognition Based on Context Awareness

3

Table 1. Definition of states State

Definition

NULL OBJECT POSE Global Gesture Local Gesture

No object in view. Something is moving, but it is not human. Behavior is detected, and he/she is posturing. Behavior is detected, and he/she has a global motion. Behavior is detected, and he/she has a local motion.

and computer. This system uses only cameras for communication, and does not contain the situation using sound or any other sensors. In this paper, we define the human gestures as five different states based on movement of subject. And then, we can represent the human gestures by continuous sequence of states, also we can understand intention of actor using the state information. And we present definition of five different states in Table 1. In the first step, the system is trained with the image sequences. Through the training process, we can get the probability of state transition and the weights of feature vectors. And then, the system decides the state transition by using this information. Finally, we can apply the reasonable recognition algorithm to the system according to the context information.

3

Probabilistic State Transition Model

3.1

Feature Extraction and Training

In this paper, we use image features to decide a transition of state, and these image features are as follows: f1 f2 f3 f4 f5

: : : : :

Motion information in image Locational information of subject Distance from subject Motion information of subject 3D trajectories of head and hands

The system can extract and track the subject’s head and hands regions using skin tone every frame from stereo image. And then, we calculate threedimensional depth information using this information and we can get the image features. The training image sequences include the state information, and system can be established the probabilistic model using these sequences. Through the training process, we can get the probabilities of each image features and its weights. Let the 5 image features is fi (i =1,2,3,4,5), feature vector set F and its probabilities P (F ) can be denoted as equation (1). F = {f1 , f2 , f3 , f4 , f5 } P (F ) = P (fi ) i

(1) (2)

4

S.-J. Hong et al.

Also, if the set of states is Sk (k = 1, 2, 3, 4, 5) conditional probabilities of feature vector set on each state are as follows: P (F |S1 ) : Prob. of set F on the NULL P (F |S2 ) : Prob. of set F on the Object P (F |S3 ) : Prob. of set F on the Pose P (F |S4 ) : Prob. of set F on the Global Motion P (F |S5 ) : Prob. of set F on the Local Motion 3.2

State Transition

We adopt Bayes’ Theorem to decide transition between each state. First, the probabilities of set on the each state (a prior probability) are calculated from training process. And then, we can estimate the probabilities of transition to the each state (a posterior probability). Finally, the system is updated to the state information which has the maximum probabilities. The probabilities of transition with feature vector set F are denoted as equation (3). Also, the system adjusts the weights of features wi (i = 1, 2, 3, 4, 5) using the weight information as shown in equation (4). P (Sk )P (F |Sk ) P (Sk |F ) = P (Si )P (F |Si )

(3)

i

P (F ) =

wi P (fi )

(4)

i

If the feature vector set is extracted on the state Sjt at time t, the probabilities of transition to each state is calculated using adjusted feature vector set F and prior probabilities. And then, the system is updated to the state Skt+1 which has the maximum probabilities as shown in equation (5). k = arg max P (Sjt+1 |F ) j

4

(5)

Gesture Recognition

The proposed recognition algorithm is marked by recognizing the gestures according to the situational information. It is easy to develop the restricted and system dependent algorithms, because all of human gestures can not be classified using only a small number of features. In this section, we propose the recognition system that can reduce limitations using context awareness. 4.1

Feature Extraction: Active Plane Model

In general, the scene includes a lot of objects in the background that may disturb gesture recognition. This background must eliminate because it is not need to processing information of gesture. In this paper, we propose Active Plane Model(APM) that represent simultaneously gesture feature information and three dimensional depth information.


5

We use a simple foreground extraction method using depth information. First, detect a face region of the frontal person using skin tone, and then estimate the distance from the robot camera to the face by calculating stereo geometry. Finally, we can obtain the closer region of input images as the foreground. It is shown in Equation (6). 0 : D(x, y) > Df + c F (x, y) = (6) F (x, y) : D(x, y) ≤ Df + c In the equation, F (x, y) is gray-level value of input image at position (x, y) of the corresponding disparity image, D(x, y) is the distance from robot camera for each pixel, Df is distance from camera to the face, and c is a constant value considering thickness of the body.

Fig. 1. Foreground extraction using depth information(Disparity Image)

The APM is characterized by that it can represent three dimensional depth information and two dimensional shape information at the same time. And you can see standardization model of APM in Fig. 2 and deformed APM examples in Fig. 3.

Fig. 2. Standardization of APM (Red rectangle of left side is the face region)

APM consist of several nodes connected to each other in grid shape and it is deformed by the shape and depth information. For the first step to deform the APM, we find the contour of foreground region. The contour is searched from the minimal rectangular boundary surrounding foreground region into center of rectangle. If we find the contour, outer nodes of APM are fixed, then, inner nodes are moved to new location in order to keep the same distance between neighbor nodes. That means the inner nodes are rearrange by calculating the average

6

S.-J. Hong et al.

Fig. 3. Deformed APM examples(Normal, I Love You, Stop, Bye Bye, Point)

position between the most outer nodes as shown in equation (7). In equation (8), T expresses a suitable threshold distance. By adopting the criterion of equation (3), we can stop the deformation quickly. Nit+1 = average position of neighbor nodes of Nit

(7)

Nit+1 − Nit ≤ T , (t ≥ 0)

(8)

Therefore, multi dimensional features are extracted for every frame by using the equation (9), Ft = {N1 , N2 , ..., Nn } Ni = {xi , yi , zi } (9) (0 ≤ t ≤ T , 1 ≤ i ≤ nT , n ≥ 4) Where Ft is a feature vector set at time t,Ni is i-th nodal position of APM, and n is the total number of APM nodes. 4.2

Pose Symbolization Using PCA

A common method for linear dimension reduction is Principal Components Analysis (PCA). The method is performed by applying eigenvalue decomposition on the covariance matrix of feature data. A small subset of resulting eigenvectors, covering a desired amount of the data variance from a new linear subspace for representing the data set. Extracted feature vector make projection of model gesture sequence into the gesture space in Fig. 4. PCA is clearly capable of reducing the dimensionality of data sets with structure, but not necessarily suitable for finding spatio-temporal structure for several reasons. The method assumes that the underlying structure of the data is linear. Unless the underlying structure is linear, the resulting subspace will overestimate the reduced dimensionality and yield wrong principal components and reduced data that do not allow for the discovery of structure. The features extracted from an image sequence are multi dimensional and we can not solve the multi dimension problem easily. By subtracting the average vector c of the all features, as in equation (10), the new feature matrix X is obtained. The covariance matrix Q of gesture features can be obtained from equation (8). Then the PCA


7

Fig. 4. Projection of model gesture sequence into the gesture space

is straight-forward requiring only the calculation of the eigenvectors satisfying equation (12). N c = (1/N ) xi (10) i=1

X∼ = [x1 − c, x2 − c, · · · , xN − c]T

(11)

Q∼ = XX T

(12)

λi ei = Qei

(13)

And as input feature, set x is subtracted from an average vector c and projected into the eigenspace as in equation (14). pi = [e1 , e2 , · · · , ek ]T (xi − c)

(14)

Using the Principal Component Analysis, a small number of component vectors are chosen which can represent the whole space, and we call it dimension reduction. 4.3

Symbolic Gesture Recognition Using HMM

HMM is a stochastic process and also a probabilistic network with hidden and observable states. A time domain process demonstrates a Markov property if the conditional probability density of the current event, even though all present and past events are given, depends only on the j-th most recent events. If the current event depends solely on the most recent past event, then the process is a first order Markov process. The initial topology for an HMM can be determined by estimating how many different states are involved in specifying a sign. Fine tuning for this topology can be performed empirically. HMM λ is represented by the several parameters. The parameter aij indicates state transition probability of HMM changing from state i to state j. The parameter bij (y) indicates that probability of that the output symbol y will be observable in the transition of

8

S.-J. Hong et al.

state j from state i. And another parameter πi presents the probability of the initial state. Learning of HMM is equal to estimating the parameters {π, A, B} of a sequential symbolic data. ξt (i, j) =

P (st = i, st+1 = j, Y |λ) αt (i)aij bij (yt+1 )βt+2 (j) = N N P (Y |λ) αt (i)aij bij (yt+1 )βt+2 (j)

(15)

i=1 j=1

γt (i) =

N

ξt (i, j)

(16)

j=1

In equation (15), ξt (i, j) is the probability of being in state Si at time t and in state Sj at time t + 1 , and γt (i) is the probability of being state Si at time t. Using equation (15) and (16) the gesture model can be estimated. πi = γ1 (i) T −1

αij =

ξt (i, j)

t=1 T −1 t=1

(18) γt (i)

T

bij (k) =

(17)

t=1,s.t.yt =k T t=1

γt (j) (19)

γt (j)

Given the observation sequence [Y], the HMM model λ can be calculated using the forward variable αt (i) and backward variable βt (i) with the following equating (20). This means that a symbol chain, namely an input image is recognized as a model which has the maximum value of equation (20). P (Y |λi ) = αt (i)aij bj (yt+1 )βt+1 (j) (20) i

5

j

Experimental Result

For the training and evaluation of the algorithm, we construct an image database about different gestures. The resolution of the image is 320 240 in pixels and capturing speed is about 10 frames per a second. We recorded six kinds of gesture image for ten different persons. We made an experiment for recognizing gestures with test images. In the experiment an APM of 1510 in resolution was utilized, so that 450-dimensional feature vector was extracted for every frame. We used only five eigenvectors,


9

Table 2. Recognition rate for the test image Gesture

Correct(%) InCorrect(%)

Normal I love you Good bye Stop Come here Point

98 95 85 80 85 100

Come here(2) Normal (5) Normal (10), Come here (5) Come here (20) Normal (15) -

Average

90.5

-

and this low-dimensional vector can represent the whole gesture space well. We used six different HMMs for the six different gestures. The HMM parameters were estimated with the Forward-backward algorithm under assuming that each HMM had 6 states. ”Good bye” and ”Stop” gestures have recognition rates that are lower than the average rate respectively. We suppose that the gesture contain little motion or 3D information comparing to other gestures.

6

Conclusion

In this paper, we describe the gesture interface method using context awareness. We define five different states of behavior. In the training process, system can be established the probabilistic model from the pre-defined image sequences. And we can recognize the context of gestures and catch the intention of user by analyzing the state transition model. Also, we can apply the different recognition algorithms to the system according to the context information. And one novel gesture recognition method using 2D shape and 3D depth information simultaneously is proposed. The algorithm works very well since it is robust to shape and depth variation. There are many gesture recognition algorithms using visual information. However, most algorithms work under many restricted conditions. To eliminate such constraints, we have proposed Active Plane Model (APM) to recognize human gesture. This model comes from structured model approach instead of geometric feature model approach using edge or corners. Therefore, we can make a general model for recognition of human gesture in real world. But, there are some problems in the case of being multiple persons in image because the system can not track user’s head and hands regions. And motion can be occurred regardless of user’s intention. However, efficient tracking algorithm can solve these problems, and selection algorithm of recognition target can improve the system. If these problems are solved, proposed algorithm can be used for controlling intelligent home appliances or humanoid robot more effectively.

Acknowledgment This research has been supported by research funding of ”Development of humanoid technology based on network, KIST, Korea, and Culture Technology Research Institute, Chonnam National University, Korea.

10

S.-J. Hong et al.

References [1] Xia Liu, Kikuo Fujimura, ”Hand gesture Recognition using depth data”, AFGR, 2004. Proceeding 6th IEEE International Conference on, May 2004 [2] F. J. Huang and T. Chen, ”Tracking of Multiple Faces for Human-Computer Interfaces and Virtual Environments”, IEEE Intl. Conf. on Multimedia and Expo., July 2000 [3] Baihua Li; Holstein, H. ”Articulated point pattern matching in optical motion capture systems”, QinggangMeng; Control, Automation, Robotics and Vision, 2002. ICARCV 2002. 7th International Conference on , Volume: 1 , 2-5 Dec. 2002 [4] Yu Su; Allen, C.R.; Geng, D.; Burn, D.; Brechany,U.; Bell, G.D.; Rowland, R., ”3D motion system (datagloves):application for Parkinson’s disease”,Instrumentation and Measurement, IEEE Transactionson , Volume: 52 , Issue: 3 , June 2003 [5] James Davis, ”Recognizing Movement using Motion Histograms”, MIT Media Lab. Technical Report No. 487,March 1999 [6] Ross Cutler, Matthew Turk, ”View-based Interpretation of Real-time Optical Flow for Gesture Recognition”, Third IEEE International Conf. on Automatic Face and Gesture Recognition, 1998 [7] Ho-Sub Yoon; Byung-Woo Min; Jung Soh; Young-Iae Bae; Hyun Seung Yang, ”Human computer interface for gesture-based editing system”, Image Analysis and Processing, pp. 969 - 974, 1999

Evolving Creatures in Virtual Ecosystems Nicolas Lassabe, Herve Luga, and Yves Duthen Laboratoire IRIT-UT1, Université des Sciences Sociales (UT1) - Manufacture de Tabacs - 21, Allée de Brienne - 31000 Toulouse, France Abstract. In this article, we introduce the first steps of our 3D ecosystem project. We present the latest research on morphology, behavior and virtual environments of evolving creatures and we try to investigate future possibilities for improving more complex lifeforms in dynamic environment. We propose our model of an ecosystem which uses a new type of controller for evolving creatures based on classifiers. Our model comprises three different lifeforms: insects, plants and creatures based on a genetic coding and evolving by evolutionary methods. In the future, we intend to add more different lifeforms and more interactions. To focus on this aspect, we have chosen Breve an existing physic engine centered on artificial life simulation. In this article, we show the results of evolving creatures being able to crawl, to walk and to climb stairs.

1

Introduction

In this article, we will present the interests of combining different creatures in a complex environment. During the last years, many articles have been released about evolving creatures [4,10,13,16,19,20,24,25,26,27]. They often focus on the morphology and behavior of creatures without taking into account the complexity of the environment. In these cases creatures evolve without interaction between them, to optimize a preset fitness. Conversely, some simulations propose to model more complex environments with simpler creatures [32,33]. It is just recently that works presenting evolving creatures in ecosystem have been published [6]. In this article, we propose an architecture for a virtual ecosystem allowing the conjoint evolution of different kinds of creatures with natural selection. We will present works in progress and a new type of controller based on the classifiers. We will try to define what kind of interactions between the creatures allow the emergence of interesting adaptation strategies. To allow for the emergence of prey and predator like behaviors, species are differentiated by characteristics such as color, which control attraction and repulsion mechanisms within the ecosystem. Defensive mechanisms such as the ability for species to become poisonous to other species and corellation between color and poisonous nature have also been simulated to establish food chains. Such complex ecosystems will also enable us to study extinction and specification processes to which the species are subject.

2

Why Ecosystem?

The realization of an ecosystem can allow the generation of lifeforms able to adapt to a dynamic environment, the specification of species and the emergence Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 11–20, 2006. c Springer-Verlag Berlin Heidelberg 2006

12

N. Lassabe, H. Luga, and Y. Duthen

of life cycles. The interests of simulating such virtual ecosystems are numerous. In the field of biology, it could lead to a better understanding of the evolution and extinctions of species as well as of some mechanisms of life [1]. The evolution of morphology and behavior in relation to a specific environment has an interest in the design of robots and of their controllers [12,19,11]. A complex environment should generate adapted survival strategies during an evolution. Interesting questions are whether different evolution scenarios can emerge from an identical initial environment and whether the environment complexity is a relevant criteria in the emergence of different survival strategies [2,22,23]. If that were the case, how does one evaluate complexity? What are the elements necessary for the emergence of complex life forms and behaviors? Is the use of a complex physics one of these elements or can biological complexity be supported by simpler physical models ? What is the importance of the interactions ? These are open questions which may be answered through experimentation with artificial ecosystems.

3

Related Work

There are many existing solutions for the representation of morphologies and controllers. Karl Sim’s creatures certainly remain the most evolved [25,26]. He has realized evolutions bound by very few constraints and obtained a large diversity of creatures giving birth to crawling and swimming behaviors. Thereafter many works tried to reproduce these results, like the L-System creatures of Hornby [13] and Lipson’s Golem project [19] with different approaches for the representation of morphologies and controllers. Their goals being to optimize 3D creatures in relation to a preset fitness inside a physics engine. Another approach to obtain more complex behavior is to optimize a set of 2D creatures in a virtual pool by means of natural selection [32]. Each creature has a quantity of energy and tries to survive looking for food in the environment. The advantage of this approach is that is possible to have many creatures. Komosinski tries to do the same thing in a 3D environment [16]. Recently, the project EvolGL [6] proposes to simulate a virtual world with worms. The evolving worms belong to herbivorous, carnivorous and omnivorous species and develop different strategies of survival. These works show that it is possible to distinguish two ways to optimize complex behavior evolving creatures: with a preset fitness or by means of natural selection. The works with a preset fitness continue to be improved with recents results like evolving flying creature [24] and evolving creatures for combat [20]. The works based on natural selection are becoming achievable and will certainly give more adaptive solutions and perspectives.

4 4.1

Proposition Introduction

In this article, we propose a model of an ecosystem to evolve virtual lifeforms. The goal of our model is to provide more adapted and more complex strategies of

Evolving Creatures in Virtual Ecosystems

13

Food

egg

Reproduction Reproduction

Food

Insect

od Fo

d

Fo od

Reproduction

Plant er

y

En

erg

En

gy

granulate

Creature

Fo o

Food

Food

light

water

Food Food Lifeform

grain

energy

Reproduction Energy

Fig. 1. Food chains and a part of the interactions between the elements of the environment

survival with a high interactivity between the different lifeforms. We do not want to focus on a specific representation of morphology or controller. The ecosystem must be capable of easily supporting different kind of lifeforms and manage their interactions. We would also like to allow the development of prey and predator strategies to obtain a complete food chain (Fig. 1). In this model, every lifeform has a quantity of energy provided by light or by food consumption. A creature dies when it does not have any energy left or when it is eaten by an another creature. 4.2

The Diet

The diet of a creature is coded by one its genes. Links in the food chain (Fig. 1) determine what a creature is allowed to eat : eggs, insects, plants, granulates, and another creatures like itself. The creature can eat five different kinds of foods. The links can be encoded in one of the creature’s genes composed of five bits

14


each representing one of the mentioned links. If a bit is set to 1, the creature is allowed to eat the corresponding food. For example the gene 01010, allows a creature to eat only insects and granulates. A creature is not allowed to eat another bigger than itself. This interaction is not sufficient to enable the development of complex behaviors. Therefore we defined for each creature an attractive color define by another gene, a repulsive color and of course its own color. The combination of these four genes will allow the development of more complex strategies. A predator’s color of attraction will evolve toward the color of other creatures which it can eat. One might think that the creatures will change colors very quickly from one generation to another leading to an unstable balance. However this does not happen because the successive crossings within a species guaranty a certain uniformity of the individuals within the species and an inertia making species variations slow and continuous. Another criteria differentiating species is their size. The reproduction process takes more time for the big creature than for the small ones. The strategy for a small species could be to develop a large population because they are more vulnerable. 4.3

The Concept of Poison

We also propose to introduce the concept of poison in our ecosystem. It could be a useful strategy of defense for the plants and the little creatures. The poison is represented by a gene of eight bits. The ability for creatures to eat another without being poisoned is based on whether its same eight bits are complementary. For example if the poison gene of a species is 0101110101, the complementary gene allowing for unharmful consumption will be 1010001010. We use the # character coding for both 0 and 1. The gene ########## protects again all the poisons but on the other hand, the creature’s poison is inoffensive. When a creature eat another one, it gains energy due to food consumption but loses energy due to poisoning. To know the quantity of energy lost due to poisoning, we calculate what percentage of bits do not have the good complementary value. If none do, the eating creature dies. If it is half complementary, the creature loses fifty percent of its energy. 4.4

The Energy

The energy is necessary for the survival of all lifeforms. The creatures use it for their movements. If their movements are not effective, the creatures will not find food easily. The quantity of energy spent for movement depends on the size of the creature. For the plants, the energy is important to produce more seeds and have better chances of reproduction. The energy allows the plant to grow faster. A plant’s energy collection is made by its leaves which collect the light transforming it in energy. If a plant grows faster, it needs more water. One of the strategies for the plants will be to find a good compromise between the number of leafs, the number of seeds produced and the quantity of water necessary for its survival.


4.5

15

Conclusion

With the concept of poison and attractive colors, we have the basis for a very complex environment. It could be interesting to study the evolution of the number of the species, their extinction and the evolution of their population sizes. If the process becomes stable, we will obtain ecosystem. An interesting observation could be to see if the same strategies appear from different simulations with different initializations. We can ask ourselves if slightly different initializations lead to different evolution scenarios. If the simulations are completely different after many generations with the same random initialization for the crossovers and the mutations, we can say that this environment is chaos determinist because the laws governing it are determinist. It could be also interesting to measure the complexity of each creature and measure the degree of complexity necessary to generate complex autoorganization. If we do not obtain any auto-organization, we must find what are the missing criteria that prevent it to appear.

5 5.1

Environment and Architecture Breve

The implementation of a physics engine is a difficult task requiring a full year of work. Our choice was therefore to use Breve [15], an engine based on ODE (Open Dynamic Engine), that is well suited for artificial life simulation. Our program is written in C++ and inserted in Breve in the manner of plugins. This way our program is not dependent of Breve and can be used like a library with another physics engine. Moreover Breve already implements a lot of useful functionalities like the quantity of light perceived by an object. 5.2

Morphology

There are three different lifeforms with a different representation for each (Fig. 1). The insect have a simple morphology, their characteristics which are subject to evolution are their color and their ability to fly or not. The plants use an Lsystem [18] representation (Fig. 2). Every plant has a number of leaves and seeds defined by its genotype and its environment. The creatures use a different morphology based on graphs like Karl Sims’ creatures [25,26]. The morphology is not constant, we want to keep the ability of generating various lifeform inside the ecosystem. The morphology of our creatures (Fig. 3) is satisfactory for the moment, but we plan to broaden the types of joints between blocks and types of blocks defining the creatures’ morphologies. 5.3

Controller

There are different controllers for each kind of lifeform. The insects have a preset behavior. They are attracted by some colors defined in their genes and flee some others. This way we hope to allow for the development of predator and prey

16


Fig. 2. Evolving trees represented by l-lsystem

Fig. 3. Evolving creatures walking and climbing a stair

behaviors. For the moment, the plants and the trees do not have any behavior but we want to allow them to orient their leafs in further developments. For the creature, the controller use a classifier system. The controller needs to activate the articulations correctly (junction between the blocs). The signal to do this need to be continuous and must vary in time according to incoming sensors reception of information. This signal is generally a sine wave or a composition of periodical functions. It is imaginable to compose a signal from some preset patterns (Fig. 3). We have tried to do this in our work. The main problem is to conserve a continuity of the signal, while selecting the next adapted pattern. We


17

start by defining a data base of random patterns. We have used one thousand generated patterns. The conditions for two appended patterns to compose a signal are the following: – The patterns need to be defined between [-1,1] and their minimum and maximum too. – If P1 and P2 are two patterns and if P2 is append to P1 it is necessary that: P1 (1) − P2 (−1) < ε Here ε = 0.01 – The classifiers control the selection of patterns. The entries of the classifiers which define the information incoming from the environment are the sensors of the creature. According to the sensors’ state, a set of rules pointing to patterns in the data base is triggered. The composed signal is used by the effectors controlling the muscles of the creature. If the resulting action is satisfactory, the rules used to generate that action are rewarded, else they are penalized. The advantages of this method lie in the fact that it is easy to enrich the used data base and obtain signals more complex than those obtained by simple primitive signal composition.

Fig. 4. Signal for a controller

6

Implementation

For the moment, the creatures and a good part of plants are implemented. An evolution of the creatures with a preset fitness based on the traversed distance has given some good results. For the final simulation, we will use Breve network implementation to deport all the calculations on different machines. The complete simulation will require a lot of computation and won’t be computed on a single machine. Every lifeform is implemented in C++ in different plugins and these plugins are interfaced with Breve. We try to maintain a great facility in adding different lifeforms in order to get the broadest, most various and interesting results possible. We would like also to keep the possibility to use the plugings with another engine.

18

7


Results

In the experiments, we use a population of 100 individuals (creature) and the tournament selection is 3. The fitness maximizes the walked distance by a creature. After 100 generation and around 3h of computed time, the best creature is been able to walk, crawl or climb stairs 1 .

8

Perspectives

The next steps for our project will be to implement and improve all the interactions between the different life forms, and also to confront the classifiers with more complex situations to see their adaptation capacity. Introducing an evolving language between the creatures could be interesting and allow a significant increase in the number of evolving strategies. It could be interesting to test the same ecosystem with a simple 2D representation and see what are the difference of results obtained. Concerning the controllers used, we plan to use the modular networks already tested in our chess engine to evaluate chess positions [3].

9

Conclusion

It is becoming possible to evolve a colony of evolving virtual creatures in a complex ecosystem with physics. The implementation of these ecosystems becomes essential to encourage the development of some adaptation strategies and to obtain more complex behaviors which were not possible in simple worlds. It will also allow to regroup evolving simulations, extinction simulations and others inside one application and therefore allow to observe the relations between them. We also show that it is possible to use classifiers as controllers and that this may in certain cases be a good alternative to the use of neural networks.

References 1. Abramson, G. (1997). Ecological model of extinctions Phys. Rev. E. 55, p 785. 2. Adami, C., Ofria, C., Collier, T. C. (2000). Evolution of Biological Complexity Proc. Nat. Acad. Sci (USA) 97. 3. Autones, M., Beck, A., Camacho, P., Lassabe, N., Luga, H., Scharffe, F. (2004). Evaluation of Chess Position by Modular Neural Network Generated by Genetic Algorithm. EuroGP 2004: 1-10. 4. Bongard, J. and Paul, C. (2000). Investigating Morphological Symmetry and Locomotive Efficiency using Virtual Embodied Evolution, From Animals to Animats: The Sixth International Conference on the Simulation of Adaptive Behaviour. 5. Bongard, J. and Pfeifer, R. (2001). Evolving Complete Agents using Artificial Ontogeny. 1

A video of a part of the best creatures is on the Nicolas Lassabe’s home page: http://www.irit.fr/∼Nicolas.Lassabe/videos/icat2006.mov


19

6. Carbajal, S. G., Moran, M. B. and Martinez, F. G. (2004). EvolGL: Life in a Pond. Proc. of Artificial Life XI, ed. by Pollack, J., Bedau, M., Husbands, P., Ikegami, T. and Watson, R. A., p75-80. 7. Cliff, D. and Miller, G. F. (1996). Co-Evolution Of Pursuit And Evasion II Simulation Methods And Results, From animals to animats 4. 8. Conradie, A. and Miikkulainen, R. and Aldrich, C. (2002). Adaptive Control utilising Neural Swarming In Proceedings of the Genetic and Evolutionary Computation Conference. 9. Dellaert, F. and Beer, R. D., (1996). A Developmental Model for the Evolution of Complete Autonomous Agents. A developmental model for the evolution of complete autonomous agents. In [14], pages 393–401. 10. Eggenberger, P. (1997) Evolving morphologies of simulated 3d organisms based on differential gene expression, Evolving morphologies of simulated 3d organisms based on differential gene expression. In [10], page in press 11. Floreano, D. and Nolfi, S. (1999). Learning and Evolution. Autonomous Robots, 7(1), 89-113. 12. Floreano, D. and Mondada, F. (1994). Automatic Creation of an Autonomous Agent : Genetic Evolution of a Neural Network Driven Robot, From Animals to Animats 3. Proceedings of the Third International Conference on Simulation of Adaptive Behavior. 13. Hornby, G.S. and Jordan B. Pollack (2001) Evolving L-Systems To Generate Virtual Creatures 14. Hornby, G. S. and Pollack, J. B. (2002). Creating High-Level Components with a Generative Representation for Body-Brain Evolution. Artificial Life, 8:3. 15. Klein, J. (2002). Breve: a 3D simulation environment for the simulation of decentralized systems and artificial life. Proceedings of Artificial Life VIII, the 8th International Conference on the Simulation and Synthesis of Living Systems. The MIT Press. 16. Komosinski, M., (2000). The world of Framsticks: simulation, evolution, interaction, In: Proceedings of 2nd International Conference on Virtual Worlds, 214-224. 17. Koza, J.R. (1992). Genetic Programming. MIT Press. 18. Lindenmayer, A. (1968). Mathematical models for celluar interaction and development. Parts I and II Journal of theoritical biology. 18 280–315 19. Lipson, H. and Pollack J. B., (2000). Automatic Design and Manufacture of Artificial Lifeforms. Nature 406, pp. 974-978. 20. O’Kellym, M. J. T. and Hsiao, K. (2004). Evolving Simulated Mutually Perceptive Creatures for Combat. Proc. of Artificial Life XI, ed. by Pollack, J., Bedau, M., Husbands, P., Ikegami, T. and Watson, R. A., p111-118. 21. Funes, P. and Pollack, J. (1997). Computer Evolution of Buildable Object. Fourth European Conf. on Artificial Life, 358–367. 22. Seth, A.K. and Edelman, G.M. (2004). Environment and behavior influence the complexity of evolved neural networks. Adaptive Behavior, 12(1):5-21. 23. Seth, A. K. (1998). The Evolution of Complexity and the Value of Variability. Proc. of Artificial Life VI, ed. by Adami, C., Belew, R., Kitano, H., and Taylor, C. pp 209-218. 24. Shim, Y. S., Kim, S. J., and Kim, C. H., (2004). Evolving Flying Creatures with Path Following Behaviour. ALIFE IX: In Proceedings of 9th International Conference on the Simulation and Synthesis of Living Systems. pp.125-132. 25. Sims, K., (1994a). Evolving 3D morphology and behavior by competition. Proc. of Artificial Life IV, ed. by R. Brooks and P. Maes, pp. 28-39.

20


26. Sims, K. (1994b). Evolving Virtual Creatures. Proc. SIGGRAPH ‘94, pp. 15-22. 27. Taylor, T. and Massey, C. (2001). Recent Developments in the Evolution of Morphologies and Controllers for Physically Simulated Creatures, 28. Teo, J. and Abbass, H.A. (2002). Trading-Off Mind Complexity and Locomotion in a Physically Simulated Quadruped, Proceedings of the 4th Asia-Pacific Conference on Simulated Evolution And Learning, vol 2,776-780. 29. Terzopoulos, D., Tu, X. and Grzeszczuk, R. (1994). Artificial fishes: Autonomous locomotion, perception, behavior, and learning in a simulated physical world. Artificial Life, vol 1, 327-351. 30. Terzopoulos, D. and Grzeszczuk, R. (1995), Automated Learning of MuscleActuated Locomotion Through Control Abstraction, Computer Graphics, vol 29,63–70. 31. Ventrella, J. (1994). Explorations in the Emergence of Morphology and Locomotion Behavior in Animated Characters. Proceedings of the 4th International Workshop on the Synthesis and Simulation of Living Systems Artif icialLif eIV , pp. 436-441, MIT Press. 32. Ventrella, J. (1998a). Designing Emergence in Animated Artificial Life Worlds. Proceedings of the 1st International Conference on Virtual Worlds (VW-98), LNAI, Vol. 1434, pp. 143-155. 33. Ventrella, J. (1998b). Attractiveness vs. Efficiency: How Mate Preference Affects Locomotion in the Evolution of Artificial Swimming Organisms Proceedings of the 6th International Conference on Artificial Life (ALIFE-98), pp. 178-188, MIT Press. 34. Zaera, N., Cliff, D. and Bruten, J., (1996). Evolving collective behaviours in synthetic fish. From Animals to Animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior, p. 635-644.

The Application of Affective Virtual Human in Intelligent Web Tutoring System Huiling Peng Department of Information & Technology, Central China Normal University Wuhan, 430079, P.R. China [email protected] Abstract. Aiming at the defects of the difficult emotional interaction of traditional Intelligent Web Tutoring System, based on the theory of artificial emotion of virtual human, an intelligent affective Web Tutoring System was presented in the paper. Second, the key module of the system was designed and the modeling on the Affective Virtual Human was carried out. Finally, the simulation command on it was made. The result showed this model coinciding with the emotion of human behaviors. Keywords: Artificial Emotion, Virtual Human, Affective Computing, Intelligent Web Tutoring System.

1 Introduction Traditional Intelligent Web Tutoring System can realize the intellectualized and personalized teaching on the cognitive level. Although it has this strongpoint, this system cannot apperceive students’ emotion so as to fail to realize emotionally personalized education and timely communication. In fact, a real personalized teaching system should not only possess cognitive personalized teaching function, but also emotional personalized teaching function. So, how to set up an Intelligent Web Tutoring System with emotional teaching function to realize personalized internet teaching of the interaction of cognition and emotion is very significant in the field of study of Intelligent Web Tutoring System. Based on the theory of Affective Computing of Virtual Human, the application of expression and gesture identification bring forward an emotional Intelligent Web Tutoring System, which can seize and identify every student’s study expression, make emotional feedback according to different student’s study emotion and give emotional guidance so as to realize personalized teaching of the mutual correspondence of cognition and emotion. Meanwhile, the key technology of this system--the model of Affective Virtual Human is accomplished.

2 The Integrated Structure and Function of Each Model of Intelligent Web Tutoring System for Affective Virtual Human This system adopts B/S structure. A Video input device (camera) will be installed at each student’s side to obtain video image of student’s study expression. Students and Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 21 – 27, 2006. © Springer-Verlag Berlin Heidelberg 2006

22

H. Peng

teachers use The Tutoring System through user side’s browser. The Tutoring System can obtain student’s study information on cognition and emotion, through analysis of this information to instruct student’s study process. The communication among students, teachers and the server of The Tutoring System proceed through Web Service. Its web system structure as Figure 1: Video input device

...

Video input device

Student 1

...

Student n

Teacher

Internet

Web Server

Tutoring System

Fig. 1. Web system structure

Compared with traditional Intelligent Web Tutoring System, this system adds emotional model, including Affective Virtual Human and emotional instruction as well as video interface in the interaction modeling. Its web system structure as Figure 2 and the function of each model as follows.

Fig. 2. System structure

The Application of Affective Virtual Human in Intelligent Web Tutoring System

23

Student Model: Keep record of initial knowledge level and study capability of each student and keep record of personality. It can ask and answer the history questions between students and system and make dynamic adjustment on each student’s study progress circumstance. Tutoring Model: Make intellectualized tutoring decision according to Student Model, judge student’s study effect, put forward improvement method and advises etc. Repository: Storage the due teaching subject domain and tutoring knowledge. Interaction Model: Cognitive interface can understand natural language; realize the dialogue between human and machine in the common sense. Video interface can obtain student’s study image. Affective Model: Make affective identification and judgment according to the obtained student’s study image by video interface and make feedback for student’s emotion; meanwhile, make affective tutoring analysis according to student’s study emotion, cognitive and personalized feature in the Student Model so as to provide personalized affective tutoring instruction.

3 The Design of the Key Module of the System and the Key Technology 3.1 The Design of Affective Model 3.1.1 The Design of Affective Virtual Human Avatar is one type of virtual human, called as the embodiment of the real human, whose behaviors are controlled by real human. Using Avatar virtual human to denote student’s virtual role in the web world, it can realize a personalized writing, voice and action， and can also talk with human beings through sound identification system on the terminal of the teacher, it will show student’s Avatar. First of all, video input devices Seize Video image of the student. Then the emotion was judged. Then the Affective Virtual Human recognizes the emotion and expresses its feeling. As a result, the teacher and the student can communicate with each other well. Figure 3 is the structure of Affective Virtual Human.

[

Fig. 3. The structure of Affective Virtual Human

24

H. Peng

3.1.2 The Design of Affective Tutoring Model The affective tutoring model make affective tutoring analysis according to student’s study emotion, cognitive and personalized feature in the Student Model and Mood teaching principle in Education psychology, and provide personalized affective tutoring instruction so as to adjust student’s mood for better study. 3.2 The Affective Virtual Human Modeling The Affective Virtual Human modeling is the key technology, so, this paper will put forward a method of Affective Virtual Human modeling. Detail introduction will be followed below. 3.2.1 The Affective Virtual Human The Affective Virtual Human is a kind of virtual human with particular personality and capability of affective interaction (emotion recognition and expression), possessing a particular artificial emotion mathematic model, emotion recognition and expression way. It is the denotation of geometry and behaviors characteristics in virtual environment, research contents of multi-function apperception and affective computing and the concrete application of artificial psychology in the field of virtual reality. 3.2.2 The Personality and Emotion Model of the Affective Virtual Human (1) The personality and emotion of the Affective Virtual Human Every individual is an incessant entity. So, when we refer to the individual, we always indicate the comparative time t. It starts to exist on the moment t=0. Lt denotes an individual on the moment t. An individual has the state of personality and emotion. The personality is invariable, which has an initiate value on the moment t=0. The emotion is variable, which has an initiate value of zero on the moment t=0. According to OCEAN model and PEN model, we can suppose that an individual is an n-dimension vector. Every dimension has a value between [0 1]. The meanings of 0 and 1 are as follows: 0 denotes nonexistence of a dimension; 1 denotes the dimension having the biggest proportion. The denotation of personality is as follows:

，

P T = [ μ1 K μ n ], ∀ ∈ [0,1] : μ i ∈ [0,1]

(1)

Emotion state, which has a similar structure with personality, is variable with time and is a series of emotion which has emotion intensity. et , which is the denotation of emotion state, is a n dimension vector. Every dimension has a value between [0， 1]. The meanings of 0 and 1 are as follows: 0 denotes nonexistence of a dimension; 1 denotes the max. The denotation of the vector is as follows:

⎧[δ kδ ], ∀i ∈ [0, m] : δ i ∈ [0,1] if t > 0 e Tt = ⎨ 1 m if t = 0 ⎩0

(2)

ηt denotes the past emotion states, including all the emotion states from the i=0 to i=t, as follows:


25

η t =< e 0 , e1 , K, e t >

(3)

Based on the above model, personality Lt is defined as a 3D function of (P， mt， et). p denotes personality, mt denotes the mood and et denotes emotion state on the moment t. Emotion is not only One-dimensional, but also multi-dimensional. Every value of emotion is between [-1， 1]. Supposing emotion is a k-dimensional vector, it can be denoted as follows: ⎧[θ Kθ ]， ∀i ∈ [ −1, k ] : θi ∈ [-1,1] if t > 0 (4) m Tt = ⎨ 1 k if t = 0 ⎩0 λi denotes the past emotion state, including all the moods from the moment i=0 to i=t, as follows:

λt =< m 0 , m1 , K , m t >

(5)

(2) Emotion transferring For each emotion, the definition of emotional information is the ideal emotional intensity changes. Emotional information has a value between [0 1], which represented by the vector of ε, including changes of each m-dimension emotional intensity.

，

ε T = [δ 1 K δ m ]， ∀i ∈ [1, m] : δ i ∈ [0,1]

(6)

There are two steps to transfer emotion. The first step is to change the mood, and the second is to change emotional state. The functions of χm (P， ηt， λt， ε) and Vm (P， ηt， λt) denote mood changes. χm (P， ηt， λt， ε) is decided by p, ηt, λt and ε. Vm (P， ηt， λt) denotes internal changes of emotion (Enhancement or diminishment). New emotion is as follows:

m t +1 = m t + χ m (p,η t , λ t , ε ) + Vm (P,η t , λt )

，，

(7)

，

The functions of χe (P ηt ε) and Ve (P ηt) denote mood state changes. χm (P， ηt， λt， ε) is decided by p, ηt and ε. Vm (P， ηt) denotes internal changes of emotion (Enhancement or diminishment). New emotion state is as follows:

e t +1 = e t + χ e (P,η t , ε ) + V(P,η t )

，，，

(8)

Emotion is decided by external function χe’ (P ηt λt+1 ε) and internal function Ve (P ηt λt+1). χe’ (P ηt λt+1 ε) refers to the new and the past mood. Ve’ (P, ηt λt+1) also refers to the new mood. Therefore, the new emotion transferring is as follows: ’

，，

，，，

e 't +1 = e t + χ e' (P,η t +1, ε ) + Ve' (P,η t , λ t +1 )

，

(9)

4 Computer Simulation The restriction condition that the experiment follows is that emotional changes of Affective Virtual Human begin with neuter character, then delay and decay for a certain extent, and finally go back to neuter character.

26

H. Peng

4.1 Feedback for Different Stimulus on the Same Model Parameters In the initial emotional state [0 0 0], happiness, sorrow, surprise and neuter character are inputted in the experiment. The feedback is showed by figure 4:

stimulus happiness sorrow surprise neuter character

mapping in personality space optimism Pessimism optimism the usual mentality

Fig. 4. Feedback for different stimulus

4.2 Feedback for the Same Stimulus on Different Personality Model Parameters In order to survey the feedback for the same stimulus on different personality model parameters, we select three different personalities: extroversion, introversion and the character between extroversion and introversion, and make them act on the same external stimulus: happiness. The feedback is showed as figure 5: personality extroversion diffidence the character between extroversion and diffidence

the probability of emotional changes bigness smallness Somewhat between above

Fig. 5. Different feedbacks for the same stimulus

5 Conclusion An intelligent affective Web Tutoring System is designed in this paper, which concerned student’s affective changes. Then affective virtual human modeling is made. Finally a simulation is made on it. Testing results indicate that this model can realize emotional interaction of Affective Virtual Human. Since the emotional feedback of this model accord with the rule of human behaviors, so it’s practicable and possesses a feature of flexible formation of different personalities. Due to the short notice, many problems maybe exist in this system and more improvement and perfect are still required. Emotional modeling also needs further improvement and refinement as the development of artificial emotion.


27

References 1. Picard R. Affective Computing. Cambridge [M] . MA: MIT Press , 1997. 2. Yang Guoliang, Wang Zhiliang. A survey of affective modeling [J].Techniques of Automation and Applications,2004,23 (11). 3. Wilson I. Believable Autonomous Characters [D] . Dept of Artificial Intelligence, Univ of Westminster ,1996. 4. Goleman D. Emotional Intelligence [M] . [ s. l. ] :Bantam Books , 1995. 5. Xue W M,Wang ZL , Wei Z H. A new method for simulating human emotions [J ] . Journal of University of Science and Technology: Beijing, 2003, 10(2):72 - 74. 6. Damasio A R. Descartes’Error: Emotion, reason and human brain [M]. New York: GrossetPPutnam Press, 1994. 7. Picard, Rosalind W. Toward computers that recognize and respond to user emotions [J]. IBM Systems Journal, 2001, 39(3 &4):705.

Extracting Behavior Knowledge and Modeling Based on Virtual Agricultural Mobile Robot Xiangjun Zou1,2, Jun Lu1, Lufeng Luo1, Xiwen Luo1, and Yanqiong Zhou2 1

The College of Engineering, South China Agriculture University, Guangzhou 510642 2 Nanhua University, Hengyang, Hunan, 421001, China

Abstract. Aiming at complexity, unknown and uncertainty of picking object of agricultural intelligence mobile robot, extracting behavior knowledge and modeling based on the robot was put forward to help them obtain information effectively during operation, thereby to make decision for their behaviors. Firstly, behavior was studied based on picking behavior of the robot in virtual environment. Propose a behavior and method of extracting knowledge in virtual environment those are based on the association rules and, classify and express the entities such as robots, fruit trees and litchi, etc. Secondly, knowledge bases and models were built for reasoning. Thirdly, put forward for the first time to behavior knowledge classifies based on rough sets systematically, and classify the behaviors into obstacle-avoidance, picking, reasoning and fusion behavior to reduce redundant knowledge. Finally, an example for reasoning and simulation of the behavior was given. It realized picking behavior by message and route mechanism. Keywords: Intelligence mobile robots, Behavior, Knowledge extraction, Uncertainty, Virtual environment.

1 Introduction Agricultural intelligence mobile robots are unable to obtain certain knowledge information and make decision effectively and accurately during operation due to complexities, unknown and uncertainties of agricultural environment and its operating object. Therefore, knowledge modeling, uncertain reasoning and fusion of agricultural intelligence mobile robots’ behavior have been the key technology and theory basis of simulation. Researches on agricultural picking robots have been lasted for over 30 years. American scholars, Schertz and Brown, propose for the first time to adopt robots to harvest fruits and vegetables. Then, researches are developed among countries such as America, Canada, Holland, Japan, and UK etc. Japanese invented robots to pick tomatoes, cucumbers and Spanish invented robots to pick oranges [1]. And mobile robots were used to survey the crop to obtain its growth knowledge [2]. There are few agricultural robots in China, but things will be better. Researches on agricultural robots and their basic theories are begun to be recognized. South China Agricultural University has developed an operation platform that has machine vision, GPS (Global positioning system) and navigation control system [3], which provides a Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 28 – 37, 2006. © Springer-Verlag Berlin Heidelberg 2006

Extracting Behavior Knowledge and Modeling

29

mobile platform for agricultural mobile robots. Nanjing Agricultural University has researched on identification of the multi-resolution route in machine vision navigation. They hold the opinion that how to syncretize the priori knowledge to improve the identification ability of machine vision’s mode is an important research content in this area [4]. Zhe Jiang University optimized the design of manipulator by optimizing the fruit picking manipulator and simulation experiment [5]. Now, the research of mobile robot concentrates its attention on vision, graphical processing, path selecting, motion of mechanism and simulation, and agronomists rarely research mobile robot’s behavior decision and knowledge modeling. But, behaviors of robot have been researched in the fields of industry and military. In the literature [6], the behavior control of robots which is based on the behavior control realized by artificial intelligence, was realized by the reasoning and fusion based on fuzzy rules and expert knowledge. In the literature [7], the process of uncertain knowledge and information is realized by the partial knowledge and the possibilistic logic approach. In the literature [8], as a robot, it has no a priori knowledge, has to explore a graph. So exploration method was studied on robot. An example for picking litchi and longan in the south of China is given in the paper. It studies knowledge extraction and modeling, uncertainty reasoning and fusion of agricultural mobile robot’s picking behaviors, presents the knowledge classification of robot’s behaviors based on rough sets, classifies behaviors of the robot into obstacle-avoidance behavior, picking behavior, knowledge reasoning behavior and knowledge fusion behavior. By the attribute reduction of knowledge, treating behaviors as a design problem and the modified evidence theory [9], the uncertainty of reasoning and simulation can be solved, and the optimal behavior decision is obtained by reasoning and fusion.

2 Picking Object and Behavior Characteristics of Mobile Robots 2.1 Picking Object The production of the litchi and the longan, which are specialties of the south of China, especially, the Guangdong accounts for about a third of that of the world. Generally, litchi trees are planted in gardens and on hillsides, and its fruits grow in Chains. Every year, in the season of picking, to complete the picking costs a great deal of manpower, its labor intensity is heavy and means of picking are rude, and mainly depends on people to climb the trees to pick litchi . The cost of picking accounts for about 50% to 70% of that of production. So, the agricultural robot which is a kind of automatic machines will take the place of handwork. 2.2 Behavior Characteristics of Agricultural Mobile Robots There are more uncertainties with behavior decision of agricultural mobile robots. For example, the position of picking object and the size of litchi are uncertain, flexible and damageable, and the irregularity of road scene, the weather and the environment are uncertain and beastly. At the same time, when the robot is on working, its object may be covered by leaves and shadow, and its knowledge obtaining, vision and behavior

30

X. Zou et al.

decision will be interfered by noise, so, there are more uncertainties with the robot and its behavior control and decision. Through knowledge modeling of complex environment and behaviors of robots, the complex and uncertain problems can be classified and simplified, and behavior decision and simulation of the robot in virtual environment (VE) can be realized.

3 The Knowledge Extraction and Modeling of Robot’s Picking Behavior How to use a laconic mean to realize knowledge modeling and extraction is the key and also the difficulty to knowledge modeling of the robot’s picking behavior in VE. The following analyzes the entities in VE and their behaviors at first, and then makes knowledge expression to them based on their association rules. 3.1

The Characteristics of the Entities and the Analysis of the Uncertain Problems

Picking behavior of the robot in VE is complex, the entities in this environment which can embody external characteristics of entities, are composed of some geometric entities and these entities such as ambient light and material and so on. Behaviors can embody both external characteristics and internal characteristics, and the state behavior that embodies external characteristics is the representation of the physical function of entities. The picking behavior can be classified into static entity and dynamic entity. The dynamic entity, which is important to the reasoning and decision making of picking, is a result of the entity’s state change. Entities in virtual environment are primarily mechanism of robot, transportation mechanism of fruit, transducer, pick-up head, fruit tree, and fruit in chains, ground, weed, road-block, clouds, fog, sunlight and rain, etc. In the paper, these entities are classified into robot system, road, obstacle and climate. The knowledge about picking behavior should include the common sense needed by behavior decision, the characteristics of the growth of biology, the fruit, the agroclimate and the geography. Generally, the knowledge about the static entities in virtual environment such as fruit tree, obstacle and climate, etc, is certain, but uncertain when the robot is on picking. For example, the size of the chain of fruit and the color are variable and sunlight varies randomly. So, there should be knowledge classification, expression and modeling to be done with knowledge extraction and robot can’t make behavior decision validly without the good judgment of the variety of scene knowledge. 3.2 The Extraction and Expression of the Association Rules In order to make reasoning of picking behavior easy and to simplify program structure, the association rules based on object is adopted. When to express behavior knowledge, the basic knowledge should be expressed by rules, the structure knowledge should be expressed by the object-oriented method, and then express the knowledge and structure knowledge about fact and rules in the form of example with the object-oriented method. The express method that is similar to man’s way of thinking is easy for description,


31

expression, abstraction, inheriting, organization and package. Next, the expression of robots’ behavior function and virtual scene knowledge is discussed and is mapped to relation database to be stored, and, knowledge base is built in order to make the call and the management easy and efficient. The association rules can make the asking of behavior knowledge of robot easy. If there is a rule set T {ti|i=1, 2… }, the association rules will be used to study the influence which one item of T puts on another item [10]. Take the virtual scene of the picking behavior of the robot for example to explain how to make knowledge extraction and expression. In the scene, there are entities such as fruit trees, ground, and robots, etc. The trees in a scene, which can be treated as a set, should include much information such as the tree’s name, position, height, kind, fruit’s size and color, etc. If we input the key words litchi, type and color to ask the kind of the litchi in certain garden, we can get the information that 85% of litchis are reddish Fei Zixiao. The constrained association rule with a high reliability can be obtained by the algorithm for mining association rules. The rule can be expressed as following: Identifiers 1, 2 Number attribute 1, attribute 2, attribute 3, and attribute 4, Conclusion, reliability. When to plan behaviors preliminary, the knowledge that can help robot to make plan decision can be obtained by the Association Rules. For example, the system will give us the information about the matching of picking, scene, fruit tree and climate when to plan behaviors preliminary in computer-human-interaction virtual system. If the sensors obtain the information that the litchi is reddish, the value of reliability that is 0.85 will lead the system to search out the knowledge about the picking object matching with the value in knowledge base. The common knowledge about picking behavior obtained from the knowledge base is as following:

ｎ

LZ=1 # litchi tree Varietal=2 # Fei Zixiao (a kind of litchi) Color=3 # reddish Mature=1 # maturity CF=0.95 # reliability The knowledge (including the position of manipulator) about the object of scene obtained by the sensor is as following: LZ=1 # litchi tree Varietal=2 # Fei Zixiao Color=3 # reddish High-Z=0.1 # the height of the object in direction Z High-X=0.1 # the distance of the object in direction X High-Y=0.1 #the distance of the object in direction Y Action=1 # take action CF=0.9 The upper two kinds of knowledge can be expressed in another form, as the following:

32

X. Zou et al.

IF {LZ (I) =1 and I Varietal=2 and Color=3 then Mature=1} IF {LZ (I) =1 and Color=3 and Mature=1 then Action=1} The knowledge for decision can be obtained by the association rules. The dynamic behavior of entities in VE is an expression of the function of the entity, a part of the function knowledge and expression of the structural state; so, the knowledge extraction of behavior knowledge is based on the expression of structure and function. 3.3 The Object Expression of Structure, Function and Behaviors of Virtual Robot Based on Rules The paragraph studies how to express objects, classes and their attributes, operation, methods and relations. The association chart expresses their structure knowledge and behavior function, and the object chart can express three kinds of relationships [10]. (1) Association and linkage: association and linkage can express the relations between objects, classes and object and class. All the linkages in association connect to the objects of the same class. (2) Summarization and inherit: The summarization expresses the inclusion relation between super class and subclass based on the view of super class. The inherit expresses the relation based on the share attribute, the operation principle, the view of subclass and that characterization and summarization are two different views of a same relation. (3) Polymerization: treating an object as certain modules, expressing the object as an assemblage of these modules, modeling each module and then assembling these modules according to each other’s relation. In the object model chart, the diamond represents the end point of polymerization relation. The structure and behavior function knowledge of robot expresses all kinds of association between entities such as the attributes of objects and classes and the operation by the hiberarchy of the chart and some symbols. An example of agricultural mobile robot for the expression of the knowledge model of the entities in VE is taken in the Fig 1. In Fig 1, B- function is Behavior function, P- index is Performance index, O-principle is Operating principle, B-rules is Behavior rules; A –control is Auto control, F- control is Function control, S- control is Simulation control; Ob-location is Object location, O-avoidance is Obstacle-avoidance, D-making is Decision-making, Pbehavior is Pick behavior.3-D is three-dimensional. 3.4 The Organization of the Behavior Design Knowledge Repository of Robots Behavior design knowledge repository provides knowledge storage, management and maintenance for behavior function, motion, poses, target identification, picking and obstacle-avoidance behavior. (1) The behavior knowledge object class mapping to the database table: Each kind of class maps to one table or many tables (a table also can be used in many classes). When simultaneously assigns the host key word, and supposes its value for none. Uses the SQL code to found and establishes the host key word index. Put the data type maps to the SQL level.


MRobot MRobot ID Name

The Robot

Behavior function

Function Function ID Name

Has

Track Robot Behavioral model Structure model Association rules

33

B-function P-index O-principle B-rules …

A -control F-control B -rules … S-control

Wheeled Robot Behavioral model Structure model

Constraint

i

Ob-location O-avoidance D-making P-behavior … S-control

Constraint relation

Constraint name Constraint type Robot Behavioral model Principle model Association rules

Robot Behavioral model Principle model Association rules

Problem solving () Problem analysis ()

1

Parts

n

Connecting CAD 3-D model

Connect – relation () Connect – type ()

Rules Connect-selection ()

Transfer mechanism

Manipulator

CAD

CAD

3-D

3-D

Model

Model

Rules

Rules

Virtual scene

Sensor CAD 3-D

…

CAD 3-D

Model

Model Rules

Rules

Connect - analysis ()

Parts Cutting mechanism Equation of motion Association rules … Simulation result

F Leadscrew Parameter … Rules

Standard

Cam Parameter Principle … Rules

˖Inherit class

T Screw thread Parameter Quantity Material … Code

Litchi tree Parameter Association rules … Simulation result

Litchi Variety Parameter … Rules

˖Polymerization

Fig. 1. The knowledge model of virtual mobile robot

(2) The association mapping to the table: Multi-association-to-multi-association maps to the tables, we need to satisfy the object’s association relations. The class which correlative with the behavior knowledge object and the linking attribute of main key word are all became the attribute of the table. We know that the connection between two objects requests two objects mutually understand, which by the association characteristics.

34

X. Zou et al.

(3) Summarization maps the table: Maps each subclass to a table, which causes the object marking to be preserved through using sharing ID. The logic relations of this expression are definitude. For example the MRobot’s (a mobile Robot) table is allowed to summarize to the table’s attribute names: MRobot ID, Track MRobot ID and Wheeled MRobot ID. (4) The polymerization mapping to the table according to the above mapping rule, we can establish the polymerization mapping to the table. (5) Mapping object structure to date table can put more object model’s rules and semantics to the database and make the knowledge inquiring of intelligence mobile robot, the gaining behavior reasoning and fusion and the realization of decision making become easy.

3 Behavior Reasoning and Decision Based on Rough Sets Theory Disposing the behavior problems of virtual robots as one kind of design problems, drawing up the design criterion, one solution based on the rough sets and the case was put forward to realize behavior design decision. Knowledge extraction based on rough sets theory and object classification have common characteristics on method, that is to say both of them make classification according to the views by which human see the things. Knowledge classification is necessary while extracting property. One method is that, first of all, classify a type of knowledge (such as experience analogy, etc.) According to the characteristic of virtual robot’s behavior attributes to obtain different division of the knowledge property, then classify again until the division meet the needs, and eventually, aggregate the classified information to obtain the property sets which we hope to extract. Instance of behavior simulation Instance of behavior

Instance of

New behavior

behavior knowledge

instance

classification Express behavior knowledge as characteristic property

Disposition of quantitative problem

Disposition of redundant knowledge

Main characteristics and index of property

Decision of behavior design

Fig. 2. The design process and searches of behavior instance based on rough sets


35

Behavior decision of the robot in complex environment is the decision knowledge based on uncertainty and multi-property. Analyzing the knowledge which has been classified, determining the degree of importance of characteristics, and establishing the decision-making table (including finding partial knowledge contrast), all of these, provide a basis to extract the key features of property, to build simple models of knowledge and reasoning models based on rough sets, thereby to achieve reasonable and certain design approach so as to fuse by similar matching and instance. Searches for the instance (in Fig 2) concludes the expression and conversion of the uncertain problems, the establishment of the decision table, the analysis of the importance of property and the establishment of index. Finally, make comprehensive evaluation by system similarities model and availability and solve the uncertain problem through the modified evidence theory to get the reasonable behavior decision.

4 Picking Behavior Realization of the Robot in VE Picking behavior realization of the robot is realized by message and route mechanism on the platform of EON Studio 5.0 and VC++. Human-computer Interaction is realized by sensor, event driver and route mechanism. For instance, Fig 3 is a part of route mechanism on behavior realization of robot in VE and Fig 4 is a 3D dynamic simulation of picking process of agricultural mobile robot in VE.

Fig. 3. The route mechanism of behavior control in virtual environment

In VE the elementary require of robot behavior is apperceiving environment by its sensors or effecting environment by operation and realizing interaction by communications. In software, sensors and data are defined as all kinds of equipments on VR such as data-gloves, keyboard mouse and so on. After functions are inputted, the inputting instructions from users are received, according to which the behavior is

36

X. Zou et al.

performed. In Eon Studio, the sensors have mostly Keyboard Sensor, Mouse Sensor, Box Sensor, Time Sensor and Click Sensor, etc, by which the system can receive inputting instructions and perform its behavior according to client's demands after the client send instructions. The node in the system is an object with functions and data, which has different types. Generally, the data information of node concludes two aspects, the filed and the event. The final realization of action and interaction could be achieved by the transmission of message, which is the bridge of communication.

Fig. 4. The dynamic simulation of robot's picking process in virtual environment

5 Conclusions The article studied how to extract the structure and the knowledge models of behavior function of robots, and established knowledge bases and models. For the first time proposed the behavior that was based on the association rules and the knowledge extraction of virtual scene, classified and expressed the entities such as robots, fruit trees and litchi, etc. adopted object-oriented method to express the basic knowledge, expressed the structure and behavior knowledge of robots based on the rough sets, treated the behavior problems of virtual robots as a design problems, developed design standards, solved the certain and uncertain problems of behavior decision, and realized the behavior decision by the rough sets and the case method. This paper took the behavior of agricultural mobile robots as an object to realize the knowledge modeling and simulation of the picking behavior of robot in VE.


37

Acknowledgments. This work was supported by (1).Natural Science Foundation of Guangdong, NO: 05006661. (2).Educational and Research Foundation of Hunan NO: 04B21. (3).the National Natural Science Foundation of Chine NO: 60574029. (4). the “211 Project” of South China Agriculture University.

References

，

[1] S. I. Cho S. J. Chang; Y. Y. Kim; K. J. An.Development of a Three-degrees-of-freedom Robot for harvesting Lettuce Using Machine Vision and Fuzzy logic Control [J]. Biosystems Engineering. 2002. 82 (2): 143–149. [2] Jun Qiao; Akira Sasao1; Sakae Shibusawa1 et al. Mapping Yield and Quality using the Mobile Fruit Grading Robot，Biosystems Engineering (2005) 90 (2), 135–142. [3] Lou Xi-wen;Qu Ying-gang;Zhao Zuo-xi, Research and development of intelligent flexible chassis for precision farming, TRANSACTIONS OF THE CHINESE SOCIETY OF AGRICULTURAL ENGINEERING, 2005. 21(2): 83-85.(in Chinese) [4] Zhou Jun;Ji Chang-ying, Multi-resolution Road Recognition for Vision Navigation, TRANSACTIONS OF THE CHINESE SOCIETY OF AGRICULTURAL MACHINERY, 120 124. (In Chinese) 2003.34 11 [5] Liang X-feng, Experiments of Optimization and Simulation on Kinematics of a Tomato Harvesting Manipulator, TRANSACTIONS OF THE CHINESE SOCIETY OF AGRICULTURAL MACHINERY, 2005.36(7): 97-101.(in Chinese) [6] Eugenio Aguirre, Antonio Gonz_alez ， Fuzzy behaviors for mobile robot navigation: Design, coordination and fusion [J]. International Journal of Approximate Reasoning (S0888-613X). 2000. 25: 255-289. [7] Claudio Sossai, Paolo Bison, Gaetano Chemello.Sequent calculus and data fusion [J]. Fuzzy Sets and Systems (S0165-0114).2001. 121: 371–395. [8] Pierre Fraigniaud, David Ilcinkas, Guy Peer, Graph exploration by a finite automaton [J]. Theoretical Computer Science (S0304-3975). 2005. 345: 331–344. [9] Zou Xiang-jun,Sun Jian,He Han-wu,Zheng De-tao,Zhang Ping, Research on Combination of Conflicting Evidences of Multi-Agent Decision in Virtual Environment, JOURNAL OF SYSTEM SIMULATION, 2006.18(4) :1010-1014(in Chinese) [10] Zou Xiang-jun, Snu Jian ,Gu Bang-jun. Knowledge information extracting and modeling of virtual product design based knowledge，Modern Manufacturing Engineering.8 (2006): 6-8 (in Chinese).

（）：－

Human Motion Interpolation Using Space-Time Registration* Zhihua Chen1,2, Lizhuang Ma2, Yan Gao2, and Xiaomao Wu2 1

East China University of Science & Technology 201512, Shanghai, China [email protected], [email protected] 2 Department of Computer Science & Engineering, Shanghai Jiao Tong University, 200030, Shanghai, China Abstract. Human motion interpolation plays an important role in virtual reality and human animation. In this paper, we propose a motion interpolation algorithm with space-time registration. Firstly, time registration based on motion cycles is used to seek the corresponding frames between input motions. Secondly, space registration based on analysis method is employed to align spatially the motion frames. Then the corresponding frames are interpolated according to the interpolation methods of transition and angles, respectively. Finally, we reconstruct the constraints of the resulting motion. Experimental results demonstrate that our algorithm succeeds in interpolating input motions to create new motions with new style.

1 Introduction Motion interpolation is an important technique for blending different motions to create a new motion of new style. For example, the walk motion and run motion of human can be used to produce the jog motion by means of this technique [1, 2]. Motion interpolation has been applied to many areas including virtual reality, computer animation, digital entertainment, human machine interaction, etc. In this paper, we study how to interpolate the parameters of human motion, and propose a new motion interpolation algorithm based on space-time registration. The algorithm consists of four steps, i.e., time registration, space registration, parameter interpolation and constraint reconstruction. Firstly, we utilize the coordination and periodicity of movements of human joints, construct time registration function that is monotone and consistently increase, and then align temporally input motions. Our method can avoid the error of ‘time reverse’. Secondly, we spatially align the input motions. Considering that human usually move on a plane surface, we reduce human motion registration to the problem of 2D space registration, i.e., how to calculate the optimal solutions of 2D transition and rotation transformation. We introduce analytic method to deduce these solutions from a distance function. Our method reduces greatly the time that registration process costs. *

This paper is supported by National Science Foundation of China (No 60373070).

Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 38 – 47, 2006. © Springer-Verlag Berlin Heidelberg 2006

Human Motion Interpolation Using Space-Time Registration

39

Thirdly, we interpolate the parameters of the corresponding frames of input motions. In this step, the interpolation of transitions of the root joints is executed, and then the interpolation of angles of all joints is implemented with quaternion technique. Finally, fuzzy theory is introduced to constraint definition of resulting motion, and the constraints of resulting motion are reconstructed with IK (Inverse Kinematics) technique.

2 Related Works Time Registration Bruderlin et al use the method of signal processing to warp motion time [3]; Sloan et al align temporally input motions with weight average [4], but their method may result in ‘time reverse’. Many researchers utilize human machine interaction to choose key time, and then implement time registration [1,5,6,7]. Despite their simplicity, their algorithms require user’s intervention. Kovar implements time warping based on DP (dynamic programming) [8]. His method is automatic, however, it runs slowly. Other than the former methods, our algorithm makes use of periodicity of human motions, warps the motion time, and automatically carries out time registration of input motions. Our method is rapid, and can avoid the error of ‘time reverse’. Space Registration Kovar and Gleicher et al present coordinate frame alignment to implement space registration for motion frames [9, 10]. Lee et al define a distance function based on position and velocity of joints, but their function has its shortage. For two frames where motion poses are similar but motion coordinates are much different, the distance between frames by Lee’s method are big. We adopt the distance function that Kovar et al present [9,10] and deduce mathematically the optimal transformation. They firstly calculate spatial registration between all frames of input motions, and then implement time registration with DP method. However, our method is mainly different from their method. We carry out time registration based on motion cycles and then execute spatial registration between corresponding cycles of input motions. Our method greatly drops the computation cost. Parameter Interpolation Perlin and Winkin use weighed average method to interpolate Euler angles of two input motions [11,12]. Their methods fail to create smooth motion. Park et al propose a motion interpolation method based on RBF [1]. Unlike these methods, we divide motion data into two parts: spatial transition of root joint and Euler angles of all joints. We apply linear interpolation method based on motion velocity to the former, and apply quaternion interpolation method to the latter.

3 System Overview The interpolation algorithm includes four steps: time registration, space registration, parameter interpolation and constraint reconstruction(Fig.1).

40

Z. Chen et al.

Time Registration

Space Registration

Transition Interpolation Constraint Reconstruction

Angle Interpolation Parameter Interpolation

Fig. 1. The frame of the motion interpolation algorithm based on space-time registration

4 Time Registration Motion Cycle Human motion is of high cooperation and the movements of human joints interact. In practice, we can obtain experience from real life: some human motions such as the motions of human jog and run, share similar variety in joint movements. For example, the movements of limbs take on ‘diagonal symmetry’, i.e., human’s left hand swings as similarly as his right foot, and human’s right hand swings as similarly as his left foot. So we can compute the cycles of human motion according to the swing frequency of human’s limb. The computation method is as follows: 1. According to the forward kinematics, use Euler angles of limb joint links to compute the transition function p e (t ) of end effectors of all input motions. The formula is as follows:

p k ,e (t ) = f F ( A k (t ))

(1)

where the suffix k represents the k-th input motion, A k (t ) represents the Euler angle set of the k-th input motion, and f F (•) denotes the forward kinematics function. 2. Calculate the velocity function v e (t ) of End Effectors of human limbs. The formula is as follows:

v k , e (t ) =

dp k , e (t )

(2)

dt

3. Calculate the interval time set Th of motion cycles. The formula is as follows:

{

Th = t i | v k ,e (t i )

2

}

=0

(3)

4. Take out elements in sequence from Th and then obtain motion cycles Ci : Ci = {ti , ti +1}

(4)


41

where t i ∈ Th . In light of these motion cycles, we divide the human motion sequences into motion chips. Every motion chip lasts a motion cycle. Time Registration Time registration of two motion sequences can be simplified to correspondence between chips and correspondence between frames: 1.Correspondence between chips Align in sequence motion cycles of input motions. 2. Correspondence between frames After corresponding chips are obtained, we can seek corresponding frames in these chips. The computation formula is as follow (Fig.2): f i ' − f1' f'− f' = 2 1 f i − f1 f 2 − f1

(5)

Ci f1

motion1

f2 fi

f’i

motion2

f’1

f’2 C’i

Fig. 2. Time corresponding between motion frames

where f1 and f 2 represent the first frame and last frame of motion cycle Ci of input motion m1 (t ) , respectively. Similarly, f1' and f 2' represent the first frame and last frame of motion cycle Ci' of input motion m 2 (t ) , respectively. Given an arbitrary frame f i of motion cycle Ci of input motion m1 (t ) , its corresponding frame f i' of motion cycle Ci' of input motion m 2 (t ) can be computed as follows:

f i' = f1' +

f 2' − f1' ( f i − f1 ) f 2 − f1

(6)

5 Space Registration After temporal registration, Input motions must be aligned spatially. Otherwise, There will be errors during interpolation resulting from the difference of human’s positions.

42

Z. Chen et al.

In our registration method, spatial positions of all joints are calculated by means of forward kinematics, and then every joint of a motion frame can be represented as a point in 3D space. All joints of a motion frame form “point cloud”. A distance function is introduced to define the difference of two motion frames, i.e., “two point cloud”. In the end, we calculate the optimal rigid transformation to make the distance of two frames least and implement the spatial registration of human motions. The distance between two frames is defined as follows: N

f d (m(ti ), m(t j )) = ∑ wk m(ti ) − T ⋅ R ⋅ m(t j )

2

(7)

k =1

For simplification, we only consider that human moves on the plane surface. In this situation, spatial registration only requires 2D rigid transformation. Then Equation (7) is simplified as follows: N

f d (m(ti ), m(t j )) = ∑ wk m(ti ) − T( X * , Z * ) ⋅ R (θ ) ⋅ m(t j )

2

(8)

k =1

The distance between two frames is defined as the least value of the distance function under 2D transformation: D (m(ti ), m (t j )) = min f d (m(ti ), m (t j )) * *

(9)

θ , X ,Z

The required transformation can be represented as:

⎧θ = arg min f d (m (ti ), m (t j )) θ ⎪ ⎨ * * f d (m(ti ), m(t j )) ⎪( X , Z ) = arg* min ( X ,Z * ) ⎩

(10)

θ and ( X * , Z * ) . Suppose that two input motions are needed registration and denoted as m1 (t ) and m 2 (t ) 。 The equations below can be obtained by using Equation (8) and (9)： Then we try to calculate the value of

N

D (m(ti ), m(t j )) = min ∑ wk m(ti ) − T( X * , Z * ) ⋅ R (θ ) ⋅ m(t j ) * *

2

θ , X , Z k =1

(11)

Equation (11) can be extended to: D = min * *

N

∑ wk [(CX + SZ + X * − x) 2 + (Y − y ) 2 + (− SX + CZ + Z * − z ) 2 ]

θ , X , Z k =1

(12)

Where ( x j , y j , z j )T and ( X j , Y j , Z j )T represent the j-th joint’s coordinates of the ith corresponding frame of m1 (t ) and m 2 (t ) , respectively. Differentiate Equation (12)：

∂D = 2∑ wk [ S ( xX + zZ − X * X − Z * Z ) + C ( X * Z + zX − xZ − Z * X )] ∂θ

(13)


Let

43

∂D = 0 and ∂θ −

−

−

−

∑ wk x = x , ∑ wk z = z , ∑ wk X = X , ∑ wk Z = Z k

k

k

(14)

k

Then we get the value of θ :

θ == arctan

− −

− −

− −

− −

∑ wk ( xZ − zX ) − ( x Z − z X ) k

∑ wk ( xX + zZ ) − ( x X + z Z )

(15)

k

Similarly, we differentiate Equation (12) again: ∂D ∂X *

= 2∑ wk (CX + SZ + X * − x)

(16)

k

Alike， differentiate Equation (12) again and then we get: −

−

−

−

−

−

X * = x− C X − S Z , Z * = z + S X − C Z

(17)

6 Parameter Interpolation 6.1 Transition Interpolation

Given an arbitrary input motion m(t ) = {P(t ), A (t )} , P (t ) and A(t ) represent transitions of root joint and angles of all joints, respectively. Parameter interpolation of transitions can be carried out according to Equation (18): Pb (t ) = ∑ wk (t )Pk (t ) k

(18)

Where Pk (t ) represents the k-th input motion after space-time registration and

wk (t ) represents the weight of the k-th input motion. 6.2 Angle Interpolation

We convert Euler angles of human joints into quaternions, and then implement rotation interpolation by spherical linear interpolation. So the rotation interpolation of the j-th frame can be calculated:

A b, j (t ) = slerp( A1, j , A 2, j , w(t ))

(19)

Where A1, j , A 2, j are the rotation quaternions of the j-th frame of the first and second motion, respectively, and slerp (E1 , E 2 , w) represents interpolation function [5], the weight,

w is

44

Z. Chen et al.

7 Constraint Reconstruction The input motions are interpolated to create the target motion. However, it may break original constraints. So the step of constraint reconstruction is required. Groundtouch constraints are main considerations because they are the most important constraints of human motions. We apply fuzzy mathematics method to the constraint state of the resulting motion. Suppose that F represent the set of all motion frames, Fc represent the set of the constrained frames, then: ⎧0, t ∉ Fc S k (t ) = ⎨ ⎩1, t ∈ Fc

(20)

Where S k (t ) represent the constraint state function of the k-th input motion. Let Sb (t ) represent the constraint state function of the resulting motion, then:

当当

^ ⎧ ⎪0, Sb (t ) < λ S b (t ) = ⎨ ^ ⎪1, S (t ) ≥ λ b ⎩

Where

λ

(21)

is a threshold that is bigger than 0 and: ^

Sb (t ) = ∑ wk (t ) S k (t )

(22)

k

After its constraint state is obtained, the resulting motion can be reconstructed by simple IK (inverse kinematics) method.

8 Results and Discussion 8.1 Experimental Results Our experiments consist of two types of motion interpolation. The first type is standard interpolation (Figure 4), and the second is the interpolation with the specified path (Figure 5). All experiments can be run in real time in the condition of computer configuration: P4 2.4G, 512MB.

(a)

(b)

Fig. 3. The captured motions motion of run and walk are showed in (a) and (b)


45

We apply our algorithm for motion interpolation to the capture data of human motions. The captured motions include two kinds: human walk and run (Figure 3). The standard interpolation results are showed in Figure 4. Figure (a) shows the result of Rose’s interpolation algorithm that is without space-time registration. The hands of human character twitter abnormally. Figure (b) shows the result of our algorithm: run and walk motions are interpolated to create jog motion. In contrast to Rose’s result, our result eliminate the error of abnormal twittering.

(a)

(b)

Fig. 4. The results of the motion interpolation algorithm. (a) The result of interpolation without space-time registration. (b) The result of our algorithm.

In some situations, the user hopes to specify the path of the interpolated motion. In order to create this motion, path correction must be add after motion interpolation. In our implementation, path correction includes two parts: location correction and orientation. Our interpolation result of run and walk motions is shown in Figure 5. The red curve indicates the path that user draws in advance.

Fig. 5. The motion interpolation based on given path. Red curve shows the given path.

8.2 Discussion Our time registration algorithm is based on motion cycles. Unlike many exiting methods [5,6,7], it requires no user’s intervention. Furthermore, our algorithm is simple and runs in real time, other than Kovar’s method based on DP. In addition, we overcome the error of ‘time reverse’ that red curve indicates (Figure 6). Our method can greatly reduce the computation complexity of space-time registration. Suppose that the count of frames of two input motions are M and N , respectively, the computation complexity of Kovar’s algorithm is O( M ⋅ N ) ; In contrast, the M N computation complexity of our algorithm drops greatly down to O( ⋅ ) ( l1 and l2 l1 l2 are the count of cycles of input motions, respectively).

46

Z. Chen et al.

Real time t

Real time t

Reference time t’ T1

T2

T3 (a)

T4

Reference time t’ T1

T2

T3

T4

(b)

Fig. 6. The schematic diagram of the time registration step. (a) Principles of Sloan’s algorithm; (b) Principles of our algorithm. Our algorithm overcomes the error of time reverse that red curve indicates.

9 Conclusion In this paper, we propose a new algorithm for human motion interpolation based on space-time registration. The algorithm includes time registration, space registration, parameter interpolation and constraint reconstruction. Experimental results demonstrate that our algorithm succeeds in interpolating input motions. Furthermore, our method can greatly reduce the computation complexity of space-time registration and can avoid the error of time reverse.

Reference 1. Park M. J., Choi M. G., Shin S. Y: Human motion reconstruction from inter-frame feature correspondences of a single video stream using a motion library. In Proc. of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation. ACM Press(2002), 113-120. 2. Menardais S., Multon F., Kulpa R., Arnaldi B.: Motion blending for real-time animation while accounting for the environment. In Proceeding of Computer Graphics International (2004), 156- 159. 3. Bruderlin A., Williams L.: Motion signal processing. In Proc. of SIGGRAPH, ACM Press(1995), 97-104. 4. Sloan P., Rose C., Cohen M.: Shape by Example. In Proc. of 2001 Symposium on Interactive 3D Graphics , ACM Press(2001), 135 – 143. 5. Guo S., Robergé J.: A high-level control mechanism for human locomotion based on parametric frame space interpolation. In Proc. of the Eurographics workshop on computer animation and simulation(1996), 95—107. 6. Rose C., Bodenheimer B., Cohen M.: Verbs and Adverbs: Multidimensional Motion Interpolation. IEEE Computer Graphics and Applications(1998) 18(5), 32-41.


47

7. Park M., Shin S.: Example-based motion cloning. Computer animation and virtual worlds(2004), 15, 245-257. 8. Kovar L., Gleicher M.: Flexible Automatic Motion Blending with Registration Curve. In Proc. of Eurographics/SIGGRAPH Symposium on Computer Animation. ACM Press (2003), 214–224. 9. Kovar L., Gleicher M., Pighin F.: Motion Graphs. In Proc. of SIGGRAPH 2002, ACM Press(2002), 473-482. 10. Kovar L., Gleicher M.: Flexible Automatic Motion Blending with Registration Curve. In Proc. of Eurographics/SIGGRAPH Symposium on Computer Animation. ACM Press(2003), 214–224. 11. Perlin K.: Real time responsive animation with personality. IEEE Transactions on. Visualization and Computer Graphics (1995), 5-15. 12. Witkin A., Popovic Z.: Motion warping. In Proc. of SIGGRAPH, ACM Press(1995), 105-108.

Multi-stream Based Rendering Resource Interception and Reconstruction in D3DPR for High-Resolution Display Zhen Liu, Jiaoying Shi, Hua Xiong, and Aihong Qin State Key Lab of CAD&CG, Zhejiang University, 310027,HangZhou, PRC {liuzhen, jyshi, xionghua, qinaihong}@cad.zju.edu.cn

Abstract. In order to bring Direct3D9 application to run on PC clusters for high-resolution display with no modification, we have firstly presented D3DPR parallel rendering system. According to multi-stream scene data organization and storage mode of Direct3D9, rendering resource has been classified as command stream, vertex stream, index stream and texture stream in D3DPR and the rendering resource interception and reconstruction becomes the most crucial problem. In this paper, above all we give an overview of D3DPR system architecture. Subsequently we bring forward multi-stream based rendering resource interception in D3DPR. In this section, we design DPGL, i.e. Direct3D9-based Parallel Graphics Library, through which the rendering information of Direct3D9 application can be converted into the above described rendering resource. Finally we focus on multi-stream based rendering resource reconstruction in D3DPR, through which the rendering command and scene data was reconstructed.

1

Introduction

With technological advances, large-high resolution displays are becoming prevalent in many fields, such as immersive virtual environment, command and control centers, geospatial imagery and video, scientific visualization, public information displays, vehicle design and so on [10]. Because standard computer monitors have remained at a resolution of around one million pixels, usually these high-resolution displays require the combination of several display devices into a single display environment, providing large physical size and high resolution by combining several images [11]. Traditionally, high-resolution display virtual environment has been driven primarily by powerful graphics supercomputers, such as SGIs Onyx System. Unfortunately, it comes at high cost. However the emergency of high-performance 3D commodity graphics cards has opened the way to use PC clusters for highresolution display virtual environment. In order to exploit the available and potential parallel rendering ability, we must take into account important design considerations associated with PC clusters rendering for high-resolution display system. These issues involve parallel Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 48–57, 2006. c Springer-Verlag Berlin Heidelberg 2006

Multi-stream Based Resource Interception and Reconstruction

49

rendering library and interface, sorting in the rendering pipeline, task distributing, load balancing and network performance etc. Among them, the parallel rendering interface is a critical factor and determines whether the user could make use of the parallel system conveniently. Most research in parallel rendering field has focused on OpenGL interface. With respect that Direct3D has been developed rapidly and widely used in multimedia, entertainment, games, 3D animation and 3D graphics computation, it is imperious to develop a Direct3D interface to the users. We firstly presented D3DPR [9], a Direct3D9-based parallel rendering system, which is applied to high-resolution environment. It allows that Direct3D9 application can run on PC clusters with no modification. D3DPR parallel system has two type of logical node, one is Geometry Distributing Node (D-node), and the other is Geometry Rendering Node (R-node). Because Direct3D9 library is medium layer between Direct3D9 application and graphics hardware, the rendering resource interception on D-node of Direct3D9 application and reconstruction on R-node is the key problem of D3DPR. Compared with OpenGL, Direct3D9 provides COM-based interface and multi-stream scene data organization to the application. Therefore in this paper, we will research on multi-stream based rendering resource interception and reconstruction in D3DPR parallel system. This paper is organized as Follows. The following section reviews the background and related work in parallel rendering. Section 3 describes multi-stream scene data organization and storage in Direct3D9. Section 4 gives an overview of D3DPR system architecture. Section 5 and section 6 explains rendering resource interception and reconstruction algorithm in D3DPR. Section 7 shows one experimental result. Finally in section 8, we will conclude our work and discuss future work.

2 2.1

Background and Related Work Parallel Rendering Algorithm Classification

In 1994, Molnar et al. described a parallel rendering algorithm classification based on where the sort takes place in the translation from object coordinates to screen coordinates [1]. This is believed to be fundamental whenever both geometry processing and rasterization are performed in parallel. To summarize their classification scheme, parallel rendering algorithms fall into one of three classes: sort-first (early during geometry processing), sort-middle (between geometry processing and rasterization) and sort-last (after rasterization), shown as Fig. 1. 2.2

Popular PC Clusters Based Parallel Rendering System

Many famous universities and institutes, for instance Stanford, Princeton, UIUC and Zhejiang University etc, have devoted to the field of parallel rendering based on PC clusters, which supports high-resolution environment. A number of excellent systems have been produced. Stanford University Computer Graphics

50

Z. Liu et al.

G

R

G

R

G

R

G

R

G

R

G

R

G G

R R

G G

R R

G G

R R

Tiled Display

Tiled Display

Tiled Display

G-Geometry Processing Stage R-Rasterization Stage (a)sort-first

(b)sort-middle

(c)sort-last

Fig. 1. Parallel Rendering Algorithms Classification

Lab designed WireGL [2] and its successor Chromium [3]. WireGL is the first sort-first graphics rendering system on PC clusters and is platform-independent to hardware. Chromium provides Stream Processing Unit (SPU) and can be configured to sort-first, sort-last and hybrid parallel graphics architecture. Princeton University described four different systems to do parallel rendering [4].GL-DLL Replacement implements the OpenGL API in a fashion comparable to WireGL, although not as optimized. System-level Synchronized Execution (SSE) and Application-level Synchronized Execution (ASE) are comparable to the Aura [7] Multiple Copies renderer. The fourth renderer is not a really renderer but a virtual display (VDD) and is used for 2D (desktop) rendering. The integrated Systems Lab at University of Illinois at Urbana Champaign presented DGD (Distributed Graphics Database) [5], which provides the user with a scene graph interface and hides all parallel rendering related issues underneath it. Now DGD project has been renamed to Syzygy [6]. State Key Lab of CAD&CG in Zhejiang University has presented several PC clusters based parallel rendering systems for high-resolution display, such as AnyGL [13], MSPR [12] and D3DPR [9]. AnyGL is a hybrid sort-first and sortlast rendering system, which supports large scale geometry data distributing rendering with high scalability. MSPR is a retained-mode based multi-screen parallel rendering system that offers programmers OpenGL-like API. D3DPR provides completely transparent Direct3D9 application parallelization.

3

Multi-stream Scene Data Organization and Storage

In order to intercept rendering resource of Direct3D9 application on D-node and reconstruct them on R-node in D3DPR parallel rendering system, first of all we should focus on how the scene data are organized and stored in Direct3D9 application. 3.1

Multi-stream Scene Data

Direct3D9 library provides Vertex Buffer and Index Buffer for application to organize scene data. Vertex Buffer, represented by IDirect3DVertexBuffer9 interface, is sequential memory to store vertex data of scene, including position,


51

normal, color, material, and texture coordinate etc. Index Buffer, represented by IDirect3DIndexBuffer9 interface, is sequential memory to store index information for addressing topological relationship of vertex data in the Vertex Buffer [8]. This manner that vertex data and index data are stored sequentially by Vertex Buffer and Index Buffer can be defined as stream-based scene data organization. Furthermore, if position, normal, color, material and texture coordinate property of vertex data was stored in different Vertex Buffer, this manner could be named for multi-stream scene data organization and each Vertex Buffer could be considered as one stream. But for once specific rendering, multiple Vertex Buffers could only correspond to one Index Buffer. 3.2

Multi-stream Scene Data Layout

Conceptually, the vertex declaration is a way of programming the vertex direct memory access (DMA) and tessellator engine of the Direct3D9 graphics pipeline. It expresses the stream-based data layout and the tessellator operation concisely. To address the complexity and extensibility of declaration in Direct3D9, vertex streams are now declared with an array of D3DVERTEXELEMENT9 structures. Each element of the array describes one vertex property, such as position, normal or texture coordinate. This is a new style of declaration that replaces the older style that uses flexible vertex format (FVF) codes [8]. 3.3

Multi-stream Scene Data Storage

Multi-stream scene data organized by Vertex Buffer and Index Buffer has three layers storage mode in the memory space, i.e. system memory, AGP memory and video memory of graphics card [8]. Because AGP memory, as extension of video memory, is one part of system memory and AGP bus provides a high-bandwidth path from the graphics card to system memory, AGP memory enhances the potential storage capacity of graphics card to a certain extent. Therefore the scene data, which should be stored in video memory, now can be put into AGP memory. To summarize, multi-stream scene data organization manner, formed by the combination of Vertex Buffer and Index Buffer, and its three layers storage mode is one efficient approach to implement high-performance rendering making use of Direct3D9.

4

D3DPR Overview

D3DPR uses a cluster of PCs with high-performance graphics accelerators to drive an array of projectors. It consists of two types of logical nodes, Geometry Distributing Node (D-node) and Geometry Rendering Node (R-node), shown as Fig. 2. D3DPR allows existing Direct3D9 application to run on the parallel system without any modification. According to the multi-stream scene data organization, the rendering resource of Direct3D9 application can be classified as command stream, vertex stream, index stream and texture stream in D3DPR parallel rendering system. Through the rendering resource rendering interception

52

Z. Liu et al. Pixel D3DPR_Server Win32 Rendering Resource

d3d9

Graphics Hardware

D3DPR_Client Direct3D9 Application Win32

Projector Projector Network

DPGL d3d9

Graphics Hardware Geometry Distributing Node

Projector Projector D3DPR_Server Win32

d3d9

Graphics Hardware Pixel

Tiled Display

Geometry Rendering Node

Fig. 2. D3DPR System Architecture

on D-node and rendering resource reconstruction on R-node, not only the details of network topology architecture, communication and synchronization that are related to network transmission, but also rendering resource interception, rendering resource reconstruction, data distribution strategy that are related to parallel rendering are hidden from the application. Therefore D3DPR is a completely transparent parallel rendering system that support Direct3D9 application parallelization and the tiled display environment can provide better immersion and reality to us.

5

Rendering Resource Interception in D3DPR System

As we known, Direct3D9 library is medium layer between Direct3D9 application and graphics hardware. And Direct3D9 application usually sets the current state of graphics rendering pipeline before it starts up the pipeline to render the scene by calling the ”drawprimitive” method of ”IDirect3DDevice9” interface. Consequently, if D-node intercepts all the render states of pipeline before its startup, it could obtain the rendering process of the application. Furthermore by intercepting from the corresponding stored memory space of vertex buffer, index buffer and texture buffer, i.e. from corresponding system memory, AGP memory or video memory, D-node could obtain the vertex stream, index stream and texture stream of Direct3D9 application. The intercepted command stream corresponds to the rendering process of application and the intercepted vertex stream, index stream and texture stream correspond to the scene geometry data of application. Although this method has eliminated the rendering resource computation, it can record the entire rendering resource of Direct3D9 application for parallel rendering.


53

In order to intercept the rendering resource of Direct3D9 application, we design DPGL, i.e. Direct3D9-based Parallel Graphics Library on D-node in D3DPR system. It is COM-based dynamic linked substitute library and has the same interfaces as Direct3D9 library. Because Direct3D9 library is the interface between Direct3D9 application and graphics rendering pipeline, so the substitute library DPGL can be considered as filter between Direct3D9 application and Direct3D9 library, which prevents Direct3D9 application from invoking Direct3D9 library. Through this method, DPGL can hide rendering resource interception and realize completely transparent parallelization for Direct3D9 application.

Direct3D9 Application

Rendering Command Interception

Command Stream Command Description

Vertex Stream Vertex Description

Index Description

Texture Stream Texture Description

Parameter Pointer

Vertex Buffer Pointer

Index Buffer Pointer

Texture Data Pointer

Parameter Value List

Vertex Buffer Data

Index Buffer Data

Texture Data

Index Stream

Fig. 3. D3DPR Rendering Resource Interception

Depicted as Fig. 3, when Direct3D9 application is executing on D-node in D3DPR, the rendering information of it can be converted into rendering resource described by the above four types of stream after the application routes through rendering command interception layer. The four types of stream could record the scene geometry data and rendering process of every frame of application. For each stream, DPGL should not only record the stream data, for instance parameter value list, vertex buffer data, index buffer data and texture data, but also record the description of each stream data, for instance command description, vertex description, index description and texture description. The purpose of the recorded description is to reconstruct the rendering resource on R-node in D3DPR system.

6

Rendering Resource Reconstruction in D3DPR System

For the purpose of accomplishing parallel rendering on R-node in D3DPR system, in the first place R-node should reconstruct the rendering command and scene data according to the rendering resource described by four types of stream received from the D-node, including command stream, vertex stream, index stream and texture stream.

54

Z. Liu et al.

Receive Stream

Vertex or Index Stream

Allocate Memory to Store Vertex or Index Stream Waiting for Lock Command

StreamType Command Stream Allocate Memory to Store Command Stream

Texture Stream

Allocate Memory to Store Texture Stream

Reconstruct command including COM interface, method and parameters

Lock or LockRect

Command Type

Relate Stream to Clear or Present Corresponding COM

Synchronization

Others

Waiting for LockRect Command

UnLock or UnLockRect Reconstruct Corresponding Buffer by Copy Stream to them

Execute Directly

Fig. 4. D3DPR Rendering Resource Reconstruction

The reconstruction process is illuminated as Fig. 4. When the R-node having received stream from D-node, first of all it determines which type the stream belongs to. If it is vertex stream, index stream or texture stream, R-node will allocate temporary memory to store them and then wait for ”Lock” method of corresponding interface. If it is command stream, R-node will reconstruct the rendering command, including COM interface, method and parameters according to the command description of rendering resource interception. Because the parameters can include COM interface, so R-node should also reconstruct them in the process. Subsequently R-node chooses which type the command attributes to. If it is Lock or ”LockRect”, R-node will relate the stream to the corresponding COM interface, i.e. IDirect3DVertexBuffer9, IDirect3DIndexBuffer9 or IDirect3DTexture9 and wait for ”UnLock” or ”UnLockRect” method. If it is ”Clear” or ”present”, R-node should execute synchronization operation between all the R-node. And if it is others, R-node will execute the command directly. In the following performing process, R-node will reconstruct real IDirect3DVertexBuffer9, IDirect3DIndexBuffer9 or IDirect3DTexture9 by copying memory from the temporary space to the space that the reconstructed COM interface has created. Till now the above interface can represent the multi-stream scene geometry data in deed. After reconstructing all the rendering command and scene data, each R-node can render a sub-frustum of the


55

original scene and all the R-node can be configured into single high-resolution display.

7

Experimental Result

Our experiment is performed on 5 PCs that are connected via 1000M Ethernet network. Each PC is configured as Table 1: Table 1. PC configuration CPU Memory Mainboard Graphics Card Switch Network Card

Two Intel XEON 2.4GHz Kingston 1G DDR Memory Supermicro X5DAL-G, AGP8X graphics card data transmission NVIDIA Geforce FX5950 Enterasys N7, 1000MT Intel Pro 1000MT dual ports

In addition, we select ”StencilMirror” application from Direct3D9 SDK. The sample uses stencil buffers to implement a mirror effect. In the sample, a watery terrain scene is rendered with the water reflecting a helicopter that flies above. In our experiment, One PC is as D-node and fours PCs are as R-node. The D-node intercepts the rendering resource of StencilMirror application and the R-node reconstructs the rendering command and scene data. And each R-node is responsible for drawing a quarter scene of the application. Experimental result is shown as Fig. 5. Four projectors have tiled into high-resolution display virtual environment and the resolution has arrived at 2048*1536.

Fig. 5. High-Resolution Display of ”StencilMirror” Application

56

8

Z. Liu et al.

Conclusions and Future Work

D3DPR is a Direct3D9-based parallel rendering system, which is applied to highresolution display virtual environment. It allows Direct3D9 application to run on PC clusters with no modification. In order to accomplish the purpose, we focus on the rendering resource interception on D-node and reconstruction on R-node in this paper. It is the most crucial problem to implement D3DPR parallel rendering system. In Direct3D9, scene data is organized by multi-stream manner, so rendering resource of Direct3D9 application can be classified as command stream, vertex stream, index stream and texture stream. On D-node, we adopt COM-based dynamic linked substitute library DPGL to realize transparent interception of rendering resource. On R-node, we reconstruct the rendering command and scene data according to the received streams from D-node. Each R-node can render a sub-frustum of the original scene and all the R-node can be configured into single high-resolution display. Topics for further study include development of algorithms for improving the performance of D3DPR, including task decomposition, load balancing and communication overhead reduction etc.

Acknowledgements This paper is supported by National Basic Research Program of China (Grant No:2002CB312105)and Key NSF Project on Digital Olympic Museum(Grant No: 60533080).

References 1. S.Molnar, M.Cox, and et al.:A Sorting Classification of Parallel Rendering. IEEE Computer Graphics and Applications, 14(4)23-32, July 1994 2. Greg Humphreys et al.:WireGL: A Scalable Graphics System for Clusters. In Proceedings of ACM SIGGRAPH 2001 3. Greg Humphreys et al.:Chromium: A Stream-Processing Framework for Interactive Rendering on Clusters. ACM Transactions on Graphics, vol 21(3), pp. 693-702, Proceedings of ACM SIGGRAPH 2002 4. K.Li, H.Chen, and et al.:Early Experiences and Challenges in Building and Using a Scalable Display Wall System. IEEE Computer Graphics and Applications, 20(4): 671-680, July/Aug, 2000 5. B.Schaeffer.:A Software System for Inexpensive VR via Graphics Clusters. http://www.isl.uiuc.edu/ClusteredVR/paper/DGDoverview.htm 2000 6. B.Schaeffer, C.Goudeseune.:Syzygy: Native PC Cluster VR”, Proceeding of IEEE Virtual Reality, 2003 7. D.Germans, H.J.W. Spoelder, and et al.:VIRPI: A High-Level Toolkit for Interactive Scientific Visualization in Virtual Reality. Proc. Immersive Projection Technology/Eurographics Virtual Environments Workshop, Stuttgart, May 2001 8. Microsoft DirectX 9.0 Update (Summer 2003). http://www.microsoft.com/ downloads 9. Zhen Liu et al.:D3DPR: A Direct3D-Based Large-Scale Display Parallel Rendering System Architecture for Clusters. ACSAC 2005, pp. 540 - 550, 2005


57

10. Tao Ni, Greg S. Schmidt, and et al.:A Survey of Large High-Resolution Display Technologies, Techniques, and Application. IEEE Virtual Reality 2006, March, 2006 11. Oliver G. Staadt, Justin Walker, and et al.:A Survey and Performance Analysis of Software Platforms for Interactive Cluster-Based Multi-Screen Rendering. In Proceedings of the Workshop on Virtual Environments 2003, pp. 261-270, Zurich, Switzerland, 2003, ACM Press 12. Chao Li, Zhefan Jin and Jiaoying Shi.: MSPR:A Retained-Mode Based Multiscreen Parallel Rendering System.The the 4th International Conference on Virtual Reality and its Application in Industry, Tianjin, P.R.China, 2003 13. Jian Yang, Jiaoying Shi, Zhefan Jin and Hui Zhang.: Design and Implementation of A Large-scale Hybrid Distributed Graphics System. Eurographics Workshop on Parallel Graphics and Visualization, Saarbruecken, Germany, 2002

VirtualPhonepad: A Text Input Tool for Virtual Environments Jaewoo Ahn1 and Kyungha Min2, 1

Mobience Inc., Seoul, Korea Sangmyung Univ., Seoul, Korea Dept. of Digital Media, Sangmyung Univ., 7 Hongji-dong, Jongro-gu, Seoul, 110-743, Korea [email protected] 2

Abstract. This paper presents an effective text input tool for virtual environments. The key idea of this paper is that we make use of the text input mechanism of mobile phones. We present a 3 × 3 keypad widget whose user interface is similar to the keypad of mobile phones, where each key on the widget has several characters. We present indexing schemes by which users locate the key on the keypad they want and selection schemes by which users choose the desired character from the characters on the keys of the keypad using diverse input devices.

1

Introduction

As applications of virtual reality are widening their influences over various areas rapidly, the number of virtual environments that need text input schemes is also increasing. For example, a user in a traditional building walkthrough application may want to send a message to other users, or a user in a virtual museum may want to ask a query about items he or she is looking at. In all these applications, a common requirement is a scheme that enables text input in virtual environments. Since the traditional keyboard is not available in these environments, it is not intuitive to develop text input schemes for virtual environments. In this paper, we present VirtualPhonepad scheme for text input in virtual environments. This scheme is developed by exploiting the text input scheme on mobile phones. Users who possess mobile phones send text messages using SMS (Short Message Service) applications on the phones. Most of the SMS applications on mobile phones from various vendors present simple text input mechanisms, which are intuitive to use. We analyze the mechanism of the text input of the SMS applications and design a similar text input scheme for virtual environments. Figure 1 illustrates the overview of the proposed VirtualPhonepad scheme. This paper is organized as follows. In Section 2, we briefly review related works, and we analyze the text input mechanism in the mobile phones in Section 3. We design the VirtualPhonepad in Section 4, and we implement the scheme in Section 5. In Section 6, we summarize this paper and suggest future work.



VirtualPhonepad: A Text Input Tool for Virtual Environments

59

Fig. 1. Overview of the VirtualPhonepad: (a) User interface of the scheme, (b) Indexing a key that contains the desired character, (c) Selecting the character by pressing the key in appropriate number of times

2

Related Works

In virtual reality, several schemes have been proposed to introduce text input in immersive virtual environments. Poupyrev et. al. [Poupyrev et al. 1998] proposed a virtual notepad, which records characters written on a notepad in the virtual environment. This scheme enables users to leave a short hand written message in the virtual environment. This scheme, however, cannot be regarded as a text input scheme, since only the drawings of the characters written on the notepad are simply recorded and played back. Fels and Hinton [Fels & Hinton 1998] proposed GloveTalkII scheme that converts gestures into speech by recognizing the predefined gestures using neural nets. The hand gesture contains ten control parameters, which are combined to represent the context to express. The limitation is that the range of expressible contexts is limited and that it requires a reasonable exercise to use this scheme. Rosenberg and Slater [Rosenberg & Slater 1999 ] presented a chording glove, which interprets combinations of touches of fingers on a hard surface as a character. Even though this scheme can represent all the alphanumeric characters, users are required to memorize the correspondences between the characters and the chords of fingers. Bowman et. al. [Bowman et al. 2001] have presented a virtual keyboard implemented by a pinch glove and tested its usability over other text input schemes. This scheme presents an intuitive interface for the text input by displaying the keyboard in immersive virtual environments and selecting the characters on the keyboard by the interactions of the pinch glove. This scheme, however, suffers from the inefficiency of the input procedure. Shin and Hong [Shin & Hong 2006] presents the Keypad glove scheme for the text input in wearable computing environments. A data glove in one hand plays the role of keypad of a mobile phone by assigning each key to the corresponding finger segment on the glove, and the other hand plays the role of pressing the key. The difference from the selection step of the text input scheme in a mobile phone is that a character among the characters on a key is selected directly by assigning different indices to the three fingers of the other hand.

60

J. Ahn and K. Min

Frees et. al. [Frees et al. 2006] have presented Connect The Dot scheme for text input in immersive virtual environments. In the scheme, the user interface is composed of nine dots organized in 3×3 structure. Users input characters by connecting the dots in some predefined orders. In addition to these researches, several researchers [Thomas et al. 1998, Lindeman et al. 1999, Bowman et al. 2002] have tested existing text input schemes for virtual environments and wearable computing environments and compared them through empirical studies. Ogi et al. [Ogi et al. 2001] have exploited the interface of cell phone in navigating virtual environments. However, they didn’t provide any text input schemes in their works.

3

Analysis of the Text Input Mechanism on Mobile Phones

The fundamental operation of the text input mechanism for alphabetic characters is to choose one of possible twenty six characters. When we use a device that has twenty six individual keys each of which is dedicated only to one alphabetic character such as a keyboard or a typewriter, the only operation required to enter a character is to locate the key that contains the desired character. Mobile devices whose small sizes do not support full-sized keyboard require some mechanisms that enable text input for the devices. The text input mechanisms developed for such devices can be classified into two categories: using keypads of limited keys or using character recognition techniques. Many mobile phones use the keypad-based mechanism and many PDA’s use the recognition-based mechanism. We focus on the keypad-based mechanism used in mobile phones. In the keypad-based approach, each key on the keypad contains about three characters due to the limited number of keys. The basic philosophy of the text input operation in this approach is to separate the operation into two steps. The first step, which is denoted as indexing process, is to locate a key whose character set contains the desired character and the second step, which is denoted as selection process, is to select the desired character from the set. In most of the mobile phones, the selection process is commonly practiced by repeatedly pressing the key. For example, a user who wants to enter ”A” indexes a key that contains the character set of ”A B C” and presses it once, and a user who wants to enter ”B” indexes the same key and presses it twice, and so on.

4 4.1

Design of VirtualPhonepad Basic Concept

The VirtualPhonepad is designed by a combination of a widget and a text input scheme. The widget, which is similar to the keypad of mobile phones, is a 3 × 3 matrix of keys each of which contains several characters. Similar to the mechanism in mobile phones, the text input mechanism of VirtualPhonepad is composed of indexing the key whose character set contains the desired character and selecting the character from the character set.


4.2

61

Design of the Widget

The design strategy of the widget is to arrange 26 alphabet characters and a space character in 3 × 3 keypad. The configuration of the keys can be modified according to that of the keys on users’ mobile phone. The benefit of the 3 × 3 structure over the 3 × 4 structure is that each key on 3 × 3 structure corresponds to the nine directions in two dimensional coordinate, which is a combination of (up, middle, down) and (left, center, right). This benefit extends the possible devices for text input even to joystick devices, whose only inputs are direction information. 4.3

Indexing

Indexing a key is implemented in two ways. One way is to trace the position of the device and identify the key whose position is occupied by the pointer of the device, and the other way is to recognize the direction of the input device and identify the key that is lying on the direction. 4.4

Selection

In the case of phones, the number of key presses within a predefined time interval selects a character on the key. For example, pressing a key that contains ”A”, ”B”, and ”C” once selects ”A”, twice selects ”B”, and three times selects ”C”. This selection scheme is due to the limited input interface of phones. In this paper, we use two classes of selection schemes. Single-button Device: The selection scheme for single-button devices is designed by the number of key presses. For this scheme, we assign a threshold value tp to the time interval between the key presses for the selection. A subsequent press of a button within tp denotes that it is for the current character selection. Similarly, a press not within tp denotes that it is for the next character selection. Multi-button Device: The selection scheme for multi-button devices can be designed by utilizing the multiple buttons. We assign different selection identifiers for the individual buttons or fingers on such devices. The selection schemes for the devices exploited in this paper are listed as follows: Data glove: When using a data glove, we assign the selection function to different fingers. For example, the index finger denotes the first character on the button, the middle finger denotes the second, and the ring finger denotes the third. Two-button mouse: When using a two-button mouse, the left-button click denotes the first character, the right-button click denotes the second, and simultaneous presses of left-button and the right-button denotes the third. Multi-button joystick: Similar to the case of a two-button mouse, combinations of buttons in the joystick can be used to select a character from the character sets on a button.

62

5

J. Ahn and K. Min

Implementation of VirtualPhonepad

Figure 2 illustrates the snapshots of VirtualPhonepad. We present a visual feedback by changing the color of the key and by displaying the selected character in bold-face. We have tested VirtualPhonepad scheme in four different versions classified by the indexing scheme and the selection scheme. The indexing scheme is either to direct the key by a data glove or to position the key by a mouse, and the selection scheme is either to press the key with a button repeatedly within a predefined time interval (single-button) or to press the key with different buttons or combinations of the buttons (multi-button). We have grouped twenty subjects

(a) Shape of the widget

(b) Entering ì aî

(c) Entering ì tî

(d) Entering ì rî

Fig. 2. Screen-shots of VirtualPhonepad scheme


63

single-button multi-button directing 12.8 (15.3) 15.6 (19.2) positioning 10.7 (13.1) 14.3 (17.5) Fig. 3. Comparing performances of combining various selection schemes and indexing schemes. Note that single-button and multi-button are selection schemes and directing and positioning are indexing schemes. The number indicates words per minute.

into four different versions of the VirtualPhonepad and tested the scheme by entering five different sentences. We also have tested them at the final step by entering the first sentence again. The result is illustrated in Figure 3. The test reveals that the combination of directing and multi-button makes the best performance. The number in parenthesis is the average performance for the re-entry of the first sentence. We have got an experimental argument that the most bottleneck of the text input process is the unfamiliarity of the character locations on the keypad. This argument is somewhat proved in that the performance of the input is improved as experiments are progressed, although it has limitations.

6

Conclusion and Future Work

In this paper, we have presented VirtualPhonepad, a text input scheme for virtual environments. The scheme is proved to be very powerful and robust in comparison to other text input schemes. The future research direction is to improve the scheme to include the numerics and symbols as well. We are going to find a new arrangement of the characters on the keypad so that it is familiar and intuitive to users for an improved text input tool in virtual environments.

References [Bowman et al. 2001] Bowman, D. and Wingrave, D. and Campbell, J. and Ly, V.: Using Pinch Gloves for both Natural and Abstract Interaction Techniques in Virtual Environments. Proceedings of HCI International, 629–633, 2001. [Bowman et al. 2002] Bowman, D. and Rhoton, C. and Pinho, M.: Text Input Techniques for Immersive Virtual Environments: an Empirical Comparison. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2154–2158, 2002. [Fels & Hinton 1998] Fels, S. and Hinton, G.: Glove Talkll: A Neural Network Interface which Maps Gestures to Parallel Formant Speech Synthesizer Controls. IEEE Transactions on Neural Networks, 9(1), 205–212, 1998. [Frees et al. 2006] Frees, S. and Khouri, R. and Kessler, G.: Connecting the Dots: Simple Text Input in Immersive Environments, Proceedings of IEEE Virtual Reality, 25–29, 2006. [Lindeman et al. 1999] Lindeman, R. and Sibert, J. and Hahn, J.: Towards Usable VR: An Empirical Study of User Interfaces for Immersive Virtual Environments, Proc. Of the SIGCHI ’99, 64–71, 1999.

64

J. Ahn and K. Min

[Ogi et al. 2001] Ogi, T. and Yamamoto, K. and Yamada, T. and Hirose, M: Experience of Virtual World Using Cellular Phone Interface, Proceedings of IEEE PCM 2001, 32-39, 2001. [Poupyrev et al. 1998] Poupyrev, I. and Tomokazu, N. and Weghorst, S.: Virtual Notepad: handwriting in immersive VR. Proceedings of the Virtual Reality Annual International Symposium, 126–132, 1998. [Rosenberg & Slater 1999 ] Rosenberg, R. and and Slater, M.: A Chording Glove: A Glove Based Text Input Device, IEEE Transactions on Systems, Man, and Cybernetics, 29(2), 186–191, 1999. [Shin & Hong 2006] Shin, J.H. and Hong, K.S.: Keypad gloves: glove based text input device and input method for wearable computers, Electronics Letters, 41(16), 15– 16, 2006. [Thomas et al. 1998] Thomas, B. and Tyerman, S. and Grimmer, K.: Evaluation of Text Input Mechanisms for Wearable Computers. Virtual Reality, 3(3), pp. 187199, 1998.

An Animation System for Imitation of Object Grasping in Virtual Reality Matthias Weber, Guido Heumer, Heni Ben Amor, and Bernhard Jung VR and Multimedia Group TU Bergakademie Freiberg Freiberg, Germany {matthias.weber, guido.heumer, amor, jung}@informatik.tu-freiberg.de

Abstract. Interactive virtual characters are nowadays commonplace in games, animations, and Virtual Reality (VR) applications. However, relatively few work has so far considered the animation of interactive object manipulations performed by virtual humans. In this paper, we first present a hierarchical control architecture incorporating plans, behaviors, and motor programs that enables virtual humans to accurately manipulate scene objects using different grasp types. Furthermore, as second main contribution, we introduce a method by which virtual humans learn to imitate object manipulations performed by human VR users. To this end, movements of the VR user are analyzed and processed into abstract actions. A new data structure called grasp events is used for storing information about user interactions with scene objects. High-level plans are instantiated based on grasp events to drive the virtual humans’ animation. Due to their high-level representation, recorded manipulations often naturally adapt to new situations without losing plausibility.

1

Introduction

Achieving realistic body movement in virtual humans for interactive object manipulations is still a difficult task. Previous work in this field has led to two different approaches to realistic character animation: data-driven methods, using motion-capture data, and model-driven methods, synthesizing animations at runtime. Usually, the data-driven approaches suffer from limited adaptability to new situations, such as changing environments and different character proportions. Model-driven approaches on the other hand allow for high adaptability but suffer from an increasing number of control parameters. This normally results in a tedious trial-and-error process for finding an accurate set of parameters. Although one could hard-wire such parameters in a model-driven animation system, flexibility against modified objects or differently sized virtual humans would be lost as a consequence. This paper follows a novel, imitation based approach to animation called Action Capture [1] which combines the strengths of data-driven and modeldriven techniques. The interactions of a real user with the virtual environment are recorded, processed and abstracted, such that they can be adapted to new Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 65–76, 2006. c Springer-Verlag Berlin Heidelberg 2006

66

M. Weber et al.

situations. This allows for fast and easy recording of new animations without need for parameter optimization and, at the same time, also enables the flexible reproduction of interactions with scene objects. While the generic conceptual framework for Action Capture including its relationships to the fields of imitation learning and robotics has been described elsewhere [1], this contribution focusses on the details of a specific implementation for imitation of object grasping by virtual humans. For this, a hierarchical control architecture for animation of virtual humans is realized. On the highest level of abstraction plans describe animations as sequences of object interactions. Plans instantiate goal-oriented behaviors to influence body movement. Behaviors use low-level motor programs to directly move the body’s limbs. Additionally, for storing information about user interactions with scene objects a new data structure called grasp events is introduced. In turn, grasp events are converted into high-level plans in order to drive the virtual human’s movement. This imitation-based approach provides great ease-of-use when creating animations for object grasping. Taking full advantage of immersive VR, the animation process reproduces a human user’s natural 3D interactions without the need for cumbersome retargetting of motion capture data or trial-and-error parameter optimization. After discussing related work, the remainder of this paper presents an operational system implementing the imitation-based approach to character animation.

2

Related Work

In order to enable a virtual human to manipulate other objects, it is of importance to specify different meaningful ways of interaction. An interesting approach to this was introduced by Kallmann and Thalmann [2]. The basic idea of the smart object approach is that each object “knows” the way an interaction with it should look like. Additionally, each object stores a number of parameters and other meta-data, that can be accessed and used by the virtual human. This metadata has to be specified by the system developer when creating the objects and the world. Still, there is the question of how to synthesize an appropriate animation from this data. Early work on animation and control for manipulation and grasping was reported by Sanso and Thalmann [3]. Kuffner and colleagues [4] proposed the use of a path planner and inverse kinematics in order to plan a collision-free trajectory connecting an initial arm position with a goal position. In an extension of this algorithm [5] they were also able to bias the inverse kinematics algorithm towards natural-looking poses using a motion database. In contrast to this, the work of Douville et al. [6] describes a behavioral approach to motion synthesis. Here, different objects are grasped using specialized grasping behaviors and an object specific reasoner. The latter helps to break down the high-level plans into motions. In many applications it is desirable to have a virtual human imitate the behavior of a real person while at the same time being subject to the physics of its own virtual environment. While this approach preserves the plausibility of the animation, it also demands for special algorithms which can abstract actions

An Animation System for Imitation of Object Grasping in Virtual Reality

67

from raw motion capture data. Such algorithms have long been studied in the robotics and Artificial Intelligence literature; a survey on this topic was published by Jung et al. [1]. Pollard and Zordan [7] present an algorithm, which extracts a set of parameters from motion capture data. The parameters are in turn fed into a controller used for performing active and passive grasping. The system presented in this paper combines several of the above ideas with a hierarchical architecture for animation synthesis. In contrast to previous work, the motion capture data is not only used for parametrizing particular controllers or behaviors. Instead, the motion is analyzed, abstracted and transformed into an intermediate representation. This allows the recorded action to be performed in different environments and settings.

3

Animation Synthesis

This section presents a hierarchical control architecture for interactive animation of virtual humans, with a particular focus on animating object manipulation tasks. The architecture is currently implemented as an extension of the VR system Avango [8]. In principle, however, it is independent of any particular VR software. At its core, the framework consists of hierarchically organized modules at three distinct levels: plans, behaviors, and motor programs. Fig. 1 gives an overview of the control architecture. In general, the control flow for animation synthesis is as follows: High-level plans are abstract descriptions of action sequences. The selection and execution of plans can be triggered by issuing an appropriate command from Avango’s integrated Scheme interpreter. Execution of a plan will instantiate several behaviors which in turn start one or more motor programs to control the basic movements of the virtual human, i.e. changing bone rotations of the virtual character’s skeleton. Behaviors are notified about collisions occurring between body parts and scene objects, e.g. when grasping an object. They can decide if it is necessary to stop movement when collisions occur. Behaviors terminate when all motor programs finished. A plan stops when all behaviors were terminated. Motor Programs. On the lowest level of the animation synthesis hierarchy, motor programs control the joint values of a virtual human’s bones. The specification of a motor program defines different parameters, e.g. target joint values. Additionally a scope has to be defined, i.e. the set of bones influenced by its execution in the kinematic chain of the skeleton. Motor programs do not directly set joint values, but instead modify so-called motor states. A motor state represents the orientation of a bone at a certain time. After a motor program has calculated a new state, the state is passed to an arbiter component. As many motor programs may run in parallel, conflicts may occur when calculating motor states referring to the same bone. Conflicts are resolved by the arbiter through motor state priorities, which were set by associated motor programs. The arbiter then sets the consolidated joint values. This way resolution of conflicting bone movements occurs on a fine-granular motor state level rather than on the complete

68

M. Weber et al.

Command Interpreter

Plans

Avango Collision Detection

Behaviors

VCollide Motor Programs

Virtual Human Deformation Cal3d

Animation Synthesis

Fig. 1. Control architecture

scope of the motor program, i.e. set of influenced bones, or on the higher level of behaviors. Two types of motor programs are currently supported. One type interpolates motor states from current states to specified goal states, defining the end-posture for a part of the virtual human. The second type uses inverse kinematics to let a kinematic chain of arbitrary length reach a certain position. Behaviors. Mid-level behaviors are used to accomplish certain sub-actions of object manipulations, like closing a hand or reaching for an object. To realize movements of the virtual human, they select and instantiate motor programs, that work directly on the kinematic chain of the virtual human. Behaviors are parametrized, allowing them to reactively adapt to changing virtual scenes. E.g. moving objects can be reached by only setting the object’s name as a parameter and letting the behavior calculate where the object is currently located. There are currently four types of behaviors to accomplish different tasks: Open a hand, close a hand to grasp an object, reach a certain point or object, and move joints to a given end or rest posture. These behaviors can be adjusted with parameters like begin and end time or the object to interact with. Plans. On the highest level, plans describe complete manipulation tasks, e.g. grasping an object, in a declarative fashion. They consist of plan steps, which can be executed either sequentially or in parallel. Plans are described in an XML-based language. Fig. 2 shows a simple example of such a plan description. Instantiating the plan grasp-object with the object Ball-1, the grasp port ball-port, the hand RightHand and the grasp type spherical as parameters results in selecting the plan out of a plan database and refining the plan by substituting variables in the plan with the parameters (variables in this example are $OBJECT, $PORT, $HAND and $GRASPTYPE respectively). It is not necessary for the triggering command to specify all parameters of a plan: For example, if just the target object of the grasp is specified, then suitable values for $PORT, $HAND and $GRASPTYPE will be automatically added, e.g. using default values or object annotations (see below for a description). In the example, first a parallel


69

grasp-object <parallel> Reach Body <param name="object">$OBJECT <param name="grasp-port">$PORT GraspOpen GraspClose <param name="grasp-type">$GRASPTYPE Fig. 2. Example for a plan: grasping an object

set of plan steps is instantiated for reaching the grasp port ball-port on the object Ball-1 and for opening the hand. The type , the name of the virtual character affected , and several parameters <param> can be specified for such behaviors. After reaching the object with an open and correctly oriented hand, the hand is closed (behavior type GraspClose) using a specified or automatically inferred grasp type from a grasp taxonomy. When the object is grasped, the plan finishes. Annotated Objects. To support movement generation, semantically annotated objects are used in behaviors to, e.g. find a proper place on a chosen object to grasp. Grasp ports define positions on an object, where grasping is possible. Additionally, they describe the shape of the object at that position so it is possible to choose an appropriate hand shape for grasping. Annotated objects are akin to the concept of smart objects in literature (e.g. Kallman and Thalmann [2]), differing in the way that we do not explicitly describe behaviors in the objects. Annotated objects are represented in XML structures; see Fig. 3 as an example. In this example the object type Ball is defined. Besides a specified scale factor (scale), one grasp port with a spherical form (type="SPHERICAL") is defined. The grasp port has a unique name ball-port, a certain position (position) and a radius (radius). Finally, the graphical model to load for the annotated object is specified. Virtual Human Body and Hand Model. In skeletal animation, a virtual character’s body is represented by a polygon mesh and a skeleton composed of bones. In our system, the virtual humans’ skeleton structures are based on the joint hierarchy of the H-Anim standard (http://www.hanim.org). Every bone is

70

M. Weber et al.

Ball <scale>0.00595 0.00595 0.00595 ball-port <position>0.0 0.0 0.0 5.775 <model>models/iv/ball2.iv Fig. 3. Example for an annotated object: A Ball

represented with a name and its orientation and translation relative to it’s parent bone. Several virtual characters can populate a scene, acting independently of each other. Additionally, virtual characters can be composed of several independent articulated models. We use this method to integrate different models for body, right and left hand. In this way, the modelling of the virtual human’s body is separated from the modelling of its hands. Thus, the hands, which are more complex to model, can be easily re-used with different bodies. Fig. 4 shows the skeletons of a virtual human and of the right hand.

Fig. 4. Skeletons of virtual human and right hand

Since the focus of our animations is on object manipulation, particular effort has been put into a detailed virtual hand model. Just as the virtual character’s body, the hand model is a skinned mesh, deformed by an underlying skeleton structure. Each finger consists of three segments/bones, with the proximal joints having two degrees of freedom (flexion and pivot) and the others only having one (flexion). Additionally, joint angle constraints are enforced, to permit natural looking movements only. For interaction and collision detection with scene objects, the hand model has been fitted with spherical collision sensors (see Fig. 4). Currently, there is one


71

sensor in the palm, one sensor in each finger segment and an additional one at each finger tip. These sensors fire collision events, as soon as they touch a scene object. This information is used by motor programs and behaviors to adjust the animation in turn to avoid intersection of the hand with the objects.1 Collision detection is currently implemented using VCollide [9]. For the visual part, to realize skeletal animation and mesh deformation according to the bone movements, the character animation library Cal3d (http://cal3d.sourceforge. net) is used.

4

Imitating Grasp Actions

In the previous section, the process of an autonomous movement generation for virtual character animation was described. Movement is triggered by instructing the virtual character to execute specific plans. In this section, we introduce an alternative method in which the virtual character learns to imitate object manipulations performed by a human VR user. To this end, the interactions of the VR user with scene objects are analyzed and transformed to plans, that can be executed by virtual characters using the animation synthesis architecture described above. Animations are thus generated by recording and analyzing movement data from virtual reality input devices. Such devices could be, but are not limited to, optical position trackers and data gloves. The recorded data is processed to calculate certain features to describe a grasp. These features are hierarchically organized from low level features – describing low level data like the joint values of a human skeleton – to high level features, where the grasp is only described by a category belonging to a grasp taxonomy2 . Low level features need no calculation when movement of a real human is analyzed. High level features achieve high abstraction of human movement by just providing grasp types and objects to grasp. Such a high abstraction makes it possible to adapt virtual human movement to changing scenes. Fig. 5 gives an overview of the whole process of imitating grasp actions. The two parts, movement analysis with the focus on grasp event detection and movement synthesis for the reproduction of actions, will now be described. 4.1

Analyzing User Actions for Grasp Events

Analyzing movement data involves the following steps: segmentation of the hand movement, mapping of glove sensor values to joint rotation values and calculating grasp features out of the posture of the hand and the grasped object. This process 1

2

A further use of the hand model is to support the interactions of human VR users with the virtual world, particularly for grasp analysis (see Sec. 4). Here, the user hand is projected into the virtual world to provide visual feedback to the user as well as to find contact points with scene objects during grasps. In this case, joint angles are deducted from data glove sensor input and then mapped directly to the joints of the virtual hand. Grasp taxonomies classify grasps into several categories, describing the features of a grasp, e.g. Cutkosky [10] or Schlesinger [11].

72

M. Weber et al.

Movement Data

Segmenting Movements

Detecting Features

Generating Grasp Events

Analysis: Detecting Grasp Events Grasp Events

Selecting Plans

Refining Plans

Generating Movement / Animation Synthesis

Synthesis: Reproduction from Grasp Events

Fig. 5. Imitation process

is subject of other work and is not the focus of this paper. But the results are used to imitate human movement, especially in the domain of grasping objects. A novel representation form is used to store the results: Grasp events. They contain grasp features, the grasp type and a time stamp. The time stamp defines when the grasp happened with respect to the time when the manipulation sequence began. Low level features include: joint values, contact points, grasped objects, and hand position. High level features denote the resulting category of a grasp w.r.t. a certain taxonomy. Grasp events can be stored in XML files for the purpose of later playback. An example for the structure of such an XML file is shown in Fig. 6. 4.2

Reproducing Actions from Grasp Events

To animate the virtual character and reproduce the recorded sequence of grasp events, the events have to be converted into executable plans. This is done in two steps (see the synthesis part of Fig. 5): First, a plan is selected, based on the received grasp events. As a simple example, this could be the plan grasp-object whenever events of type grasp are received. After that, the selected plan is refined by means of the features contained in the grasp events. Refinement is done by filling unspecified parameters of the plan with reasonable values. This can be done in a very simple way by providing the last value that was received for a parameter. Another conceivable possibility would be to use the information provided by grasp events and the actions that already took place for computing new parameters. When only high level features, i.e. grasp taxonomy categories, are used to specify a grasp, a grasp database is used to look up the grasp type and retrieve features useful to generate a similar grasp by the virtual human. For now such useful features include the start and end postures of the grasp, see Fig. 7 for


73

<event-sequence> <event timestamp="3.895" type="grasp"> <joint-angle joint-id="r_index1">19.8306 -0.0865252 0.678805 0.729203 Ball-1 <pos>0.00730081 -0.0734563 0.0135953 Ball-1 0.0205884 0.211408 -0.97718 0 0.0805939 0.973855 0.212386 0 0.996533 -0.0831275 0.00301189 0 -0.1502 -0.626599 0.917001 1 right schlesinger spherical ... Fig. 6. An XML example for grasp events

Fig. 7. Two end postures from the grasp database: cylindrical and tip

an example of end postures for the two grasp categories cylindrical and tip. In addition to this, finger groups are used for describing similar behaving fingers. Fingers in a finger group stop whenever one of the fingers stops, caused e.g. by collision detection. Unfilled parameters in a selected plan are then refined by using these features. By execution of the refined plan, the virtual human performs the appropriate movement, using the framework described in Sec. 3. Behaviors, parametrized

74

M. Weber et al.

with the values computed by the refinement step, are instantiated, which use adequate motor programs to control virtual human movement.

5

Example

The whole process of movement generation by instruction will now be explained by considering the example of grasping a ball, as shown in Fig. 8. Grasps performed by a user are recorded and processed to generate a grasp event containing a spherical grasp for the ball Ball-1 with timestamp 3.895. To grasp the ball Ball-1, an appropriate plan is selected and parametrized to convert the features into parameters of the plan. In this plan, parallel behaviors for reaching the grasp port of the object and for opening the hand are executed. After finishing these behaviors, a behavior for closing the hand, i.e. grabbing the object, is executed. For reaching, the behavior will start a motor program that moves the virtual human’s hand to the object’s grasp port position using inverse kinematics. A scene information database is queried for information about the grasped object.

User Input

- glove values - hand position

Movement Analysis

generate Grasp Events

Grasp Events

- spherical grasp - Object: Ball-1 - timestamp: 3.895

Plan Selection

Plans

select Plan matching events

Plan: grasp(Ball-1,spherical)

execute Plan by instantiating Behaviors

Plan Execution

Close Hand

Open Hand

Behaviors

Motor Programs

Move Arm

decrease r_thumb1 flexion

decrease r_thumb2 flexion

...

Fig. 8. Example: Imitation of grasping a ball


75

In this example, this would be an annotated object of type ball with a spherical grasp port for Ball-1. Opening and closing of the hand will start motor programs that interpolate joint values to given goal motor states. These are goal angles for opening and closing the hand representing certain postures that describe an opened or closed hand respectively. When closing the hand, collision sensors are used to stop the movement of bones when fingers get contact with the annotated object.

6

Conclusion

We have presented a system for instructing virtual humans for the synthesis of virtual human-object interactions. A hierarchical three-level approach is used to generate accurate object manipulations from high-level commands. The system automatically instantiates behaviors and appropriate motor programs to control the virtual human. Using this framework, grasps performed by a real human in a virtual environment can be imitated. For this, grasps are recorded, analyzed for grasp features, and stored as grasp events. The recorded grasp events are used to select and refine appropriate plans, which in turn trigger lower level behaviors. In this way, interactions between the human and the environment can be recorded and imitated. Due to their high-level representation, recorded manipulations often naturally adapt to new situations without losing plausibility. In on-going work we are developing a more elaborate plan generator to create high level plans out of sequences of grasp events. With such plans it will further be possible to create a plan database containing several highly abstract representations of object manipulations in virtual environments. Additionally, we want to improve the refinement step for selecting plans. Missing parameters could, for instance, be computed based on predictors acquired through machine learning techniques. Learning could further be used for improved reproduction of different movement styles exhibited by different VR users. Acknowledgments. The research described in this contribution is partially sponsored by the DFG (Deutsche Forschungsgemeinschaft) in the Virtual Workers project.

References 1. Jung, B., Ben Amor, H., Heumer, G., Weber, M.: From Motion Capture to Action Capture: A Review of Imitation Learning Techniques and their Application to VRbased Character Animation. In: Proceedings ACM VRST 2006, Cyprus. (2006) 2. Kallmann, M., Thalmann, D.: Direct 3D Interaction with Smart Objects. In: Proceedings ACM VRST 99, London. (1999) 3. Sanso, R.M., Thalmann, D.: A Hand Control and Automatic Grasping System for Synthetic Actors. Computer Graphics Forum 13(3) (1994) 167–177 4. Kuffner, J., Latombe, J.: Interactive Manipulation Planning for Animated Characters. In: Proceedings of Pacific Graphics. (2000)

76

M. Weber et al.

5. Yamane, K., Kuffner, J.J., Hodgins, J.K.: Synthesizing Animations of Human Manipulation Tasks. ACM Trans. Graph. 23(3) (2004) 532–539 6. Douville, B., Levison, L., Badler, N.: Task-level Object Grasping for Simulated Agents. Presence 5(4) (1996) 416–430 7. Pollard, N., Zordan, V.B.: Physically Based Grasping Control from Example. In: Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. (2005) 8. Tramberend, H.: Avocado: A Distributed Virtual Reality Framework. In: Proceedings of IEEE Virtual Reality. (1999) 14 9. Hudson, T., Lin, M., Cohen, J., Gottschalk, S., Manocha, D.: V-COLLIDE: Accelerated Collision Detection for VRML. In: Proceedings of VRML’97. (1997) 10. Cutkosky, M.: On Grasp Choice, Grasp Models and the Design of Hands for Manufacturing Tasks. IEEE Trans. on Robotics and Automation 5(3) (1989) 11. Schlesinger, G.: Der Mechanische Aufbau der K¨ unstlichen Glieder. In Borchardt, M., et al., eds.: Ersatzglieder und Arbeitshilfen f¨ ur Kriegsbesch¨ adigte und Unfallverletzte. Springer-Verlag: Berlin, Germany (1919) 321–661

Generating Different Realistic Humanoid Motion Zhenbo Li1,2,3, Yu Deng1,2,3, and Hua Li1,2,3 1

Key Lab. of Computer System and Architecture, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100080, P.R. China 2 National Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100080, P.R. China 3 Graduate University of Chinese Academy of Sciences, Beijing 100039, P.R. China {zbli, dengyu, lihua}@ict.ac.cn

Abstract. Different realistic humanoid motion can be used in vary situations in animation. It also plays an important role in virtual reality. In this paper, we propose a novel method to generate different realistic humanoid motion automatically. Firstly, eigenvectors of a motion sequence is computed using principle component analysis. The principle components are served as “virtual joints” in our system. The number of “virtual joints” can be used to control the realistic level of motions. After given the “virtual joints” number, the actual joints’ parameters of new motion are computed using the selected “virtual joints”. The experiments illuminate that this method has good ability to generate different realistic level motions. Keywords: PCA, DOF, Motion Capture.

1 Introduction Humanoid motion plays an important role in games, movies etc. With its wildly use, different realistic motions are needed in many cases. Traditionally, in order to get a motion sequence, animators need to specify many key frames. However, creating these key frames is extremely labor intensive. Motion capture technique is triumphantly used to produce motion scripts recently, which can get motion data from living human beings directly and expediently. Though motion capture lightens the burden of animators, it has two limitations: 1.people can only use the motion scripts recorded beforehand. 2. When the body proportion in animation is not suitable with the recorded body proportion; the data must be retargeted. These two limitations counteracted its using. Humanoid motion is defined as the combination of a set of postures fi (i=1, 2... n), each posture is represented by a set of degrees of freedom (DOF). The motion space is often high, generally fifty to sixty. For many behaviors, the movements of the joints are highly correlated; this makes it possible to change the movement reality through controlling the energy of motions. The goal of making motion animation is reality, but at many situations such as in a cartoon system, people often need nonrealistic motions. In this paper, we present a method to generate different realistic level motions through selecting the dimensions Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 77 – 84, 2006. © Springer-Verlag Berlin Heidelberg 2006

78

Z. Li, Y. Deng, and H. Li

of motion space. The method could help people generate new nonrealistic motions from a given motion sequence. The rest of this paper is organized as follows: Section 2 reviews the related work of human motion space reducing, which is the base of our work. In section 3, we described our method of generating different realistic humanoid motions. We give the experiment results in section 4. And finally in section 5, we conclude with a brief summary and discussion of future works.

2 Related Works Automatically generating animations is an important problem and many good solutions have been proposed [1, 2, 3]. In the process of making animation, animators are often confused by the high motion dimensions because it’s not easy to control the data in high dimension directly. Recently people find that the movements of the joints are highly correlated for many behaviors [4, 5]. These correlations are especially clear for a repetitive motion like walking. For example, during a walk cycle, the arms, legs and torso tend to move in a similar oscillatory pattern: when the right foot steps forward, the left arm swings forward, or when the hip angle has a certain value, the knee angle is most likely to fall within a certain range. And these relationships hold true for more complex motions as well [4]. Using the correlated information, we can reduce the dimensions of working space. It also means reducing energy of the motion space. And motion space reducing is the base of our work. Degrees of freedom are correlated with each other, many research works are also benefited from the observation. Alla Safonova et al. [4] proved that many dynamic human motions can be adequately represented with only five to ten degrees of freedom. He used lots of motions with similar behavior to construct a lowdimensional space to represent well other examples of the same behavior. Arikan[6] used this observation to implement the compression of motion capture databases and got good compression results. Popović and Witkin [7] showed that significant changes to motion capture data can be made by manually reducing the character to the degrees of freedom most important for the task. Howe and colleagues [8] published one of the earliest papers on using global PCA to reduce the dimensionality of human motion. They incorporated the reduced model into a probabilistic Bayesian framework to constrain the search of human motion. Sidenbladh and colleagues [9] reduced the dimensionality of the database using global PCA and then constrained the set of allowable trajectories within a high-dimensional state space. Pullen and Bregler[4] used this observation for motion synthesis/texturing and Jenkins and Mataric [10] used it for identifying behavior primitives. There are many methods to reduce the dimensions like PCA, Kernel PCA [11], Isomap[12], Locally Linear Embedding[13] etc. These methods could implement linear or nonlinear dimension reduction. We attempts to apply these dimension reduction approaches to generate different realistic motions. With the dimension adding or reducing, we could get different realistic level motions. And these motions contain the basic intrinsic information of the original motion.

Generating Different Realistic Humanoid Motion

79

3 Proposed Method For a motion animation sequence, we compute eigenvectors V (v1, v2 … vn) from the sequence. The principle components can be represented by the eigenvectors and their coefficients, which we named “virtual joints”. Through controlling the number of “virtual joints”, we could get different realistic level of humanoid motion sequences. 3.1 Motion Definition Motion M is defined as the combination of a set of postures fi (i=1, 2... n), which is organized according time axis (fig.1). It could be a simple motion (such as walking， running) or a complex motion (such as playing basketball).

Fig. 1. Sketch map of motion definition

3.2 Different Realistic Motion Generating After we got a motion sequence M, which might be represented by fifty or sixty DOF. It’s hard for us to generate similar motions in such high dimension. PCA method is used to reduce the motion space. 3.2.1 Principal Component Analysis Each frame fi (i=1, 2... n) saved in the captured file is a point of fifty or sixty dimension. Because joint movements of human body are highly correlated, we can synthesize some main measures from the DOF of human model. These measures contain the primary information of the motion. So we can use these measures to describe the captured motion data. PCA method is useful to get such main measures. After gotten a motion capture file, we assume the frame length is n and the DOF of the human model is p. The data matrix can be represented as:

⎛ x11 K x1 p ⎞ ⎜ ⎟ X = ( x1 , ⋅ ⋅ ⋅, xn )′ = ⎜ M O M ⎟ ⎜x L x ⎟ np ⎠ ⎝ n1

(1)

For PCA to work properly, we have to subtract the mean from each of the data dimensions. The mean subtracted is the average across each dimension. x=

1 n 1 n 1 n xi = ( ∑ xi1 ,..., ∑ xip ) ′ = ( x1 ,..., x p ) ′ ∑ n i =1 n i =1 n i =1

(2)

80


So, all the xij values have x j subtracted. This produces the captured data set whose mean is zero. We use formula 3 to calculate the covariance matrix after the captured data standardization. The covariance matrix is a measure to find out how much the dimensions vary from the mean with respect to each other.

1

n

∑ = n − 1∑ ( x i =1

i

− x)( xi − x)′

We can calculate the eigenvectors and eigenvalues of the covariance matrix These are rather important, as they tell us useful information about our data. Suppose the eigenvalues of ∑ as:

λ1 ≥ λ 2 ≥ L ≥ λ p > 0

(3) .

(4)

And the corresponding standard eigenvectors are: u1, u2, …up The eigenvector with the highest eigenvalue is the principle component of the data set. So we give the principal component formula as:

Fi = ui ' x (i=1... n) Each Fi and Fj are orthogonal when i,j=1,...,n and So the principal component formula is:

(5)

i ≠. j

F1 = u11 x1 + u21 x2 + L + u p1 x p L Fm = u1m x1 + u2 m x2 + L + u pm x p

(6)

Here F1 is the maximal of the variance and called first principle component, F2 is the second principle component… and Fm is the m ordered principle component. After we get the principle components, we can calculate the principle component’s contribution factor. The accumulation contribution factor from first principle component to m principle component is: m

Em =

∑λ i =1 p

i

∑λ i =1

(7)

i

Em is an indicator of how much information is retained by projecting the frames on the m-dimensional space. If E m ≥ 85% , we can regard the m-dimensional space contains original space’s information basically. 3.2.2 Motion Representation Using PCA, we can get the principle components of a motion sequence. A motion could be represented as below:


⎛ y1 ⎞ ⎜ ⎟ ⎛ a11 ⎜ y2 ⎟ = ⎜ M ⎜M ⎟ ⎜⎜ ⎜ ⎟ ⎝ an1 ⎝ yn ⎠

⎛ a11 ⎜ Here M ⎜ ⎜a ⎝ n1

K O L

K O L

⎛ x1 ⎞ a1n ⎞ ⎜ ⎟ ⎟ x M ⎟⎜ 2 ⎟ ⎜M ⎟ ann ⎟⎠ ⎜ ⎟ ⎝ xn ⎠

81

(8)

a1n ⎞ ⎟ M ⎟ is the matrix of eigenvectors and ( x1 , x2 ,L , xn ) ' is the ann ⎟⎠

vector of coefficients, which is the value of joints rotation at every frame.

( y1 , y2 ,L, yn ) ' is the vector of principle components. We name the element of the vector ( y1 , y2 ,L , yn ) ' as “virtual joints’, which could be used to control the

realistic level of the motion. The more the “virtual joints” selecting, the more realistic level we could get. “Virtual joints” could also represent the energy distribution of the motions in the orthogonal space. After selecting the numbers of “virtual joints” as m (m ≤ n), we could compute the parameters of actual joints of every frame as follow:

⎛ x1 ⎞ ⎜ ⎟ ⎜ x2 ⎟ = ⎜M ⎟ ⎜ ⎟ ⎝ xn ⎠

⎛ y1 ⎞ ⎜ ⎟ y A⎜ 2 ⎟ ⎜M ⎟ ⎜ ⎟ ⎝ ym ⎠

⎛ a11 ⎜ Here, A is a generalized inverses matrix of M ⎜ ⎜a ⎝ n1

(9)

K O L

a1n ⎞ ⎟ M ⎟ . It is a matrix of ann ⎟⎠

n*m. From formula 9, we could get the x values of every frame in the motion sequence. Thus we could get different realistic motion sequences through controlling the number of “virtual joints” m. When m=n, we could get the original motion.

4 Experiments We used the CMU Mocap database in our experiments. The full human movement dimension is 62 degrees. In our experiments we did not care about the three position dimensions; the four dimensions of left and right clavicles and the two dimensions of right and left fingers are also careless. Thus the actual space we used is 53 degrees.

82


We select a jump motion sequence, which contains 300 frames. Firstly, we compute the eigenvectors of the motion using the method introduced in section 3. The number of “virtual joints” m we selected are 1,3,6,13,23,33,43,53 respectively. The relationship of “virtual joints” number and its energy containing percent is shown in table 1. Table 1. The relationship of “virtual joints” numbers and the ratio of energy it containing Number Percent

1 41.9%

3 76.8%

6 91.7%

13 98.4%

Fig. 2. Energy ratio chart

23 99.8%

33 99.99%

43

≈ 100%

53 100%

Fig. 3. The value of first “virtual joints”

The relationship of “virtual joints” numbers and the energy it containing is also shown in figure 2. When the “virtual joints” number changes from 6 to 53, the energy was not change greatly. In figure 3, we gave value of the first “virtual joints” changing according the frame numbers. From the experiment results, we could see all of the motions containing the basically motion configuration. But they have different view realistic results. The results of motions generated using our method is shown in figure4. The figure also reflects the relationship of “virtual joints” and their realistic level. When we selected the number of “virtual joints” as 1, the motion we generated can be recognized as jump hazily; when the number of “virtual joints” up to 13, the motion is more like a mummy jump; when the number up to 53, we could get the original motion sequence. We could see it’s a holistic method to generating humanoid motions. From table 1 and figure 4, we could also derive that the details determined the reality of motions. Throwing away some details could help us get different realistic motions like cartoon motion etc.

5 Discussion and Future Works Though generating realistic humanoid motion is important in animation and virtual reality, different level realistic humanoid motion is often required in many


Frame Num VJ Num

1

85

153

178

213

262

83

298

1

3

6

13

23

33

43

53

Fig. 4. The results of motion generating using different “virtual joints” numbers

applications. For example, cartoon motion is often used in many systems. We proposed a method to generate different level realistic humanoid motion automatically. This could help lighten the burden of animators and reuse the existed animation sequence.

84


In future work, we will try to use other dimension reducing methods to compute the “virtual joints”. The relationship of “virtual joints” with different motion types and the motion details influence to motion reality are other aspects to be particularly studied in future. Acknowledgements. This work was supported in part by National Natural Science Foundation of China (grant No: 60533090). The data used in this project was obtained from mocap.cs.cmu.edu. The database was created with funding from NSF EIA0196217.

References 1. Wilkin. A. and Kass. H: Spacetime Constraints. In Proceedings of Siggraph88, 1988: 15968 2. Shin. H. J, Lee. J, Gleicher. M. and Shin. S.Y. Computer puppetry: An importance based aproach. ACM Trans. On Graphics, 20(2), 2001: 67-94 3. Gleicher. M: Retargeting motion to new characters. In Proceedings of Siggraph98, 1998: 33-42 4. Pullen. K, and Bregler. C: Motion capture assisted animation: Texturing and synthesis. In Proceedings of Siggraph02, 2002: 501–508 5. Alla Safonova, Jessica K. Hodgins, Nancy S. Pollard: Synthesizing physically realistic human motion in low-dimensional, behavior-specific spaces. ACM Trans. On Graphics.23 (3), 2004:524-521 6. Okan Arikan: Compression of Motion Capture Databases. To appear in Siggraph 2006 7. Popovi ć. Z., and Witkin A. P: Physically based motion transformation. In Proceedings of Siggraph 99, 1999:11–20 8. Howe. N, Leventon. M. and Freeman. W: Bayesian reconstruction of 3d human motion from single-camera video. In Advances in Neural Information Processing Systems 12. 1999:820-826. 9. Sindenbladh, H, Black. M. J., and Sigal, L: Implicit probabilistic models of human motion for synthesis and tracking. In European Conference on Computer Vision, 2002:784-800 10. Jenkins, O. C, and Mataric. M. J: Automated derivation of behavior vocabularies for autonomous humanoid motion. In AAMAS ’03: Proceedings of the second international joint conference on Autonomous agents and multiagent systems, 2003:225–232 11. B. Scholkopf, A. J. Smola and K.-R. Muller: Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10(5), 1998:1299–1319 12. J. B. Tenenbaum, V. de Silva and J. C. Langford: A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500), 2000:2319–2323 13. S. T. Roweis and L. K. Saul: Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500), 2000:2323–2326

Handheld AR for Collaborative Edutainment Daniel Wagner1, Dieter Schmalstieg1, and Mark Billinghurst2 1

Graz University of Technology Institute for Computer Graphics and Vision, Inffeldgasse 16 Graz, 8010 Austria {wagner, schmalstieg}@icg.tu-graz.ac.at 2 University of Canterbury HIT Lab New Zealand Private Bag 4800 Christchurch 8004 NZ [email protected]

Abstract. Handheld Augmented Reality (AR) is expected to provide ergonomic, intuitive user interfaces for untrained users. Yet no comparative study has evaluated these assumptions against more traditional user interfaces for an education task. In this paper we compare the suitability of a handheld AR arts-history learning game against more traditional variants. We present results from a user study that demonstrate not only the effectiveness of AR for untrained users but also its fun-factor and suitability in environments such as public museums. Based on these results we provide design guidelines that can inform the design of future collaborative handheld AR applications. Keywords: AR education, collaborative AR, usability studies.

1 Introduction Augmented Reality (AR) interfaces have been developed for many application areas such as medicine, engineering and gaming. Recently, PDAs and cell phones have reached a level of performance where they can be used for mobile AR. With built-in cameras and 3D graphics chips, these handheld devices can readily be turned into video-see through magic lenses, providing an AR view of the user’s real world. In our ongoing work to build a handheld AR platform [1], we have targeted edutainment as a suitable application area for real world deployment of multi-user AR. Our efforts towards developing high-quality AR edutainment applications is driven by the belief that collaborative handheld AR can surpasses traditional media in user satisfaction, intensity of collaboration and learning efficiency. Although edutainment AR applications are becoming more common, few studies have compared learning in an AR setting to more traditional educational tools. Research is needed to explore the educational value of mobile AR applications and inform the design of these interfaces. In this paper we present a user study comparing learning in a collaborative handheld AR environment to more traditional PC interfaces. This was done using an application, Virtuoso, that has similar gameplay across a handheld AR platform, desktop computer and a set of playing cards. Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 85 – 96, 2006. © Springer-Verlag Berlin Heidelberg 2006

86

D. Wagner, D. Schmalstieg, and M. Billinghurst

2 Related Work Our work on handheld collaborative AR games in a museum is built on related work in handheld AR, collaborative AR, AR games, AR for museums, and comparative AR user studies. In this section we briefly review past work in these areas. The Touring Machine [2] developed at Columbia University pioneered the concept of mobile AR almost 10 years ago. Since then, mobile AR interfaces have moved from cumbersome backpack computers to handheld AR devices. Initially these were tethered handheld AR displays such as Fitzmaurice’s Chameleon [3] and Rekimoto’s NaviCam [4] but in 2003 the first author developed the first fully self-contained handheld AR application [1]. Originally based on a PDA port of ARToolKit [6], this work has since then been extended into a full blown cross-platform version of Studierstube [7]. Most recently, mobile AR interfaces have been developed for cell phones by Möhring [8] and Henrysson [9]. Around the same time, Augmented reality as a collaboration tool emerged in a number of “shared space” systems [10][11][12][13] for face to face collaboration. Other work focuses on remote collaboration making use of multiple possibilities of mixed reality continuum [14] simultaneously [15][16]. AR as a new medium is attractive for education institutions such as museums. The incorporation of AR enhanced exhibits can range from a single high-performance AR display [17] to an entire exhibition realized using AR special effects [18]. Mobile AR technology can offer an attractive replacement for the traditional audio-tape tour guide. Tour guides are a recurring application theme for mobile AR research, partially because they show the strength of mobile AR, namely to present useful information registered to static locations in the real world, such as historic edifices. Some examples of outdoor AR guides are Situated Documentaries [19], ArcheoGuide [20], GEIST [21] and Signpost [22]. Work on mobile indoor AR has specifically targeted museums, for example the Guggenheim Bilbao museum [23] or Sweet Auburn [24]. As can be seen from the related work mobile AR interfaces have migrated from bulky backpacks to self contained handheld systems. Similarly the applications that people are using these systems for have evolved and more and more people are interested in educational use of AR technology. Despite this there have been few examples of user studies comparing education in a mobile AR setting to more traditional educational experiences. Our work is unique because it is the first paper that compares education with a handheld AR platform to more traditional tools. The application itself is also interesting as it provides one of the first examples of using handheld AR as a platform for producing educational content in a collaborative setting.

3 The Handheld AR Platform We have created a component-based software architecture for the development of PDA-based handheld AR applications. This is the first feature-complete version of the Studierstube software framework that runs cross-platform on Windows, Linux and Windows CE. The technical details of the platform are beyond the scope of this paper, see [5] for more details. This section summarizes the key capabilities of the Studierstube framework for handheld devices.

Handheld AR for Collaborative Edutainment

87

Graphics and multimedia: OpenGL ES is an emerging standard for 3D graphics on embedded devices. Klimt1 is a software library that executes OpenGL compliant software on top of a third party hardware OpenGL ES implementation or Klimt’s own built-in software renderer. Using Klimt, it is possible to execute a high-level objectoriented scene graph library such as Coin2; we call this version Coin ES. Tracking: We have developed ARToolKitPlus3 as an optimized successor to the popular ARToolKit [6] marker tracking library. ARToolKitPlus shares the basic working principles with its ancestor, while the code has been rewritten in fixed point arithmetic and incorporates several novel computer vision algorithms. The resulting code tracks in real time on a high-end PDA (5ms per image on the Axim X51v). Networking: Muddleware is a networking middleware loosely inspired by the idea of a Tuplespace. It provides state synchronization for multi-user applications, persistency and resource streaming. Application framework: The Studierstube application manager coordinates the various system components mentioned above. It loads and executes concurrent applications.

4 Art History Application: Virtuoso Using the Handheld AR framework we implemented an educational game called Virtuoso. Besides the AR version, the game was also implemented in Macromedia Flash on a desktop PC and as a paper game without any technology. The players’ objective in the Virtuoso game is to sort a collection of artworks according to their date of creation along a timeline – left means earlier and right means later. The timeline has a number of slots, each of which can hold exactly one artwork. Initially all slots are randomly filled with artworks. Each player can pick up one item from a slot and put it into another empty slot. If the desired target slot is full, the player can ask a team member to help by picking up the item in the target slot. When taking an item from the timeline, the player is provided with basic information about the item, such as its name and a brief description. The player can ask Mr. Virtuoso, the virtual arts expert, for more information about the item. The game ends when all items are correctly located on the timeline. The art history application features an overall selection of 20 artworks from which the game master can select an arbitrary subset for play. The set of artworks covers a range from 20.000 BC to 2004 AD including pieces from western and eastern culture. We specifically created polygonal 3D models for each artwork. Most items are textured; some include audio or video features. The PDA version of the Virtuoso game shows a video-see through AR scene. It runs on an arbitrary number of PDAs which are synchronized via WiFi using our Muddleware collaboration running on a hidden PC. Paper fiducial markers placed 1

Klimt: open source 3D graphics library for mobile devices, http://studierstube.org/klimt Coin3D: open source scene graph renderer, http://www.coin3d.org 3 ARToolKitPlus: http://studierstube.org/handheld_ar/artoolkitplus.php 2

88


along a wall are used to mark the slots of the timeline (Figure 1). Every marker can carry one artwork which is only visible on the player’s PDA. A player can pick up any artwork from the wall onto her PDA, by clicking on the artwork on the display. Next she can drop it onto a free position on the wall by clicking on an empty marker on the display. While an item is located on the PDA, the player can access basic information about it such as its name. To get more information the player can place the object onto the desk of Mr. Virtuoso, an animated virtual 3D character who plays the game’s arts expert (Figure 3). Mr. Virtuoso will then be prompted to provide his expertise on the subject through the use of text, audio playback and animation. The player can take back the artwork onto her PDA at any time. After an artwork is placed at its correct position, it cannot be moved again. Besides the explanation from Mr. Virtuoso, the game engine can provide more explicit help by showing arrows pointing “left” and “right” next to the artwork if it should be placed earlier or later on the timeline (Figure 2). Furthermore, the game engine can display an item’s date of creation when the item is placed at its correct location on the timeline.

Fig. 1. Using the AR PDA version

Fig. 3. PDA version

Fig. 2. Virtual arrows providing game hints

Fig. 4. PC version

Fig. 5. Paper version

The desktop PC version was programmed in Macromedia Flash (see Figure 4). Players can move artworks on the timeline displayed on the screen using drag and drop operations with the mouse. Consequently the PC version can only be operated by one player at the time. While this restricts the user interface of the game, it provides a more realistic example of typical PC-based edutainment application or museum installation. As usual, when an item is located at its final position its date of creation

Handheld AR for Collaborative Edutainment

89

is displayed below of it and the item can no longer be moved. Moving an item to the left-top pane provides basic information, while the animated Mr. Virtuoso on the topright pane provides detailed explanations in text and audio. Items can be directly dragged from one slot on the timeline onto another. For the paper version of the game we printed pictures of the artworks on playing cards (Figure 5). On the front, the playing cards show the introductory text, while on the back more detailed descriptions by Mr. Virtuoso are provided. Players are only allowed to hold one playing card at a time and must only put it back into a free position. A human game master takes care that the game’s rules are not violated. The game master will also reveal an item’s date of creation when an item is located on its correct position on the timeline. So this version of the game has the same functional characteristics as the AR PDA and desktop PC versions.

5 Experimental Evaluation We compared our collaborative AR application with the two non-AR variants of the game (the desktop PC and paper-based version). In the experiment participants were grouped into teams of four. In each game, players had to sort seven items on the timeline. After a team finished its task they filled out a questionnaire about the game they just played including detailed art history questions about the items they just arranged, and how they felt about the game interface. Then all teams moved on to another version of the game. After the participants played all three versions of the game they filled out another questionnaire asking to rank the conditions in several categories. The introductory instructions to the participants emphasized the focus on collaboration and the need to learn about the artwork items, more than completing the task as fast as possible. To further motivate cooperation between players, players were told, that the goal of the game was to get a high team score, rather than personal scores on the arts history questions they had to answer.

6 Results There were 48 participants 26 female and 22 male, aged from 20 to 43. 25 people stated that they had never used a PDA before. The experiment lasted about one hour for each subject including introduction and a short finishing discussion. Data analysis was performed by using SPSS version 13 and the main effect was tested with repeated ANOVA. If a main effect was found, pair-wise post-hoc comparisons using the Bonferroni adjustment for multiple comparisons were performed. The questions the participants had to answer after each game can be grouped into four main categories: collaboration, easiness of the task, learning effect and fun factor. Subjects were asked to mark on a Likert scale of 1 to 7 how much they agreed or disagreed with a number of statements (1 = Strongly Disagree and 7 = Strongly Agree. We asked two questions about the way people collaborated: Q1: I collaborated intensively with my team members. Q2: I knew exactly what the others were doing.

90


Table 1 shows the average results for each of these questions. Subjects felt that they collaborated more in the Paper and PDA versions; an ANOVA test found a significant difference for Q1 (F(2,94)=3.94, P 0 THEN FOR each object o in Llocal (i , o)

IF T[ g u

] > 0 THEN multicast o to g u(i , o)

REPEAT − Situation 2. (region changed) The instance’s corresponding update region has been changed by region modifying services. The dirty flag variable fu(i,o) in the local instance record can represent this situation, and operation for the situation is defined as: ( i ,o )

FOR each object o in Llocal that f u

==true

multicast o to g u(i , o)

REPEAT − Situation 3. (object instance verification) A subscriber needs to know whether their discovered remote instances are still existing and valid in the federation execution on account of the consistency issue, so a periodically multicast of all the local instances is necessary even without any changes and requests. We define the operation as follows: FOR each object o in Llocal multicast o to g u(i , o) REPEAT We design our mechanism by assigning a periodically sending thread called “heartbeat” thread to all the publishers. The thread will multicast local instance information to related subscribers when any of the three situations occur. We choose a fixed time interval variable τ and make the heartbeat thread working as follows: 1. In every τ ms, do operation on situation 1. 2. In every 2τ ms, do operation on situation 2. 3. In a relatively long period (perhaps 10-15τ), do operation on situation 3. This approach achieves a balance between simulation requirement and host overload. The accurate value of τ is adjustable according to the actual condition of network. 3.4 Distributed Receiver-Side Matching and Feedback

After receiving update region information from the multicast group, the subscriber will take a receiver-side matching to determine its sl(i, o) (LoI of remote object instance). The LoI will be used for data filtering in the next step. For convenience of discussion, we introduce our matching algorithm under the condition of 2 dimension (x,y) and 3 Layers (NO_LAYER, LAYER_VISION, LAYER_ABOUT). Fig. 3 shows the matching rules for the three layers. The auxiliary regions for LAYER_VISION and LAYER_ABOUT is introduced, the former is congruent with the original subscription region, while the latter is achieved by dividing each range of the former region with a constant value EXTEND. The two auxiliary regions indicate the rules for region matching. The subscriber performs

340

S. Wei, Z. Zhou, and W. Wu

Fig. 3. Matching Rules for Three Layers

accurate matching for each pair of update region and subscription region, and for which the update region intersects with, the corresponding layer is returned. Otherwise, if no intersections occur, NO_LAYER is returned. At last, the subscriber sends the sk(i , o ) back to the publisher for the sender-side data filtering. Meanwhile, it also preserves one copy of

s k(i,o ) for

each discovered instance o

of class i in its Lremote in order to perform the receiver-side filtering. 3.5 Data Filtering Based on LoI

The aim of receiver-side region matching is to reduce the data transmission in data routing. During the data routing period, the publisher and subscriber will both take a filtering operation to discard irrelevant data. The following three steps are taken: − Step 1. sender-side filtering As discussed in [7], the publisher calculates Ru(i,o) according to the receiver-side feedbacks and its own Pm(i) . Since Pm(i ) contains necessary matching result of the subscriber, it becomes the measurement for sender-side filtering. According to the Update Rule Theorem mentioned in Section 2, the data packets whose η (ji,o) is greater than or equal to pm(i,o) will be discarded. − Step 2. data routing in the network based on multicast After sender-side filtering, the data packets reserved will be sent to the multicast group that the publisher holds. Multicast assures that all of subscribers interested in this data packet will receive it from their previously joined multicast groups. − Step 3. receiver-side filtering The sender-side filtering can hardly eliminate all the irrelevant data. Every subscriber checks data packets it receives by comparing s k(i,o) with the η (ji,o) attached in the update packets. Each instance o whose

sk(i,o) is

greater than η(ji,o) will be discarded, as

described in the Reflect Rule Theorem mentioned in Section 2.

An Interest Filtering Mechanism Based on LoI

341

4 Experiment and Result Analysis In order to verify the new mechanism, we have implemented it into our HLA compatible RTI software-BH RTI. Experiments are carried out based on BH RTI within a tank war-game scenario as shown in Fig. 4. The experiment was taken on 2 PCs with 3.0 GHz Intel Pentium 4 CPU and 512 MB RAM on 100M Ethernet.

...

End Point

Update Point

M ov

e

...

Start Point

Subscription Region: X Y 0 < i < 50

Fig. 4. Scenario of the Experiment

For comparison with the region-based approach, we record the region matching count (indicating load on hosts) at the receiver’s side as well as the data transmission amount on the network (indicating load on the network). The results of experiment are based on different number of instances and grid cell divisions, as shown in Fig.5 and Fig.6. The region-based approach is indicated by 1*1 grid cells. 6000 ) 5000 s e m i t ( t n 4000 u o C g n 3000 i h c t a M 2000 n o i g e R 1000

4500 ) s 4000 e g a k 3500 c a p ( t n 3000 u o m A 2500 n o i 2000 s s i m 1500 s n a r T 1000 a t a D 500

150*150 100*100 50*50 20*20 1*1(Region-Based)

0 1

10

50

100

Number of Instances

Fig. 5. Region Matching Count

0

150*150 100*100 50*50 20*20 1*1(Region-Based)

1

10 50 Number of Instances

100

Fig. 6. Data Transmission Amount

As expected, the new interest filtering mechanism leads to less region matching count and data transmission amount on the network than the region-based approach. As the number of instances increases, the advantage becomes obvious enough to

342

S. Wei, Z. Zhou, and W. Wu

enhance the performance of both the hosts and network in large-scale DVE. We also conclude from the result that both the region matching count and data transmission amount are improved by the increase of grid cell. The receiver side filtering is inevitable for accurate data reflection, as we analyzed in Section 3.5 Step 3. In large-scale DVE, the amount of data that the receiver filtered is also a factor to influence the load of hosts. We calculate the efficiency on receiving in our mechanism by means of dividing the data packages received by the packages eventually processed at the receiver’s side. Results are shown in Fig. 7. 0.8 0.7 g n0.6 i v i e c0.5 e R n o0.4 y c n e i0.3 c i f f0.2 E

150*150 100*100 50*50 20*20 1*1(Region-Based)

0.1 0 1

10 50 Number of Instances

100

Fig. 7. Efficiency on Receiving

As the resolution of grid division increases, efficiency on receiving increases accordingly. When grid division reaches 150*150, nearly 70% of the packages received by the receivers are effective. That means only 30% of all the packages are to be filtered by the receivers. The resources of computation are greatly saved. The experiment also demonstrates that we may take higher resolution grid division in the scope of specific exercise in order to reduce unnecessary data transmission.

5 Conclusion Data filtering is of vital importance in contributing a large-scale DVE. In this paper, an interest filtering mechanism based on LoI was proposed to implement data filtering in DDM Services and support for large-scale DVE. The mechanism adopts a distribute receiver-side region matching strategy and carry out data filtering based on LoI. We have implemented the mechanism in our RTI software. Experiment on region matching count, data transmission amount, and efficiency on receiving demonstrate that the mechanism is able to lessen the overhead of both hosts and network in large-scale DVE. Acknowledgments. The research is supported by the National Grand Fundamental Research 973 Program of China under Grant No.2002CB312105.

An Interest Filtering Mechanism Based on LoI

343

References 1. Martin R. Stytz: Distributed Virtual Environments, IEEE Computer Graphics and Applications, Vol.16, No.3 (1996) 19-31 2. IEEE. 2000. IEEE Std 1516-2000: IEEE standard for modeling and simulation (M&S) High Level Architecture (HLA) –Framework and rules. 3. IEEE. 2000. IEEE Std 1516.1-2000: IEEE standard for modeling and simulation (M&S) High Level Architecture (HLA) –Federate interface specification. 4. Morse, Katherine L, Bic, Lubomir, Dillencourt, Michael: Interest management in largescale virtual environments, Teleoperators and Virtual Environments. Vol. 9, No. 1(2000) 52-68. 5. Tan Gary, Xu Liang, Moradi, Farshad, Zhang Yusong: An Agent-Based DDM Filtering Mechanism, 8th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (2000) 374 - 381 6. Raczy, C,Tan, G.Yu, J: A sort-based DDM matching algorithm for HLA, ACM Transactions on Modeling and Computer Simulation, Vol. 15, No. 1 (2000.) 14-38. 7. Zhou Z, Zhao QP: Extend HLA with layered priority. In: Proceedings of the Spring Simulation Interoperability Workshop .Orlando FL (2003) Paper 03S-SIW-012. 8. MAK Technologies, http://www.mak.com/ 9. Wood D.D., Implementation of DDM in the MAK High Performance RTI. Proceedings of the Spring Simulation Interoperability Workshop. Orlando FL (2002) Paper 02S-SIW-056. 10. Defense Modeling and Simulation Office http://www.dmso.mil 11. Mark Hyett, Roger Wuerfel. Implementation of the Data Distribution Management Services in the RTI-NG, Proceedings of the Spring Simulation Interoperability Workshop. Orlando FL (2002) Paper 02S-SIW-044. 12. B. Helfinstine, D. Wilbert, M. Torpey, W. Civinskas, Experiences with data distribution managementin large-scale federations, Proceedings of the Fall Simulation Interoperability Workshop, Orlando FL(2001) Paper 01F-SIW-032. 13. Boukerche, A, Roy, A: Dynamic grid-based approach to data distribution management, Journal of Parallel and Distributed Computing, Vol. 62, No. 3(2002) 366-392. 14. pRTI, Pitch Technologies AB, http:// www.pitch.se 15. Tan, G.,Yusong Zhang, Ayani, R., A hybrid approach to data distribution management, Fourth IEEE International Workshop on Distributed Simulation and Real-Time Applications (2000) 55 – 61. 16. Boukerche, A. Kaiyuan Lu, Optimized Dynamic Grid-Based DDM Protocol for LargeScale Distributed Simulation Systems, 19th IEEE Internationa Parallel and Distributed Processing Symposium (2005) 17. Zhou Z, Zhao QP: Research on RTI congestion control based on the layer of interest. Journal of Software, Vol. 15, No. 1(2004) 120-130.

Study on Data Collaboration Service for Collaborative Virtual Geographic Environment Jun Zhu1,2, Jianhua Gong1, Hua Qi 3, and Tao Song4 1 State

Key Laboratory of Remote Sensing Science, Institute of Remote Sensing Applications, Chinese Academy of Sciences, Beijing 100101, P.R. China 2 Key Laboratory of Poyang Lake Ecological Environment and Resource Development Ministry of Education [email protected] 3 Surveying Engineering Department, Southwest Jiaotong University, 610031, P.R. China 4 GIS Lab, Department of Geoinformatic Engineering, Inha University, Yonghyundong 253, Namgu, Inchon, S. Korea, 402-751

，

Abstract. The use of Collaborative Virtual Geographic Environments (CVGE) is one of the most promising uses of virtual reality in geographic field. Data collaboration can support multi-directional sharing of data and files, allowing ideas and thoughts to be communicated interactively among users. Thus how to integrate diverse and separated data and offer more convenient and intuitive data collaboration service becomes increasingly significant for implementing collaborative work in the CVGE. In this paper, we firstly design a Grid-based CVGE service framework and discuss the data searching mechanism and accessing control. A data collaboration service was built to integrate and share system internal resources and improve collaboration work efficiency. Finally, we built a collaborative virtual environment and implemented data collaboration service in silt dams system planning on a case study area, Jiu-Yuan-Gou watershed of Loess Plateau, China. Experiment results prove that the scheme addressed in the paper is efficient and feasible.

1 Introduction In recent years, the amount of spatial data is becoming more and more tremendous [1]. However, a lot of data cannot be used efficiently because the rapidly increasing amount of data precludes the presentation of all given data items and the complexity of many datasets surpasses the user’s ability to identify the gist or the underlying concepts [2]. Essentially, solving geographic problem is a process of collaborative work among group members, which are often distributed in geographic space [3]. To facilitate the analysis of simulation data, a variety of experts may be needed. As an integrated technology, collaborative virtual geographic environment (CVGE) offers an intuitively and efficiently interactive visualization environment. The CVGE might vary in its representational richness from 3D geographic spaces, 2.5D and 2D environments to text-based environments, which allows geographically separated users to explore complicated spatial information and conduct collaborative work [4,5]. An essential achievement of the CVGE is that it combines the participants and the information it accesses and manipulates in a single virtual geographic space. Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 344 – 353, 2006. © Springer-Verlag Berlin Heidelberg 2006

Study on Data Collaboration Service for CVGE

345

Hence improving the efficiency of data collaboration is very important to keep synchronization in the CVGE collaborative work. In fact, many related works have been implemented to resolve some certain application cases. For example, a visualization environment in a single computer has been developed. So how to integrate data, procedures and algorithms to build and offer more convenient and intuitive data collaboration service in heterogeneous and geographically dispersed environments becomes increasingly significant for collaborative work in the CVGE. The development of Grid technology endows the CVGE with a promising future. Grid’s concept is coordinated at sharing resources and solving problems in dynamic, multi-institutional virtual organizations [6]. The Grid computing system can provide different kinds of services for users [7,8]. Essentially, Grid user sees a large virtual computer despite of the fact that the resources are geographically distributed and connected over the worldwide networks [9]. Based on the concepts and technologies of Grid and Web Service communities, this architecture defines the uniform exposed service semantics (Grid Service). The OGSA (Open Grid Services Architecture) [10] can provide building blocks that can be used to implement a variety of higher-level Grid services such as distributed data management services. This framework is very flexible because services can be implemented and composed in a variety of different ways. By using Grid computing, the CVGE can integrate and share all internal resources to create a big high-performance computing environment. Since the core of the OGSA is the service idea, the OGSA-based CVGE can integrate services across distributed, heterogeneous and dynamic collaboration environment formed from the disparate resources with a single system and/or from external resources sharing and service provider relationships. In this paper, we pay attention to build up a data collaboration service to high-efficiently support collaborative work in the CVGE. The rest of this paper is organized as follows. In section 2, we focus on how to build a data collaboration service, some key problems such as Grid-based CVGE service framework, data searching mechanism, data accessing control and service implementing are discussed in detail. A simple prototype system was developed and a data collaboration experiment was implemented as shown in section 3. Finally, some concluding remarks are given in section 4.

2 A Data Collaboration Service 2.1 Grid-Based CVGE Service Framework In order to integrate and manipulate data across distributed, heterogeneous environments and improve the CVGE service capability, we designed three-layer system architecture: resources layer, service layer, and application layer (Figure 1). In the Grid-based scheme, a series of services are built to collect and integrate resources including data and clients information. By means of Grid platform, the CVGE users can register their resources and publish resources sharing services. Using these basic services, some data manipulating services, such as data searching, monitoring and transmission services, are developed and then turn to be new services for dealing with data collaboration application sceneries.

346

J. Zhu et al. Application Layer Data visualization

Collaboration work

Data transmission

Data management

Data analysis Others modules

Service Layer User Interface

State monitoring

Register service

Task Analysis

Data collaboration

Grid basic services

Data search

Content index

Resource Layer Memory

Storage space

Geographic data

Computing power

Network bandwidth

Software

Others resources

Fig. 1. Grid-based CVGE service framework

The resource layer contains many resources, including geographic data resources and client computer information, which are registered to resource registry center for sharing. All resources can be integrated into a virtual resource environment and used by any user in the CVGE system. The service layer is built upon the resource layer, and a series of management toolkits and protocol criterions are defined to implement the sharing, integration and interoperation of all the resources in the resource layer. At the same time, the transparent and unified accessing interfaces of resources and services are afforded to support the applications in the application layer. The service layer is the core of the CVGE system, and ensures the whole system running correctly. This layer is comprised of several kinds of functional components. Firstly, we used Grid basic service to support the CVGE system. Secondly, we designed register service and data indexing service to manage data and clients computing information (i.e. network bandwidth). Finally, on the basis of aforementioned services, we designed the data searching mechanism and transmission control model to implement data collaboration tasks. In the application layer, applications are mainly related to the CVGE functions with the support of the underlying layers. It includes some basic functions such as local data management, data transmission, data visualization, and data analysis, which can be used to support collaboration work. Data management module can deal with local data and files and register them to Grid server. Data transmission module is responsible for accepting request and sending data to other clients. Visualization module can display data in 3D scene for users. According to user’s task, analysis module can know which kind of data user requires. If these data cannot be found in local database, we send data request to Grid server.


347

2.2 Data Searching Mechanism We designed user management list and data indexing list for all users in system. The user management list includes user name, IP address, bandwidth, CPU frequency and so on. The indexing list is used to manage all shared data, and includes data mark, content, update sign, user attribute and so on. The Grid-based CVGE data searching mechanism is shown in figure 2. By means of uniform accessing interface, user A can submit data request task and get processing results after a series of Grid service operating（ the broken line frame in Figure 2） . The whole Grid operation is entirely transparent, and user can get the searching results without knowing where the data resources come from. Web Portal Grid service environment Updating

Register service Task submitting User A

Task analysis

Content index

Noticing

Data transmitting

No Failure

User B

Data locating

Inquiry

？

Yes

Fig. 2. Data searching mechanism

Figure 2 shows this data searching mechanism, which is described in detail as follows: 1) Through accessing certain Web port, user A submits data request task, then the Grid service environment analyzes the task and carries out the data inquiry by means of content index service. 2) If content index service queries the expected data, the target data will be located (we may suppose user B possesses these target data), then user B will carry on data transmission to user A. Other wise, the data searching will not success. 3) After downloading these required data, user A will notice register service update data content index in Grid server, dashed line arrows in figure 2 shows the whole process. 2.3 Data Access Control The multi-user participation is one of the most important features in collaborative virtual environment. The traditional data transmission arises only between server and users. This method is very difficult to keep spatial-temporal coherence in a distributed CVGE scene and to resolve synchronization and coherence of a virtual scene. It is mainly caused by the traditional network structures, such as browser/server and client/server, in which the centralized function of the server causes some restrictions of bandwidth, speed and mobility. In order to overcome these restrictions, we can

348

J. Zhu et al.

exploit users’ resources and data directly exchange among users. Accordingly, based on aforementioned idea and Grid services, we designed data accessing control model shown in figure 3. F ailure

No Task request

Searching data˛

Yes

Q ueuing

State m onitoring

Transm itting

D ata locating

No

D ata locking˛

Yes

Fig. 3. Flowchart of data accessing control

The basic process could be described in detail as followed: 1) According to request task, sever carries out data inquiry and selects data records in the whole system environment. If the number of records is more than one, they will be queued according to their bandwidth, IP address, and computer performance. 2) In order to prevent the concurrency of data accessing, the accessed data are locked (e.g. 1 means locked, and 0 means unlocked). In order to keep the priority, we should check these data items until there signs are all 0. 3) When one most appropriate data item have been chosen, computer will be asked to send data to destination computer. In the course of the data transmission, the data item is locked till the end of task. Meantime, if the process is broken, the above steps will be restarted. 2.4 Data Collaboration Service After integrating all data and computing resources in the CVGE system environment, the CVGE users can more rapidly and conveniently query and share data. However, data collaboration is not equal to data sharing. The main difference between data sharing and data collaboration is that data sharing is one-way sharing of data and files, while data collaboration implements multi-directional sharing of data and files, allowing ideas and thoughts to be communicated interactively among CVGE users. For more powerful interactivity and real-time collaboration, we designed a data collaboration model to create a data collaboration group based on our system architecture, which can automatically finish some collaboration tasks. The algorithm of creating a dynamic group is shown in Figure 4 and its basic approach is as follows: 1) Firstly, user sends one request task to Grid server and task analysis service implements data searching and judges if the CVGE system can solve this problem. If the answer is “yes”, then continues to the next step; otherwise, stops movement and declares task failure.


349

User Sending request End Task analysis

No

Data searching

Finishing task˛ Yes

No

Creating temporary group request

confirm˛ Yes

No Yes

Solving problem˛

Building group relation

One member exit˛

Maintaining

No Yes End

Finishing task˛

Fig. 4. Flowchart of implementing data collaboration service

2) Secondly, according to the request task and user information list, the data collaboration service chooses some users that can finish the task. These users will receive from cooperative request in turn. If one user agrees, then he joins the data collaboration group; if one user refuses, then returns to judge the first step and repeat these steps until the data collaboration group is created. 3) Finally, the relation of the data collaboration group will be maintained until the request task is finished. One user node may drop out the data collaboration group according to his own situation. Subsequently the data collaboration service will confirm it. If the situation affects the cooperative task implementing, the data accessing control will look for a new node to replace the failure node’s position.

3 Application Experiment 3.1 Application Background In the case study area of Qiu-Yuan-Gou watershed, with a total area of 70.7 km2 and a length of 18 km of the major channel, there are 202 silt dams, of which 17 are the backbone dams (big dams) and 185 are the ordinary dams (middle and small dams). The whole dam system of Qiu-Yuan-Gou watershed is a typical dam system of the massive silt dam construction project on Loess Plateau. Our study aims to build a CVGE for supporting spatial planning of silt dams across the Internet. Of course, we also pay attention to address the data collaboration mechanism in CVGE.

350

J. Zhu et al.

3.2 Geographic Environments Modeling There are two types of models in terms of virtual geographic scene systems. One is the field-based model, such as the DEM and land use model, and the other is the entity-based model (i.e. the model uses entities), such as dams, trees, and roads. In our case study, we used ArcGIS to process basic scene data including SPOT 5 remote sensing image and 1:10000 DEM. In order to high-efficiently build a three-dimension scene under a web-based distributed environment, we divided the terrain data into different layers and blocks. Each layer of multi-resolutions data layers stands for one kind of resolution. Meantime, a kind of quad-tree structure [11] was adopted to organize the terrain data. Every terrain data block in the quad-tree has the same amount of data. Thus the data package based network transmission would benefit from such a feature. Furthermore, these features are also convenient to design a highefficiency LOD (Level of Detail) algorithm for simplifying terrain scene. We implemented fast terrain rendering using geometrical MipMapping algorithm [12]. Figure 5-a shows the virtual 3-D watersheds with a sub-regional DEM of Jiu-YuanGou watershed, overlaid by thematic maps of a SPOT 5 remote sensing image. Figure 5-b shows the terrain scene rendered with wireframe when LOD screen projection error is 1. In Figure 5-b, the terrain scene includes 10171 triangles, and its simplification ratio is 1.8%, the rendering speed ratio is 60 frames per second (Celeron 1.7GHZ CPU, 256M memory and 32M integrated graphics card.).

a: Terrain scene after using texture

b: Wireframe terrain scene

Fig. 5. Three-dimension terrain scene

3.3 System Development Using Java, GT3, C++ and OpenGL technologies, a prototype CVGE system for dam systems planning in watersheds was developed. In order to support different software and hardware environment, it is better to choose a development platform that is descriptive and independent from specific languages. Java is an ideal language to develop Grid services. After the compilation, Java binary code can be executed by any Java interpreter, which is cross-platform. C++ and OpenGL can deal with LOD


351

algorithm for improving terrain scene rendering speed, which can be compiled DLL file to support Java displaying virtual scene by means of JNI (Java Native Interface) methods. Because the Grid service implementation process is very complicated, we developed and offered some graphical operating interfaces, which can shield physical detail and help users improve working efficiency under the Grid environment. In the CVGE system, video function is also added to help co-workers better understand the intention, improve perceptual knowledge, and enhance working efficiency. Figure 6 shows a snapshot of the user interface of the CVGE prototype system.

Fig. 6. The user interface of the prototype system

3.4 Data Collaboration Experiment As mentioned above, all CVGE users register into Grid server and a user management list and user information list are created. Meantime all data in the CVGE system environment are integrated and a data information management list is created. In the collaborative silt dams system planning experiment, we used data collaboration service to create different service instances with a definite lifecycle and deal with different application sceneries (i.e., video and scene visualization collaboration). For example, to implement a request task such as the real-time terrain walkthrough collaboration or video collaboration, some users build a dynamic task group. In this study, one user implemented terrain walkthrough (i.e., checking certain planning result in virtual environment), meantime others can viewed the same terrain scene with the same angles and his (her) video image. By means of collaborative visualization environment, participants can give some revised comments in order to design an ideal distribution of silt dams. In order to implement high-effective data translation in the CVGE system, the data collaboration service was designed to use full client capacity. This can reduce data server load and provide better data sharing service among users. Figure 7 shows a

352

J. Zhu et al.

'ata GRZQORDGWLPHV

comparison of data transmission efficiency between our scheme and traditional mode (the experimental network bandwidth is 10 Mbps and the data capacity is 50 MB). The curves in figure 7 indicates that the data transmission efficiency of using data collaboration service is higher if the number of users is added, while the efficiency of using centralized mode is lower.

Number of user nodes

FHQWUDOL]HGPRGH

FROODERUDWLRQPRGH

Fig. 7. Data transmission efficiency under different modes

4 Conclusions The study of geo-collaboration involves diverse aspects ranging from participants and organizations to mediated tools, geo-problem contexts and supportive environments. It is very important to build a collaborative virtual geographic environment for intuitive and efficient interactive visualization, which allow distributed users to explore complicated spatial information and conduct collaborative work. This paper aims to design and develop a data collaboration service. Some key problems, such as Gridbased system service architecture, data searching mechanism, data accessing control and data collaboration service implementing, were discussed in detail. In the case study of dam systems planning in Jiu-Yuan-Gou watershed, a prototype of the CVGE system was developed. The advantages of scheme in this paper are summarized as follows: Firstly, the traditional collaborative system serves a user by deploying the whole software. However, many users only need part of the software’s functions. So we introduced the services idea to resolve this problem because it can serve a user by the means of ‘‘service’’ and the user can select the services he need and build ‘‘his software’’. Secondly, the Grid-based CVGE service framework can be not restricted in certain operation platform and system environment. Furthermore, some existent visualization algorithms and programs can be efficiently utilized. Finally, all users in the CVGE are easy to register their data and computer resources, which can efficiently integrate and share system internal resources. Data collaboration service can make use of users’ impacts and improve the data transmission efficiency. Hence the better service capacity and availability are provided to improve collaborative work quality.


353

Acknowledgement This research is partially supported by the National Natural Science Foundation Project No. 40471103, the National Key Technologies R&D Program of China No. 2004BA718B06, the Key Project of National Natural Science Foundation No. 30590374, and the Opening Foundation Project No. PK2004018 of the Key Lab of Poyang Lake Ecological Environment and Resource Development, Jiangxi Normal University, China.

References 1. Z.F. Shen, J.C. Luo, C.H. Zhou, Architecture design of grid GIS and its applications on image processing based on LAN, Information Sciences, Volume 166, Issues 1-4, 29 October 2004, Pages 1-17 2. H. Lin, Q. Zhu, Data Visualization: Virtual Geographic Environments combining AEC and GIS. Available from: < http://vrlab.whu.edu.cn/chinese/study/study12.htm>. 3. Q. Wang, Research on Geographic collaborative work and group spatial decision support system, Institute of Geographical Sciences and Natural Resources Research, CAS, PhD Thesis. 2002. 4. Jianhua GONG and Hui LIN. A Collaborative VGE: Design and Development [M], A Chapter to a book entitled "Collaborative Geographic Information Systems"edited by Shivanand Balram and Suzana Dragicevic, Simon Fraser University, Canada, 186-206, 2006 5. C. Bouras, A. Philopoulos, T. Tsiatsos, A networked intelligent distributed virtual training environment: a first approach. In Proceedings of fifth Joint Conference on Information Sciences-JCIS’2000—1st International Workshop on Intelligent Multimedia Computing and Networking, 2000, Vol.2,pp.604–607. 6. I. Foster, C. Kesselman, The Grid: Blueprint for a New Computing Infrastructure, Morgan Kaufmann, 1999. 7. Foster, What is the grid? A three point checklist. Available from: < http://www.gridtoday.com/02/0722/100136.html >. 8. Foster, C. Kesselman, S. Tuecke, The Anatomy of the Grid: Enabling Scalable Virtual Organizations. Available from: < http://www.globus.org/research/papers.html#anatomy > 9. Foster, C. Kesselman, J. Nick, The physiology of the grid: An open grid services architecture for distributed system integration. Technical report, Available from: < http://www.globus.org/research/papers/ogsa.pdf>. 10. H.L. Cai, J. Wang, D. Li, S.D. Jitender, A novel state cache scheme in structured P2P systems, Journal of Parallel and Distributed Computing 65(2005) 154-168 11. B. Chen, Z.G. Ma, G.P Wang, et al., Network Based Real-time Fly-through on Massive Terrain Dataset, CAD/Graphics’2001, (2001) 500-505 12. H. Willem, Fast Terrain Rendering Using Geometrical MipMapping. Available from: .

VCS: A Virtual Collaborative Space Based on Immersive Teleconferencing Weidong Bao, Maojun Zhang, and Wang Chen Deptartment of Systems and Engineering, National University of Defense Technology, Changsha 410073, Hunan, P.R. China

Abstract. The traditional video teleconferencing systems provide a “video in a window” interface paradigm, it’s not sufficient for naturally interactive interface for the collaborative work. We design a collaborative space called VCS. In VCS, a virtual collaborative space is built based on immersive teleconferencing. The remote conferees can discuss in the virtual space as similar to do it in the local rooms. VCS can be used to provide a new paradigm for the remote cooperative work. This paper presents the spatial model and video object extraction technique of VCS. Keywords: teleconferencing, virtual collaborative space, video object extraction, virtual reality.

1 Introduction Most video teleconferencing applications provide for the users a “video in a window” interface. Though they have been improved significantly over the years, the constraint of field of view still can’t address natural interactions in the teleconferencing systems. The field of view of a common TV or window is narrow and below 10o. The resolution of video from a typical camera is low and below 752*576 pixels. When they are used as the basic interface for a teleconference, it fails to produce a convincing feeling of presence and immersion [3]. Even when the network bandwidth problem is completely solved, one is left with relatively low spatial resolution and a narrow field of view. In order to construct one more natural and compelling teleconferencing application, we designed and implemented a immersive teleconferencing system called VCS (Virtual Collaborative Space), which combines teleconferencing with virtual reality. The traditional video teleconferencing system is composed of terminals and MCUs (Multi-point Control Unit). MCU receives video and audio from all conferees and switches one-way video and audio for one conferee as he/her demands (Fig 1(a)). In VCS, however, conferees are extracted from the background and form video objects. Video object and audio from all conferees are transmitted to MCS (Multi-point Composing Server) and composed in one virtual conferencing room by MCS. The synthesized virtual conference room is displayed on a scalable environment by tiling a display surface with multiple light projectors (shown in Fig 1(b)). Conferees can navigate in the virtual room and view all other conferees at the same time. They can talk with other conferees and work collaboratively. When conferees move in a real Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 354 – 360, 2006. © Springer-Verlag Berlin Heidelberg 2006

VCS: A Virtual Collaborative Space Based on Immersive Teleconferencing

355

place, the actions can be mapped into the virtual conference room in time. The goal of VCS is providing a virtual space contained all conferees and providing a seamless environment that users can immerse in. This paper presents the spatial model and video object extraction technique of VCS. Person A

Person B

Person B

Person A

MCS

MCU

Monitor

Person C

Person C

Fig. 1. Difference between traditional teleconferencing and VCS

2 Related Work There are some researches [1,2,3,4,5,6,7,8] devoted to the research area of immersive teleconferencing. Yun et al[1] and Buyer et al[2] at first proposed the technique of video composing in video teleconferencing, which can composite multi-way video into oneway video. Video composing can reduce the network requirements of video teleconferencing. It can, furthermore, compose conferees from multiple places into one TV or window of a CRT. Some systems combine elements of both computer graphics and video teleconferencing. CU-SeeMe VR[1] realizes a 3D shared virtual space. In this system, live video streams from conferees are textured onto planar surfaces. These planar surfaces, representing the conferees, are allowed to roam around a shared virtual conference room. This is one approach to use avatars for teleconferencing that conferees see a 3D view of the 2D avatars within the virtual space. Another approach is using 3D avatars for teleconferencing, such as VST[8], Diamond Park[12]. Criminici et al[4] propose a new algorithm for novel view generation in teleconferencing applications. Given the video streams acquired by two cameras placed on either side of a computer monitor, the proposed algorithm synthesis images from a virtual camera in arbitrary position (typically located within the monitor). This algorithm can be used to build a teleconferencing system to facilitate eye contact. Gemmell[5], Yang[8] propose the similar algorithms to rebuild the eye contact.

To allow geographically displaced conferees to experience the full spectrum of manifestation they are use to in real world meetings, 3D images of the conferees

356

W. Bao, M. Zhang, and W. Chen

are synthesized and positioned consistently around a shared virtual table[10]. Glasenhardt et al [7] propose the concept of “tele-immersion”, which enables users at geographically distributed sites to collaborate in the shared, simulated, hybrid environment.

3 Spatial Model In this paper, we propose a new method to build the virtual collaborative space (VCS). VCS support the following three aspects of context awareness:

WHERE, conferees can perceive the virtual environment and obtain the information of the space layout and scene distribution. WHO, conferees should know whom they are working with in the virtual collaborative space. WHAT, conferees should know the status of cooperative work and other participants’ work.

In order to verify the above context awareness, we build a prototype system of immersive teleconferencing based on the multiple cameras and multiple projectors. Shown in figure 2, in a teleconferencing system with three conferees, three cameras are placed before each conferee. Three ways of video are transmitted to the remote sites.

Fig. 2. The displays and cameras in VCS

The received videos and stored 3-D virtual environment are synthesized at a MCS (Multiple Composing Server). The synthesized results are displayed on the large scale display that three light projectors are stitched horizontally (shown in figure 3(a)). The size of each screen is 2m*1.5m and the display resolution of each screen is 1024*768 pixels. The two side screens intersect the central one 160 degree, i.e. Φ=160o in figure 3(b). The display provides 80 degree FOV(Field of View) if the DOV ( Distance of View ) is 4 meters. The total resolution is 3072*768. VCS is consisted of the display projectors, 3-D objects that can be manipulated, virtual scene and so on. Figure 4 shows a snapshot of a VCS. In order to build a shared, immersive and collaborative space, we need a suitable spatial model of VCS. The spatial model of VCS is built on the GCS (Global Conference Space) and LCS (Local Conference Space).


Φ

α

2m 2m

2m Φ

Φ 4m

1.5m 1.5m

357

1.5m

Φ

(a) Three stitched project screens

(b) DOV and FOV

Fig. 3. The display system of VCS environment

Fig. 4. A snapshot of VCS

DEF 1 (Global Conference Space: GCS). GCS is a virtual environment that all the conferees share and participate in. It spatially describes all the elements of the immersive teleconferencing in a uniform coordinate system, including virtual scene VS, conferee set CS, object set OS. GCS ::= ::= <panoramic image> ::= :: = … CS contains the video objects extracted from the backgrounds and audio. OS contains all the virtual objects that used in the CSCW applications, such as a designing car, virtual whiteboard, virtual pen, etc. In VCS, VS is constructed from panoramic images. Panoramic image is an image stitching all the views from different directions at one specific site and focal length. The types of panoramic images usually include spherical, cylindrical, polygonal panoramic images. Currently, VCS use cylindrical panoramic images to render the virtual space although cylindrical panoramic images implement less degree of viewing up/down. In order to insert the conferees into the proper positions in GCS, the mapping relationship between real conference room and GCS must be built. This relationship should be confirmed before each conference and don’t be modified during the conference.

358


As soon as the relationship is built, the movements of each conferee can be also mapped into GCS. When a conferee walks in real conference room, the video object of the conferee will act at similar ways in GCS. We need consider the collision between conferees and OS, such as a virtual table. DEF 2 (Local Conference Space: LCS). LCS is the projection of GCS with the view position and view direction of the local participant. Each participant corresponds to one LCS. LCS is displayed on three stitched projectors. Users can change their view direction and location using various input devices to get different LCS views.

4 Video Object Extraction The spatial model of VCS defines the spatial relationship among the virtual scenes, video objects and virtual objects. In order to synthesize the virtual conference room, we must extract the video objects from the background. To extract the video objects, subtracting the current image from the background model is straightforward. In our application circumstance, considering the background is nearly stationary and there are just tiny changes, our algorithm constructs a simple background model which is a copy of a frame without foreground at the very beginning and then updates the model with the current frame. Unlike other algorithms, our approach is very simple, fast and also works well. The detailed algorithm is given below. (a) Construct the background model Before video extraction, we choose a frame without foreground and grayed it for the background model. (b) Update the background model Let A denote background model and χ present RGBcolored current frame, the new background model A i = A × (1 - α) + χ × α , 0 < α < 1 , α is usually a very small number, in our experimentation, α is 0.003. Then set: A ← A i . (c) Video object extraction The video object extraction is finished as the following steps: z χ ← Gauss( χ ) .Gauss indicates Gaussian filtering. z

ℜ ← A−χ . ℜ ← Thres(ℜ, β)

Thres(ℜ, β) indicates using threshold β to convert ℜ to binary. z For each pixel in ℜ , ℜ(i, j) = 1 indicates this pixel z

.

is foreground, else is background. Fig. 5 shows the examples using our video object extraction algorithms.


359

(a) The original pictures

(b) The results of video object extraction Fig. 5. The examples of video object extraction

5 Conclusions Immersive teleconferencing is a new paradigm for CSCW applications. We have presented the spatial model of virtual collaborative space based on the immersive teleconferencing. This spatial model can bring all the conferees into a virtual conference room. The conferees can immerse in the virtual collaborative space and feel like seating and working together. To show the initial results, we develop a prototype system. The prototype system supports three terminals and has implemented a large scale display to show the virtual collaborative space. Our future work will focus on improving the effects and speeds of video compositing and freedom of interaction in the virtual conference room.

360


References 1. Louis C. Y., David G. M., Architectures for Multi-Source Multi-User Video Compositing, Proc. of ACM Multimedia, CA, USA, ACM Press (1993) 215-223 2. Buyer D. G., The Personal Presence System – a Wide Area Network Service Resource for Real Time Compositing 0f Multipoint Multimedia Communications, Proc. of ACM Multimedia, USA, ACM Press (1994) 3. Han, J., CU-SeeMe VR Immersive Desktop Teleconferencing, Proc. of ACM Multimedia, Boston, USA, ACM Press (1996) 199-208 4. Criminisi A., Shotton J., Blake A., Torr P.H.S., Gaze Manipulation for One-to-one Teleconferencing, Proc. of the Ninth IEEE International Conference on Computer Vision (2003) 1-8 5. Gemmell J., Zitnick C. L., Kang T., Toyama K., and Seitz S., Gaze Awareness for Videoconferencing: A Software Approach,” IEEE Multimedia, vol. 7, no. 4, 2000 (26-35) 6. Schafer R., Image Processing and Coding for an Immersive Teleconferencing System, IEEE Region 8 International Symposium on Video/Image Processing and Multimedia Communications, Zadar, Croatia (2002) 16-19 7. Glasenhardt S., Cicin-Sain M., Capko Z., Tele-Immersion as a Positive Alternative of the Future, International Conf. Information Technology Interfaces, Cavtat, Croatia (2003) 16-19 8. Yang R., Zhang Z. Eye Gaze Correction with Stereovision for Video Teleconferencing, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 26, No. 7 (2004) 956-960 9. Sun L., Zhong Y., Zhong Z., Natural Interaction Synthesizing in Virtual Teleconferencing, IEEE ICIP (2002) 405-408 10. 10.Cooke E., Kauff P., Schreer O., Multiple Narrow-Baseline System for Immersive Teleconferencing, VIPromCom (2002) 367-370 11. Silva L. C. D., Miyasato T., Kishino F., Emotion Enhanced Multimedia Meetings Using the Concept of Virtual Space Teleconferencing, Prof. of IEEE Multimedia (1996) 28-33 12. Waters R. C., Barrus J. W., The Rise of Shared Virtual Environments, IEEE Spectrum, Vol. 34, No. 3, 1997 (20-25)

A Scalable HLA-Based Distributed Simulation Framework for VR Application Zonghui Wang, Jiaoying Shi, and Xiaohong Jiang State Key Lab of CAD&CG Zhejiang University, HangZhou, P.R. China, 310058 {zhwang, jyshi, jiangxh}@cad.zju.edu.cn Abstract. HLA-based Distributed Simulation technology is employed widely in Virtual Reality (VR) applications, such as military simulation, internet games, roaming etc. To support larger number of participants, scalability is becoming a key issue of VR applications. In this paper, we explore the characteristics of distributed simulation, and analyze the scalability of servers and participants, and classify our approach to improve scalability of VR applications into three aspects: a three-tier node management mode to accommodate more participants, an efficient management of servers to manage scalable number of clients and software development interface to achieve reusability and interoperability of VR applications. We present our middleware platform, HIVE, providing a scalable HLA-based distributed simulation framework for VR applications, on which users can develop VR applications easily and quickly. Then we give the method and view of application integration with HIVE. Finally an experimental demo is given.

1

Introduction

With the development of network technology and the increasing requirements of applications in many fields, traditional Virtual Reality (VR) applications which run on only a machine are required to run on network to meet the requirement of large scale participants and scenes. Distributed Simulation technology is employed widely in large scale VR applications. VR Juggler [1] only provides non-distributed VR application development environment. Existing distributed VR application frameworks include MR Toolkit [2], NPSNET [3], MASSIVE [4], Bamboo [5], DIVE [6], AVOCADO [7], ATLAS [8], etc. Large scale VR applications such as military simulation, internet game, roaming, surgery train etc, have the trends as the following: (1) They should have a scalable architecture to accommodate more participants. (2) They should support the collaboration work of large scale distributed nodes to run efficiently to provide services to more participants. (3) They should be developed quickly and easily with unified specification and development interface. To fulfill these demands, the scalability of system architecture of VR applications must be improved. There are two parts in network application systems: Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 361–371, 2006. c Springer-Verlag Berlin Heidelberg 2006

362

Z. Wang, J. Shi, and X. Jiang

servers and participants. The key issue to improve scalability, which is related to the design of system architecture, is to improve the capacity of the system and the efficiency of servers group. Most previous systems adopt one or more of the following architectures: peer/peer, client/server, or peer/server. In those systems with servers, resource management and workload balance of servers is very important, which affects the scalability of servers directly. In another side, VR applications should be open, and developers could use the unified interface to develop their own client. In conclusion, there are three factors leading to the architecture scalability of VR applications: node management model, management of servers and software development interface. Our purpose is to build a general middleware platform, including the APIs and back-ends. On the platform, users can develop VR applications easily and quickly, concentrating more on the logic of application, and spending less time on the infrastructure of application, such as data distribution and management, network architecture [9], etc. In this paper, we propose HIVE, which gives a middleware platform for VR applications development. HIVE has a three-tier node management model: global tier, group tier and client tier. HIVE employs a kind of management of servers to improve the work efficiency of servers. To achieve the reusability and interoperability of VR applications [10], we employ the user interface specification of HLA (Draft 1.3) [11] as reference, which was defined by US DoD, and became IEEE P1516 Standard [12] in 2000. The remainder of the paper is organized as follows. In section 2, we discuss the key issues on scalability of VR applications. Section 3 depicts the implementation of HIVE in detail. In section 4, we describe VR applications’ structure and its integration with HIVE platform. An experimental demo is presented in section 5. Finally we summarize our work in section 6.

2

Key Issues on Scalability of Architecture

Scalability is an attribute of multiprocessor system. What is Scalability? After examining the formal definition of scalability, Mark D.Hill fails to find a useful rigorous definition of it, and thinks that the notion of scalability is intuitive, and use of the term adds more to marketing potential than technical insight [13]. In opinion of Dean Macri, scalability refers to the challenge of making a game that runs acceptably across system configurations that may vary in features, performance, or both [14]. But he only provides his scalability approach in the processor and the graphics subsystem. In this paper, scalability means the ability to adapt to the system resource changes. In VR applications, high scalability means that it can support large number of users to join in simultaneously and interact with others and objects naturally in virtual world. Because what we build is a general middleware platform for VR applications development, we don’t touch the aspects which are related with applications in practice. We explore the ways to improve scalability of VR application only from system architecture. There are three corresponding factors:

A Scalable HLA-Based Distributed Simulation Framework

363

Large number of nodes support. Since distributed simulation contains large number of participants, and also contains many servers to provide the services, how to organize them is one of the main problems of improving scalability. High efficient servers run. While the number of participants becomes larger, the workload of servers also becomes heavier. If the servers run more efficiently, the simulation could contain more participants. So management of servers is one of key aspect of improving scalability. To make full use of servers, it is very important to deal with participants division and migration to balance the workloads. Openness of platform. It is important that the simulation platform should have good openness to improve its scalability. It means that the simulation platform should comply with unified specification and be open to the third party, and support developers to develop their own application. More over, the simulation platform should provide the software development kit, on which user could develop the VR application quickly and easily. According to these three factors, our approach focuses on: Node management model. Node management model defines the organization structure of nodes, and classify the nodes and give different functions to them. There are three kinds of node management model of network applications. The first is centralization mode, i.e. client/server model. In this mode, the efficiency of mode management is very high because of the coordinating of server. But when the number of the client is large, the server’s burden of workload and network interface would be very heavy. The second is distribution mode, i.e. peer to peer model. In this mode, there is no bottleneck of server, and the architecture of system is scalable. But when the number of the client is large, the efficiency of node management is very low without the coordinating of servers. The third is hierarchy mode, i.e. multiple levels of client/server model. In this mode, the efficiency of node management is high, and it supports large number of node. But the implementations of hierarchy applications are very complex. To get high efficiency of node management and accommodate large number of node, we adopt a three-tier node management model. Management of servers. The function of servers is to provide services to clients. Management of servers contains server resource management, workload allocation and dispatch. The servers run more efficiently, the system will support more services to more clients, and will be more scalable. To improve the efficiency of servers run to support large number of participants, we provide two kinds of client distribution. Software development interface. Software development interface contains the specification of services which the simulation platform provides and the library of them. To make the VR applications reusable and interoperable, we employ the interface specification of HLA as reference. And to provide more convenience to developers, we provide the software development kit (SDK).

364

3


HIVE

HIVE is a Middleware Platform, containing the following: HIVE back-ends. HIVE back-ends consist of three parts, Directory Manager (DM), Group Agent (GA) and Group Manager (GM). Group Manager provides the global services, such as GM/Client resource identity service, monitor and control, workload balance, etc. Group Agent provides GM management to create/manage/destroy GM, following the instructions of DM. GM provides HIVE services to Client. HIVE library. HIVE library provide the interfaces of HIVE services. Client implements the VR application using the HIVE library. Figure 1 illustrates the network architecture of HIVE.

GA(GM)

GA(GM)

GA(GM)

DM

Client

Client

Client

Client

Fig. 1. Network architecture of HIVE. ”GM” means Group Manager. ”GA” means Group Agent. ”DM” means Directory Manager.

3.1

Node Management Model

We present a three-tier mode management model: The first tier, named global tier, contains Directory Manager and Group Agent. Group tier is the second tier, consisting of Group Managers. Client lies in the client tier, which is the third tier. A critical concept in HIVE architecture is ”routing tree”. All the nodes in VR applications are organized as a hierarchical tree - routing tree. Routing tree has three levels. The root of it is DM. The children of DM, in the second level, are GMs. The third level contains clients. DM maintains and updates the routing tree dynamically, and each GM has a copy. According to the routing tree, DM and GMs can find any client’s owner GM to deliver messages, which is the parent node of the client.


365

Directory Manager (DM). Directory manager is the process that will be launched first. The architecture of DM is illustrated in Figure 2. It provides the global services: client and GM join / leave, client and GM resource monitor, client migration, workload computation and balance, client distribution, maintaining the ”routing tree” and other global service, such as ”global time query”, ”name resolve”, etc.

Workload Computing and Balance

Routing Tree Maintenance

Client Distribution

Client and GM Join / Leave Service

Client and GM Resource Monitor

Client Migration Service

Network Communication

Fig. 2. Architecture of Directory Manager

Group Agent (GA). Group Agent is a daemon process. GA keeps communication with DM, sends the ”heartbeat” to the DM to indicate that the GA is working. GA also launches the GM according to the ”Create GM” instruction from DM, and destroys the GM according to the ”Destroy GM” instruction. Group Manager (GM). A Group Manager specifies an application. Figure 3 illustrates the architecture of GM. GM maintains the list of the clients which are assigned by the DM. It also communicates with and provides services to the clients, such as time management service, user management service, ownership management service and object distribution management service according to the clients, which will be described later. To provide services, GM keeps all the basic information of the clients, such as ”Subscribed List”, ”Published List”, ”Update List” and ”Discovered List”, and updates them. In client message delivery, GM retransmits the messages from source client to destination client. If the destination client is maintained by another GM, the messages are delivered to that GM first, and then retransmitted to the destination by that GM. Message queue is the buffer of messages, which enables processing messages asynchronously.

Time Management Service

User Management Service

Ownership Management Service

Object Distribution Management Service

Client Message Delivery

Message Queue Network Communication

Fig. 3. Architecture of Group Manager

Client. Client can act as various roles in VR, such as the participant, the scene object, or just a passive viewer [15], etc. Figure 4 shows the architecture

366


of client. HIVE Client consists five parts to provide services to VR applications. User management contains the services such as creating / destroying application, joining / leaving application, etc. Object Distribution Management provides the services such as publishing / subscribing object class, registering / discovering object, updating / reflecting object, publishing / subscribing interaction, sending / receiving interaction, etc. Ownership management provides the services such as requesting / divestiture ownership, etc. time management contains the services such as querying Time, advancing Time, etc. Callbacks are the interface of event notification services. When an event comes, the corresponding callback will be called.

Application Level

Time Management

User Management

Ownership Management

Receiving Message Queue

Object Distribution Management

Callback

Sending Message Queue

Network Communication

Fig. 4. Architecture of Client

On client side, there are two message queues. One is the sending queue, which is the queue for messages asking for a service. The other is the receiving queue, which buffers the messages and delivers them in proper time with the help of time management. 3.2

Management of Servers

In HIVE, resource management is mainly done by DM. GMs join/leave the DM, and then DM accepts them via GM Join/Leave Service. After GMs join, GM Resource Monitor will monitor and control them. When a client applies for a GM request to get HIVE Service, GM Resource Allocate Service will compute the workload of all GMs, and then select an appropriate GM for the client according to the GM Allocating Strategies. Figure 5 shows the workflow of resource management of HIVE. We provide two kinds of methods of client distribution: one is that participants are grouped and assigned to a GM according to the result of workload balance of DM, no matter where it is located during initialization or where it goes later. And participant will not be migrated to other GM except when the GM is going to exit or it needs to reduce workload if a new GM joins. This method is helpful when participants of the application move within a wide area and move back and forth quickly, and the states of them don’t update frequently. The other is that each GM is in charge of a partition of the virtual world. In this case, when participant joins the DM, the DM will find an appropriate GM according to its original location. When a participant moves to another partition, he will be migrated to the GM that is in charge of that partition. This method is


367

Directory Manager GM/Client Resource Information

Client

GM t ues Req

Result

Join/Leave

GM Workload Computing and Balance

GM Resource Monitor

GM Resource Allocate Service

GM Join/Leave Service

GM Client Distribution

GM Allocating Strategies

Fig. 5. Resource management of HIVE

useful when participants of the application will not move very frequently, but its frequency of state update can be high. 3.3

Software Development Interface

For convenience of VR applications development, we provide HIVE SDK. HIVE SDK includes APIs specifications and library files to be linked and run. To achieve the reusability and interoperability of VR applications, we employ the interface specification of HLA as reference. The APIs are classified into four parts. We adopt federate management of HLA as our user management, and adopt Declaration Management, Object Management and Data Distribution Management of HLA as our Object Distribution Management, and adopt Ownership Management and Time Management of HLA as our Ownership Management and Time Management.

4

Application Integration

There are two parts in the structure of a VR application. The first part is application logic, which is the content of a VR application. The second part is data presentation, which is the way would be used to show the result of a VR application. The data presentation can be text, 2D graphics or 3D graphics, etc. Figure 6 illustrates the process flow of HIVE client. First, it selects an avatar from the model database, and loads the scene of the VR application, and then joins in the VR world. Then it goes into the VR application loop. The user interface of the client reflects the changes of dynamic information. Finally it resigns from the VR world. Figure 7 illustrates the relationship of HIVE back-ends and HIVE applications.

368

Z. Wang, J. Shi, and X. Jiang Select an Avatar

Scene Database

Load Scene

Join in VR World

Application Loop

Resign from VR World

User Interface

Fig. 6. Process flow of HIVE client

Application Level

Client

Object Distribution Management Interface

Time Management Interface

Ownership Management Interface

User Management Interface

Event Notification Interface

HIVE Middleware Access Point

Middleware HIVE Middleware Implementation

Directory Manager

Group Agent and Group Manager

Resource Management Service

GM Management Service

HIVE Service

Fig. 7. Relationship of HIVE back-ends and HIVE applications

5

An Experimental Demo

We implement an experimental Demo, Navigator, to demonstrate how a VR application can be developed on HIVE. In Navigator, We use roaming as the application logic, and implement different 3D graphics as data presentations in different clients. Navigator loads the scenes specified by the user, then participants can select 3D models as the avatars of themselves to join in Navigator. After that, the participants can roaming around the scene, and interact with other participants simply. Passive viewer is also supported, who has no avatar of itself in the virtual world but can watch the scene and avatars of other participants. As the general implementation, Navigator Client use OpenGL to render the scene. Figure 8(a), a snapshot of a running Navigator Client, shows that three avatars (A, B and C) join in it, and A is the avatar of the participant who is viewing the window. In some cases, large screen display and high resolution of scenes are required. To meet the demands, we employ multi-screen tiled display using MSPR [16] as


369

the data representation. MSPR is a retained-mode based multi-screen parallel rendering system that offers the programmers with a OpenGL-like API. MSPR system could render the scene among multiple computers in parallel. In Navigator client with MSPR, we use three machines as a client to get a two-screen tiled display. The application logic part is run on one of the machines, and the other two are used as the rendering servers. Figure 8(b) illustrates the result of Navigator Client with MSPR. Avatar A is at the center of the client view, so one half of it is rendered by Server 1 and the other half is rendered by Server 2. Avatar B is flying from the region of Server 1 to the region of Server 2 and at the joint of them, so the image of it is also divided into two screens. But the Avatar C is rendered by Server 2 because it is fully in the region of it. Navigator is an experimental demo. More advanced and complex applications can be developed on HIVE to meet various demands of users. Currently, we are constructing more experiments and making tests on HIVE to collect results for further performance study and improvement.

(a) Navigator client

(b) Navigator client with MSPR

Fig. 8. Navigator client: three avatars A, B and C join in Navigator simultaneously, roaming and interacting with each other. It is from the view of Avatar A in the third view mode, so it can see avatar A itself which is always in the center of the view. Navigator client with MSPR: two-screen Tiled Display. The two images are captured from the two rendering servers respectively from the view of Avatar A.

6

Conclusions and Future Work

As the increasing demands of accommodating larger number of participants in VR, the scalability of participant and scalability of server are becoming the key issue. In this paper, after analyzing the requirements, we draw a conclusion that there are three fundamental factors which affect scalability of VR: node management model, management of servers and software development interface. We present a general middleware platform, HIVE, on which users can develop the scalable VR applications conveniently. HIVE has a three-tier node management model architecture to accommodate more participants, and provide an efficient management of servers to manage scalable number of clients. We employ HLA

370


as the reference to achieve reusability and interoperability of VR applications and provide the software development interface. To make HIVE more efficient and practical, we will continue to explorer the other aspects of VR application, such as communication model, scene object management and rendering system interface.

Acknowledgments This work is supported by National Grand Fundamental Research 973 Program of China under Grant No.2002CB312105 and the Key Project of Zhejiang Province under Grant No.2003C21012.

References 1. Bierbaum, A., Just, C., Hartling, P., Meinert, K., Baker, A., Cruz-Neira, C.: VR juggler: A virtual platform for virtual reality application development. Proceedings of IEEE Virtual Reality, Yokohama, Japan, March 2001, 89-96. 2. Shaw, C., Green, M., Liang, J., Sun, Y.: Decouple simulation in Virtual Reality with the MR Toolkit. ACM Transactions on Information Systems 11, 3 (1993), 287-317. 3. Macedonia, M.R., Zyda, M.J., Pratt, D.R., Barham, P.T. Zeswitz, S.: NPSNET: A Network Software Architecture for Large Scale Virtual Environments, Presence, 1994. 3, 4, 265-287 4. Greenhalgh, C. Benford, S.: MASSIVE: A Virtual Reality System for Teleconferencing, ACM Transactions on Computer Human Interfaces, Volume 2, Number 3 (1995), 239-261. 5. Watson, K.,Zyda, M.: Bamboo - a portable system for dynamically extensible, real time, networked, virtual environments. In 1998 IEEE Virtual Reality Annual International Symposium, 252-260. 6. Frcon, E., Stenius, M.: DIVE: A scaleable network architecture for distributed virtual environments. Distributed Systems Engineering Journal (special issue on Distributed Virtual Environments), 5(3) (1998), 91-100 7. Tramberend, H.: AVOCADO - A distributed Virtual Environment Framework. Proceedings of IEEE Virtual Reality 1999. Houston, Texas. 14-21. 8. Dongman L., Mingyu L., Seunghyun H.: ATLAS - A Scalable Network Framework for Distributed Virtual Environments, proceedings of ACM CVE’2002 , 47-54. 9. Wilson, S., Sayers, H., and McNeill, M.D.J.: Using CORBA Middleware to Support the Development of Distributed Virtual Environment Applications”, Proceedings of 9th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision, Plzen, Czech Republic, 98-105. (2001) 10. U.S. Department of Defense (DMSO): High level architecture run-time infrastructure programmer’s guide, Version 1.3 v5, http://www.dmso.mil. (1998) 11. U.S. Department of Defense (DMSO): High level architecture rules, High level architecture federate interface specification, High level architecture object model template specification Version 1.3, http://www.dmso.mil. (1998) 12. Simulation Interoperability Standards Committee (SISC) of the IEEE Computer Society.: IEEE Standard for Modeling and Simulation (M&S) High Level Architecture (HLA)-IEEE Std 1516-2000, 1516.1-2000, 1516.2-2000. New York: Institute of Electrical and Electronics Engineers, Inc., 2000.


371

13. Mark, D.H.: What is Scalability? ACM SIGARCH Computer Architecture News, 18, 4(1990), 18-21. 14. Dean, M.: The Scalability Problem, ACM Queue: Game Development, 1, 10 (2004), 66-73 15. Steve, B., Chris, G., Tom, R., James, P.: Collaborative Virtual Environment, Communications of the ACM, Vol. 44, No. 7 (2001),79-85. 16. LI, C., Jin, Z. F., Shi, J. Y.: MSPR: A Retained-Mode Based Multi-Screen Parallel Rendering System. In Proceeding of the 4th International Conference on Virtual Reality and its Application in Industry, Tianjin, China, 5444, 173-180 (2003)

Location-Based Device Ensemble Architecture for a Spatial Reality Enhancement of Distributed Multimedia Presentation Doo-Hyun Kim, Lila Kim, Hwasun Kwon, Dongwoon Jeon, and Songah Chae Sensor and Embedded S/W Lab. School of Internet and Multimedia Engineering Konkuk University, Seoul, Korea [email protected]

Abstract. Usual multimedia contents are represented as scenarios with markup language such as SMIL, and then delivered to be played back in terminal machines, e.g., PC or STB(Set-Top Box). However, these multimedia presentations are likely limited within single machine presentation where only the speakers and display devices attached to same computer system are used, so that the spatial reality is necessarily degraded consequently. In order to enhance even audio scenario presentation with such spatial reality, each audio clip for each audio actor, dubbing artist or background music should be assigned a separate speaker mostly close to the location where the author intended. In this paper, we propose a new architecture called L-DEAR(Location-based Device Ensemble Architecture) to support the spatial reality using location information of each node in a network environment. The core of the L-DEAR is IBEE(Immediate-But-Ephemeral device Ensemble) which is a set of nodes called out for playing back at least one assigned clip at a given time during the whole presentation. We will present the functional modules of L-DEAR and procedures for building IBEE, as well as functional requirements for L-DEAR such as extending the SMIL and using a global clock. Keywords: Spatial Reality, Location, Distributed Multimedia Presentation, Markup Language, Scenario.

1 Introduction In this paper, we propose an architecture for a new flavor of distributed multimedia presentation enhanced with the spatial reality. The usual multimedia contents are represented as scenarios with markup language such as SMIL(Synchronized Multimedia Integration Language) [1-3], and then these scenarios are played back in terminal machines, e.g., PC or STB(Set-Top Box). However, these multimedia presentations are likely limited within single machine presentation where only the speakers and display devices attached to same computer system are used. Any visual component of a scenario can not be presented beyond the single display device, and similarly, all the audio components are mixed into one audio signal and then output through a single audio device, i.e., speaker. In such paradigm, the spatial information Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 372 – 383, 2006. © Springer-Verlag Berlin Heidelberg 2006

Location-Based Device Ensemble Architecture

373

such as ”The audio clip A sounds at upper left of the audio clip B” is ignored, and the spatial reality is necessarily degraded consequently. In order to enhance even the audio scenario presentation with such spatial reality, each audio clip for each audio actor, dubbing artist or background music should be assigned a separate speaker mostly close to the location where the author intended. This enhancement is not believed to be achieved within single machine presentation paradigm, but possibly within a networked multi-node computing environment. A node is considered as a networked computing device with H/W and/or S/W for performing proper I/O operations such as decoding MP3 to its speaker. A device ensemble is defined as a set of nodes which are called out for playing back at least one assigned clip at a given time during the whole presentation. The device ensemble is definitely logical and ephemeral embodiment immediately brought into the execution of a presentation, thus, is called IBEE, Immediate-But-Ephemeral device Ensemble. Figure 1 shows the concept of the spatial reality enhancement where an IBEE is formed and engaged for presenting single scenario with multiple audio clips. The goal of this paper is to derive necessary building-blocks and their integrated architecture, called L-DEAR(Location-based Device Ensemble Architecture), for achieving the spatial reality enhancement by embodying IBEE effectively. This paper is organized into the following sections. In section 2, we will discuss architectural requirements that should be considered for deriving the L-DEAR. In section 3, we propose three possible approaches for extending the SMIL to incorporate the ability to express spatial reality requirements in the scripts for multimedia presentations. In section 4, we break down the L-DEAR architecture into five modules along with the procedures for assigning devices in an IBEE for a give presentation script. Especially, for the device assignment procedure, we discuss two approaches such as the convergence approach and divergence approach. In section 5, we deals with implementation issues by mainly focusing on the global clock and the session initiation with capability negotiation. We conclude in section 6 with discussions on further research directions.

2 Architectural Requirements In this paper, L-DEAR and its architectural building blocks are derived by listing the functional requirements covering diverse issues as follows: − Extending mark-up language for spatial relationship representation: The single machine presentation paradigm seems to have affected the mark-up language, e.g., W3C SMIL 2.0[1-3] not to consider representing spatial relationship among audio clips. It is required to extend this markup language so that authors can specify inter-clip relationships into scripts in the phase of authoring. − Acquiring location information of devices: There are many research results such as Cricket [4] for localization of mobile devices in indoor environment without using GPS. We may adopt these novel ideas for acquiring the location information of each device. Whichever location systems are used, L-DEAR should gather relatively accurate location information from each device, and use it to form a proper IBEE immediately.

374

D.-H. Kim et al.

Fig. 1. Location-based spatial reality enhancement

− Forming IBEE: The IBEE is actually to assign each clip to a specific node by seeking a match between the inter-clip spatial relationship specified in the authored script and the location information gathered from each node. This matching procedure is required to pursue an optimal matching between the authored spatial relationship requirement and physical locations of devices. − Facilitating global time within the IBEE: A device in an IBEE will eventually start playing an assigned audio clip. But if the device does no know the exact time planed to start or stop playing the specific clip, then the synchronized distributed presentation will be disordered. Therefore, we need to make the devices in an IBEE orchestrated on the basis of a global clock. As in case of the device localization, there are many excellent research results such as TMO (Time Triggered and Message Triggered Objects)[5-7] and NTP(Network Time Protocol)[8]. Among those, we may use TMO, since it supports a unified object oriented programming scheme for distributed and time-coordinated actions as well as global clock.


375

3 Mark-Up Language Extension 3.1 SMIL [1-3] The Synchronized Multimedia Integration Language (SMIL), recommended by the W3C(World Wide Web Consortium), is a declarative markup language based on the eXtensible Markup Language(XML) for scripting presentations containing multimedia; audio, video, image, and text. The SMIL has unique features supporting the timing and synchronization among heterogeneous types of media. A SMIL presentation is a structured composition of various media elements over a common timeline by using timing elements: <par>, <seq> and <excl>. The <par> element or parallel time container specifies that its children elements are all rendered in parallel. The <seq> element or sequential time container specifies that its children elements are played in sequence. It means a <seq> element's children are rendered one after the other. The <excl> element or exclusive time container specifies that only one of its children is played at a given time. If any element begins playing while another is already playing, the element in playing is stopped. In addition to the above the three basic elements, the SMIL provides timing attributes such as , and <end> for each element. The attribute explicitly specifies the start time of children of an element. The attribute specifies the duration of the children, and the <end> attribute means the end of an element. The SMIL provides a "test-attribute", e.g. system-Bitrate, system-captions, systemlanguage, system-required, and system-screen-size [3] to process an element only when certain conditions are true, for example, when the language preference specified by the user matches that of a media object. The switch element can express that a set of parts in a document are alternatives each other and that the first one fulfilling certain conditions should be chosen. This is useful to express that different language versions of an audio file are available, and that the client may select one of them. An author can specify a set of alternative elements from which only the first acceptable element is chosen. ... <par> />

In the above example, one audio object is selected to accompany the video object. If the system bitrate is 56,000 or higher, the object dutchHQ.aiff is selected. If the system bitrate is at least 28,800 but less than 56,000, the object dutchMQ.aiff is selected. If no other objects are selected, the alternative dutchLQ.aiff is selected, since it has no test attribute and no other test attributes evaluated to true. Authors should order the alternatives from the most desirable to the least desirable. Furthermore,

376

D.-H. Kim et al.

authors may wish to place a relatively fail-safe alternative as the last item in the switch so that at least one item within the switch is chosen. 3.2 SMIL Extension for Spatial Reality In order to enhance the spatial reality of multimedia presentation, it is necessary to extend the mark-up language to embrace the capabilities to allow the authors to express their spatial requests intended for each element or clip. We, in this paper, discuss three approaches to slightly extend the SMIL, including 1) default rule approach, 2) relative position approach, and 3) test-and-switch approach. 3.2.1 Default Rule Approach Basically, this approach is not to modify any of the current SMIL recommendation. Instead, the author leave SMIL player to assign each media object or clip to a proper device. The only thing that the author should know and follow is the default rule that the media objects shown in an element will be assigned to the devices by the SMIL player from left-to-right one by one at the presentation time. For example, in the following part of SMIL script, the audio2.mp3 clip will be assigned the left most speaker node in the IBEE, audio1.mp3 to the centered speaker, and audio3.mp3 to the right most speaker, respectively. ... <par> ...

This default assignment is in case when the IBEE could be organized to contain three speaker nodes. In case when the number of speaker nodes is less than or greater than the number of media objects, a curtain default rule will be applied depending on the implementation of the SMIL player. An example default rule that can be applied to the above case where only two speaker nodes exist in the IBEE is to assign the audio3.mp3 to the right speaker along with the audio1.mp3, that is, to assign all the surplus clips to the right most speaker node. 3.2.2 Relative Position Approach This approach, as shown in the following example, is differentiated from the default rule approach in that we need to add few attribute symbols so that the authors can express their requests to the SMIL player by specifying relative positions of each media object. The relative positioning attributes are such as posMaster, left, right, leftTo, rightTo, end center. The posMaster attribute is for appointing a clip as a reference point to the others. So the left and right attributes mean that the clips be assigned to the speakers located at left and right to the speaker assigned to the clips with posMaster attribute. The leftTo and rightTo attributes are used to specify more accurate positions. “loc = rightTo audio3.mp3” means this clip should assigned to one of speakers at the right to the speaker assigned for audio3.mp3. But, in case when the number of speaker nodes is less than or greater than the number of media objects, a


377

curtain default rule as in the default rule approach will be applied depending on the implementation of the SMIL player. ... <par> ...

3.2.3 Test-and-Switch Approach This approach is to use switch elements to allow the authors separately specify their spatial intensions according to the available number of devices, i.e., speakers in IBEE. For this approach, it is necessary to add a system attributes such as systemAudioDevNum and systemVideoDevNum and etc. The SMIL player organizes IBEE for a given presentation and then counts the available numbers of audio and display device nodes respectively. This counted number is compared with the specified values to systemAudioDevNum, for example, in the switch element to decide the proper part of the given SMIL document to be executed. ... <switch> <par systemAudioDevNum=2> <par>

...

4 L-DEAR Architecture The L-DEAR architecture as shown in Figure 2 consists of five major parts; 1) Extended SMIL Interpreter (ESI), 2) Device Information Base (DIB), 3) Orchestrator, 4) Global Time Base, and 5) Audio Decoding and Player(ADP). The ESI reads and interprets a SMIL script and then produces both of a time schedule for presenting clips and an IBEE. For producing the IBEE, the ESI refers Device Location Information Base(LIB) contained in DIB, and seeks matches between logical locations coded in SMIL script like “loc=left” in Figure 1 and physical locations of nearby devices. The best match is carried to the Orchestrator.

378

D.-H. Kim et al.

Fig. 2. Block diagram of L-DEAR

Fig. 3. A Spatial Relationship Tree(SRT)

The DIB is composed of CIB(device Capability Information Base) and LIB. These two information bases are acquired through a series of messages between the main node, i.e., orchestrator node and device nodes. A device node initiates a session with


379

the main node after the boot-up is finished, and then communicates with a series of messages to carry its physical location information and device capability information. The device capabilities may contain the list of audio formats that it supports. The Orchestrator executes presentation schedules over IBEE by feeding clip data files, e.g., audio.mp3, into each corresponding device in the IBEE along with start- and stop-time specifications for the clip. The TMO provides global time basis on which the main node and device nodes in the IBEE should refer for conducting designated actions at a given global time. Once the device node receives time specifications, then ADP of the node registers TMO with time-triggered action requests for executing start and stop functions in the player.

(a) Convergence Approach

(b) Divergence Approach Fig. 4. A Spatial Relationship Tree(SRT)

380

D.-H. Kim et al.

4.1 Device Assignment Procedure for IBEE The core of the L-DEAR is the device assignment procedure for constructing an IBEE. In this section, we discuss two approaches, that is, the convergence approach and divergence approach that can chosen in the implementation of L-DEAR. Ahead of our discussion about these two approaches, we need to explain a data structure, called SRT(Spatial Relationship Tree) for representing the spatial relationships among the clips. The SRT is a binary tree of which node denotes a clip and arc does relative positions, e.g., l, right, leftTo, and rightTo. As show in Figure 3, the given chunk of SMIL document is represented as a SRT with seven nodes assigned to each clip. 4.1.1 Convergence Approach This approach is to put a focus on assigning the media objects or clips to the device nodes closer to the node to which the posMaster clip is assigned. As shown in Figure 4(a), the a1, a5 and a6 are assigned to the s2 speaker node located at central part of the IBEE. 4.1.2 Divergence Approach While the Convergence Approach tends to assign the media objects or clips to the central part of IBEE, this Divergence Approach spreads the media objects to the device nodes far from the node to which the posMaster clip is assigned. As shown in Figure 4(b), while only the clip a1 has been assigned to the speaker node s2 at the central part, the clip a5 and a6 are assigned to the s1 and s2 speaker nodes located at left and right part of the IBEE, respectively.

5 Implementation Issues In this section, we discuss on two topics crucial in embodying the IBEE and executing presentations in the manner of controlling distributed and coordinated actions over a global clock. 5.1 Global Time Based Coordination of Distributed Multimedia Actions First of all, the global clock mechanism should be a primary basis on which we should implement our L-DEAR. There are many excellent research results such as TMO (Time Triggered and Message Triggered Objects)[5-7] and NTP(Network Time Protocol)[8]. Among those, we propose to use TMO, since it supports a unified object oriented programming scheme for distributed and time-coordinated actions as well as global clock. The TMO programming and specification scheme was established in early 1990's with a concrete syntactic structure and execution semantics to support economical reliable design and implementation of RT systems [5]. The TMO scheme is a generalstyle component programming scheme and supports design of all types of components including distributable hard-RT objects and distributable non-RT objects within one general structure [6].


381

TMOs are devised to contain only high-level intuitive and yet precise expressions of timing requirements. No specification of timing requirements in (indirect) terms other than start-windows and completion deadlines for program units (e.g., object methods) and time-windows for output actions is required. For example, priorities are attributes often attached by the OS to low-level program abstractions such as threads and they are not natural expressions of timing requirements. Once the high-level specification of timeliness requirements is registered, then the middleware does its best to meet the specification by using the CPU scheduler and other resource schedulers in the underlying node OS and network infrastructure. More importantly, the TMO scheme enables the programmer to design global time based coordination of distributed multimedia actions very easily. The scheme provides a sound foundation for programming and executing distributed multimedia actions requiring global synchronization among the media units or among the nodes collectively engaged in a multimedia presentation performance. The use of global time [9] can make it easy to measure QoS (quality-of-service) attributes such as delay jitter and synchronization skew[10]. Conventional approaches to measurement of QoS attributes at execution time were based on RTCP which basically uses QoS monitoring and feedback mechanisms built upon the assumption that the delay is symmetrical across the networks[11]. However, the symmetrical delay assumption is not so realistic and the RTCP based approaches do not produce high-accuracy measurements because of the feedback delay [12]. In contrast, a global time base of good precision, which is nowadays easy to realize due to advances in GPS and clock synchronization protocols [9], facilitates high-accuracy measurement of QoS attributes. 5.2 Session Initiation with Capability and Location Service In order to organize IBEE, the DIB(Device Information Base) should require nodes registered with their capability and physical locations, which LIB (Location Information Base) CIB(device Capability Information Base). All those actions including connection, capability registration, and location registration can be integrated into one protocol called signaling protocol. Among the various signaling protocol such as H.323 [13] and SIP(Session Initiation Protocol)[14], the SIP likely matches well for our implementation of the DIB. The SIP is an application layer control protocol for creating, modifying and terminating sessions with one or more participants. Entities in SIP are user agents, proxy servers and redirect servers. The SIP user agent listens for incoming SIP messages, and sends SIP messages upon user actions or incoming messages. The SIP proxy server relays SIP messages and the SIP redirect server returns the location of the host. Both the redirect and proxy server accepts registrations from users, in which the current location of the user is given. The location can be stored either locally at the SIP server, or in a dedicated location server. Deployment of such SIP servers enables L-DEAR to keep track of the locations of device nodes on the network [15]. In addition to the location service, SIP uses the Session Description Protocol (SDP)[16] for carrying out the negotiation for capability identification. Thus, SIP supports session descriptions that allow participants to agree on a set of compatible

382

D.-H. Kim et al.

media types. The information contained in SDP can be used to organize an IBEE proper to decode and play the clip files compressed with certain types of algorithms.

6 Conclusions We, in this paper, described our main theme, that is, to enhance spatial reality with IBEE(Immediate But Ephemeral device Ensemble). And also, by listing up primary architectural requirements, we derived an architecture called L-DEAR(Location-base Device Ensemble ARchiteture) to embody the IBEE. We proposed the convergence and divergence approaches for assigning networked device nodes to each media clip in the given distributed presentation. In addition, we discussed important issues in implementing L-DEAR, including such as the global clock mechanism and the location registration capability negotiation between main node and device nodes during the session initiation stage. We also raised the issues for extending the mark-up language, SMIL, for the enhanced ability to express spatial relationships among the clips. Especially, we discussed three approaches for extending SMIL including the default rule approach, relative position approach, and test-and-switch approach. All those approaches are applicable in implementing the extended SMIL interpreter of L-DEAR. We, however, could not discuss about some advanced topics such as measuring the degree of the spatial reality, and handling 3-D location information of the devices. Especially, the degree of the spatial reality will provide a human-centered objective function that the device assignment procedure should meet for enhancing the spatial reality ultimately. These topics are considered to be tackled later as further research issues. Acknowledgement. This paper was supported by Konkuk University in 2005.

References 1. W3C, “Synchronized Multimedia Integration Language (SMIL 2.0)”, 2001, http://www.w3.org/TR/2005/REC-SMIL2-20050107/ 2. Thae-hyun Kim, Kyu-Chul Lee, “SMIL (Synchronized Multimedia Integration Language)Multimedia Synchronization Language”, Journal of Korean Multimedia Society, 2000 3. Sung-hyun Han, “A Study on the Synchronized Multimedia Integration Language: SMIL”, Journal of Korean Society for Internet Information, 2001 4. Balakrishnan, H., R. Baliga, D. Curtis, M. Goraczko, A. Miu, N.B. Priyantha, A. Smith, K. Steele, S. Teller, K. Wang, “Lessons from Developing and Deploying the Cricket Indoor Location System,” http://nms.lcs.mit.edu/cricket, Nov. 2003. 5. Kim, K.H. et. al, “Distributed computing based streaming and play of music ensemble realized through TMO programming,” WORDS 2005, Kyungjoo, Korea, Feb. 2-5, 2005. 6. Kim, K.H., Ishida, M., and Liu, J.Q., "An Efficient Middleware Architecture Supporting Time-Triggered Message-Triggered Objects and an NT-based Implementation", Proc. ISORC ’99 (2nd IEEE CS Int'l Symp. on Object-Oriented Real-time Distributed Computing), St. Malo, France, May, 1999, pp.54-63.


383

7. Kim, D.H., and K H (Kane) Kim , “A TMO-based Software Architecture for Distributed Real-time Multimedia Processing,” Proc, WORDS 2003(IEEE Workshop in Objectoriented Real-time Dependable Systems), Guadalajara, Mexico, Jan. 2003. 8. http://www.ntp.org 9. Kopetz, H., 'Real-Time Systems: Design Principles for Distributed Embedded Applications', Kluwer Academic Publishers, ISBN: 0-7923-9894-7, Boston, 1997. 10. Steinmetz, R., "Human Perception of Jitter and Media Synchronization," IEEE Journal of Selected Areas in Communications, Vol. 14, No. 1, Jan. 1996, pp. 61-72. 11. 11. IETF, “RTP: A Transport Protocol for Real-Time Application,” RFC 1889 12. Kim D.H., K H (Kane) Kim, S Liu, Jin H. Kim, "A TMO-based Approach to Tolerance to Transmission Jitters in Tele-Audio Services," Int'l Journal of Computer Systems Science & Engineering, Vol. 17, No. 6, Nov. 2002, pp. 325-334. 13. ITU-T, “Packet-based multimedia communications systems,” Recommendation H.323 14. M. Handly, H. Schulzrinne, E. Schooler, and J. Rosenberg, “SIP: session initiation protocol,” Request for Comments 2543, Internet Engineering Task Force, Mar. 1999. 15. Kim, D.H., J.Y. Kwak and K.H. Lee, Using the Session Initiation Protocol for Connected Multimedia Services in a Ubiquitous Home Network Environment, MIPS 2004, Grenoble, France, November 16-19, 2004. 16. Handley, M. and V. Jacobson, “SDP: session description protocol,” RFC 2327, April 1998.

Air-Jet Button Effects in AR Yeongmi Kim1, Sehun Kim1, Taejin Ha2, Ian Oakley3, Woontack Woo2, and Jeha Ryu1 1

Human-Machine-Computer Interface Lab., Dept. of Mechatronics, Gwangju Institute of Science and Technology, Oryong-dong, Buk-gu, Gwangju 500-712 Republic of Korea {kym, kshfire, ryu}@gist.ac.kr http://dyconlab.gist.ac.kr 2 U-VR Lab., Dept. of Information and Communication {tha, wwoo}@gist.ac.kr http://uvr.gist.ac.kr 3 Smart Interface Research Team, Electronics and Telecommunications Research Institute, 161 Gaejeong Dong, Yuseong Gu, Daejeon, 300-700, Korea [email protected] http://etri.re.kr

Abstract. Providing haptic cues can generate increased levels of presence in users as they interact with tangible objects. In this paper, we present button effects delivered by air-jet displays with the aim of providing click-like sensations when virtual buttons in an AR environment are pressed. We derive the profile of the haptic output by measuring the force profile of real physical buttons. To validate our concept, we have built an AR system featuring a cellular phone model which users can tangibly manipulate through a physical AR marker object. When users press a button on the model, they experience a corresponding click-like feeling at their fingertip. In this way, our system enables users to press a button on an AR model and experience a force profile generated by our pneumatic array directly derived from the act of pushing a button in the real world. Keywords: Air-jet, button effect, AR, fingertip point extraction.

1 Introduction In order to provide immersive interaction, multisensory feedback composed of visual, auditory and haptic elements needs to be merged together in an appropriate way. Many researchers have shown that performance is substantially improved when haptic sensations are added to visual and auditory environments [5]. We suggest that appropriate haptic cues are an essential component required to support the simple and natural manipulation of widgets and controls such as buttons, sliders and joysticks. Indeed, a haptic button has been a popular mechanism for controlling computer VR applications and the force profile for buttons has been implemented on kinesthetic devices such as the PHANToMTM in several APIs in order to present the realistic feeling virtual buttons. However, these devices are expensive and due to their bulk Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 384 – 391, 2006. © Springer-Verlag Berlin Heidelberg 2006

Air-Jet Button Effects in AR

385

and desk-based interaction paradigm, unsuitable for many AR scenarios. Tactile devices which present stimuli to the skin may be more suitable in this instance. Of these, pneumatic tactile devices, which feature an array of air jets directed onto the skin, are lightweight, able to provide pressure and to modulate the shape it is displayed in [1]. In addition, they are cost-effective and they have unconstrained workspaces when compared to kinesthetic feedback devices. As the demand for additional haptic cues in AR increases, applications are being implemented for educational, military, entertainment and medical training purposes. There are several systems that use kinesthetic feedback devices to provide force feedback in AR. Kenneth Fase et al. proposed a system that features virtual training for welding tasks using a 3DOF haptic device [6]. Matt Adcock et al. have developed medical simulations using the PHANToMTM [8]. Also, Christoph W. Borst et al. presented haptic feedback for virtual panels using a force feedback glove [4]. In addition to force feedback, there have been some attempts to combine tactile feedback with augmented reality. Buchmann et al. used buzzers mounted in a glove in an urban planning workspace so that users could feel vibration when their hands made contact with an AR model of a city block [12]. Another application is based on an air-jet display consisting of 100 air nozzles and which provides feedback force of objects through an external air receiver [14]. In this paper, we present a button effect delivered by an air-jet display for interaction with a tangible AR object. In order to use appropriate pressure values in our system, we captured real button pressing forces and time plots. Furthermore, we propose a method which extracts the tips of the fingers by finding the convex hull points of the hand object and constraining the radius of palm. We use this in our AR environment, so that users can naturally interact with our haptically enhanced virtual object with their hands.

2 Pneumatic Tactile Interface In order to produce the sensation of clicking a button, pneumatic tactile system is adopted in this paper since our pneumatic display can deliver appropriately changing stimuli to the surface of the skin as the button moves through the process of being pressed. 2.1 System Configuration The existing air-jet system is used to presents button effect of the tangible object [13]. Overall configuration of our air-jet system is simple such that a user attaches air-jet display on the fingertip and input commands are coming from PC through an RS232 serial communication to Mexx ATMega 128 microprocessor controlling the state of the valves. As the valves, Yonwoo Pneumatic YSV10s are used to control the flow of air. The overall system configuration is shown in Fig.1. It consists of an air supplier, regulator, interface and control board, and pneumatic tactile display. The air supplier provides pressurized air and the regulator keeps the pressure.

386

Y. Kim et al.

Fig. 1. System configuration of pneumatic tactile interface

The air-jet display forms 5x5 arrays on the finger pad and additional 5 air nozzles which are contacted each side of finger in order to produce the lateral force. The display features air nozzles with an external diameter of 2.4mm and internal diameter of 1.5 mm. Figure 2 illustrates the display design. In order for more natural interaction, the air-jet stimulates the fingertip of index finger which usually used to press buttons. In accordance with our previous psychophysical experiments, it can deliver richer information since air-jet display is mounted on fingertip while feeling of buzzers on the finger nails may be missed for some users as Volkert Buchmann mentioned [12].

Fig. 2. Display design

3 Experiments on Measuring Force and Time of Buttons In the previous study on PTI (Pneumatic Tactile Interface) [13], we concluded our experimental data could support that we could deliver cue capability to grow or shrink in magnitude by increasing number of air-jets, and sensory threshold for pneumatic cues are considerably greater than those for vibration cues. Thus, we implemented button effects with pneumatic tactile display by modulating the intensity of air-jet. Therefore, when a button of virtual models is pressed, users are able to feel button clicking feeling. In order to produce button effects, pressing force of the real button is measured and then applied to the pneumatic tactile system. In addition to the force profile, time value is one important factor for producing the button effects. Therefore, experiments have been conducted to figure out the pressing force and time value.


387

3.1 Experimental Set-Up Creating the click-like sensation which makes people believe as if they press the physical button is desirable goal of our work. The characteristics of virtual buttons are reported in the literature [11]. In order to produce click-like feeling, we measured the forces while users press the real button. In order to measure the force-profile of physical buttons of the cellular phone [7], Tekscan's FlexiForce Sensors A201-1(01 lb. force range) and a real button panel were used [Fig.3.].

Ⓡ

Fig. 3. Measurements on button force

Basically, this sensor is equal to potentiometer; namely, the output of the sensor is proportional to the input force. AD converter in ATmega 128 is used to convert analog input to 10 bit digital data. Also, low-pass filter is adopted in order to avoid the interference of high frequency. 3.2 Force - Displacement Curve Experiment The total length of a physical button is 0.2 mm. In order to find out the force profile of this button, we measured force data as pressuring down the button with 0.005 mm increment by using micrometer and 3000 samples were acquired at each length. As shown in Fig. 4. the measurement data of real button clicking force is similar to the well known virtual button force model. In other words, at the initial position force is linearly increased, then it suddenly decreasing, and then the force is increased remarkably. Also, it describes the average force of first peak which is about 160(0.7 N) and the average force for dead band is 93(0.4N).

300 250 ] s ti b 0 1 [ e c r o f

200 150 100 50 0

0 15 30 45 60 75 90 05 20 35 50 53 56 59 70 85 00 1 1 1 1 1 1 1 1 1 2

distance[um]

Fig. 4. Result of measurement of button force

388

Y. Kim et al.

3.3 Force - Time Curve Experiment The timing is also key factor to provide button effects by the air-jet display. We conducted another experiment in order to find out the timing information and the characteristics of force profile related to the time. Eight subjects were participated in this experiment and they pressed the same button 20 times and all data were automatically saved as xls file format. Fig. 5. illustrates the result of experiment.

User 1

User 2

300

700

250

600

200

500 400

150

300

100

200

50 0

100 1

9 17 25 33 41 49 57 65 73 81 89 97

0

1

15 29 43 57 71 85 99 113 127 141 155 169 183

User 3

User 4

350

300

300

250

250

200

200

150

150

100

100

50

50 0

1

14 27 40 53 66 79 92 105 118 131 144 157 170

0

1 10 19 28 37 46 55 64 73 82 91 100 109 118 127 User 6

User 5 500 450 400 350 300 250 200 150 100 50 0

250 200 150 100 50 1

16 31 46 61 76 91 106 121 136 151 166 181 196 211

0

1 11 21 31 41 51 61 71 81 91 101 111 121 131 141

User 7 200 180 160 140 120 100 80 60 40 20 0

User 8 300 250 200 150 100 50

1

15 29 43 57 71 85 99 113 127 141 155 169 183

0

1

9

17 25 33 41 49 57 65 73 81 89 97 105

Fig. 5. Result of experiment on measuring force-time curve for a real button


389

These graphs show slightly different force profile comparing to the results described in section 3.2 since the acceleration while pressing the button may be affected to the results. Also, since the button of cellular phone has short length and the dead band time is too short to sense, the real force profile of button shows small dead band. The average time for initial increasing band is about 200 msec and the average time for dead band is 60 msec [60 msec = (30 samples x 1 sec) /500 samples]. 3.4 Mapping Displayed Force to Pressure Pressure can be obtained by following equation: Pressure [psi] = Force [pounds-force] / Area [inch2].

(1)

That equation refers that if the force and area are known, we can generate the same amount of jet force at the nozzle. As we described the size of tube is known factor and mean of 500 sample force data is given in Fig. 5. Therefore, button effects delivered by air-jet display are provided according to the results of experiments. In order to produce force (0.7 N), the area of tubes should be calculated. Area [inch2] = πr 2.

(2)

In our display, the radius of tubes is 0.75 mm (0.0295 inch) so the area of each tube is 0.002734 [inch2]. Therefore, 58 psi is required to produce the first peak force (0.7 N), when only one air-jet nozzle is used. However, by increasing the number of air-jet nozzles, less pressure is needed since total pressure is distributed into several air-jet nozzles. For example, six air-jets produce the same amount pressure at about 10 psi.

4 Interaction with Tangible Object in AR The primary goal of our system is to integrate the pneumatic feedback derived from the pressure measurements of the real button with an AR system which allows the user to interact with graphical virtual models. We aim to add touch feedback to this graphical AR environment. To achieve this we focused on a model of a cellular phone and developed an AR system which displays the model and detects the user’s fingertip position. When the user’s finger touches a button on the phone model, we display the stored button pressure profile on our pneumatic array, in order to create the physical sensation that the button is being clicked. Typical fingertip detection systems rely on fiducial markers [2] attached to various portions of the hands, but this is unsatisfactory as problems of occlusion can occur, and also it can be both unnatural and inconvenient to use. Our system extracts the fingertip points by locating the convex hull points of the hand and using the palm as a constraining radius. Thus we allow the users to naturally interact with the virtual object using their hands. This system is explained in more detail below. 4.1 2D Based Fingertip Point Extraction and Collision Detection The vision-based fingertip capture interface enables a more natural and intuitive style of interaction. In our system, users are able to interact with virtual objects using their hands and experience the results of their interactions through our pneumatic tactile

390

Y. Kim et al.

display. They are able to push buttons on the augmented virtual game-phone, and feel corresponding tactile sensations. This system is enabled by a 2D based fingertip point extraction and simple collision detection. In order to extract fingertip points, we use the “3 coins algorithm” [3]. However, this leads to the generation of a large number of convex hull points so we filter this data with the constraint that the fingertips must be relatively close to the palm: within 1.5 times its radius. We visually detect and display the hand using a combination of a skin color distribution detection algorithm [9] and a segmentation system designed to reduce visual occlusion [10]. 4.2 Implementation After the 2D finger position has been attained, this data can be easily integrated into the coordinate space of the 3D model. A simple collision detection algorithm then determines if the user is close to or touching a virtual button. When the user’s finger is near a button, we display a blue ‘cursor’ on its tip and if that blue point collides with a button, the air-jet button effect is displayed. Figure 6 shows the user’s fingertip approaching a button (left) and when it is on a button (right).

Fig. 6. Fingertip point extraction and providing air-jet button effects

5 Conclusion and Future Work The proposed system produces button effects whereas most tactile displays have mainly provided weight, shape, meaningful texts and so on. By conducting the experiments on force-distance and force-time curve, it became possible to generate jet force at the nozzle according to the physical clicking force. In AR environment users are usually not perceived any haptic feedback, since augmented objects do not provide physical haptic feedback. In addition, manipulating the widget such as buttons, sliders, and joysticks is related to the performance of the task. Therefore, we provide button effects delivered by air-jet display while interacting with tangible objects for immersive interaction. From the several fundamental studies, we examined air-jets let users feel the button effects and interact with an augmented object impressively. However, evaluation of


391

this system will be done in the future to see effectiveness of our system. Also, not only button clicking feeing but also the shape of button will be provided.

Acknowledgments This research was supported by the Republic of Korea’s Ministry of Information and Communication (MIC) through the Realistic Broadcasting IT Research Center (RBRC) at the Gwangju Institute of Science and Technology (GIST), ICRC funded by the Ministry of Science and Technology of Korea and the UCN Project, the MIC 21C Frontier R&D Program in Korea and ICRC at GIST.

References 1. Amemiya, K. & Tanaka, Y., “Portable Tactile Display Using Air Jet”, In the International Conference on Artificial Reality and Telexistence, 1999. 2. ARToolKit, http://www.hitl.washington.edu/artoolkit/ 3. Avis, D. and Toussaint. “An optimal algorithm for determining the visibility of a polygon from an edge”, IEEE Trans. Comp. C-30 (1981), 910-914 4. Christoph W.Borst, and Richard A. Volz, “Preliminary Report on a Haptic Feedback Technique for Basic Interactions with a Virtual Control Panel”, EuroHaptics 2003 conference, pp. 1-13. 5. Hakan G., Benjamin P., Sankar J., & Uma J., “Design of a Haptic Device for Weight Sensation in Virtual environments”, ASME International Mechanical Engineering Congress & Exposition, 2002 6. Kenneth Fase, Timothy Gifford, Robert Yancey, “Virtual Training for Welding”, ISMAR, pp. 298-299, Third IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'04), 2004. 7. LG-KV 3600, http://www.cyon.co.kr/good/product/product_view1.jsp?product_id=165 8. Matt Adcock, Matthew Hutchins, Chris Gunn, “Augmented Reality Haptics: Using ARToolKit for Display of Haptic Applications”, 2nd Int'l Augmented Reality Toolkit Workshop, Tokyo, Japan, October 7th, 2003. 9. Taejin Ha and Woontack Woo, “Bare Hand Interface for Interaction in the Video seethrough HMD based Wearable AR Environment”, to appear in ICEC 2006 10. Taejin Ha, Yeongmi Kim, Jeha Ryu and Woontack Woo, “Enhancing Immersiveness in Video see-through HMD based Immersive Model Realization”, IEEK 06, pp.685-686 11. Timothy Miller, Bobert Zeleznik, “The Design of 3D Haptic Widget”, 1999 Symposium on Interactive 3D Graphics Atlanta GAUSA 12. Volkert Buchmann, S. Violich, M. Billinghurst, A. Cockburn., “FingARtips: gesture based direct manipulation in Augmented Reality”, In Proceedings of the 2nd international conference on Computer graphics and interactive techniques in Australasia and SouthEast Asia (Graphite 2004). 15-18th June, Singapore, 2004, ACM Press, New York, New York, pp. 212-221. 13. Yeongmi Kim, Ian Oakley, Jeha Ryu, “Combining Point Force Haptic and Pneumatic Tactile Displays”, EuroHaptics 2006, pp.309-316, Paris, France 14. Yuriko Suzuki, Minoru Kobayashi, “Air Jet Driven Force Feedback in Virtual Reality”, Computer Graphics and Applications, IEEE, 25(1), 44-77, 2005

Perceptualizing a “Haptic Edge” with Varying Stiffness Based on Force Constancy Jaeyoung Cheon and Seungmoon Choi Virtual Reality and Perceptive Media Laboratory Department of Computer Science and Engineering, POSTECH Pohang, Republic of Korea {icejae02, choism}@postech.ac.kr

Abstract. This paper introduces a novel haptic rendering technique devised to perceptualize a “haptic edge” correctly with respect to its stiffness and height models. Our previous study showed that the traditional penalty-based haptic rendering methods are not adequate to the collocated data of surface topography and stiffness since surface topography perceived by the user can be distorted from its model. In order to overcome the problem, we have developed a topography compensation algorithm based on the theory of force constancy which states that the user maintains a constant contact force when s/he strokes a surface to feel its topography. To the best of our knowledge, our technique is the first of its kind that explicitly considers the effect of user exploratory patterns in haptic rendering. Computationally, the algorithm is adaptive and efficient, not requiring any preprocessing of original data. We also demonstrate the performance and robustness of the proposed algorithm through a psychophysical experiment.

1

Introduction

The most important requirement of haptic rendering is that the properties of a virtual object perceived by the user must closely agree to those modeled in the virtual environment. However, the requirement can be often prohibited by many causes such as the effect of haptic interface dynamics on the final dynamics that the user feels [1], instability of haptic interaction between the device and the user [2][3], and a haptic rendering technique that fails to adequately account for the governing physics of the target attribute [4]. All of these shortcomings can lead to the user’s perception of unintended artifacts in addition to the modeled haptic attributes, or only artifacts in the worst case. A recent survey on the issues and advanced techniques of haptic rendering can be found in [5].

This work was supported in parts by grant No. 1RE0601701 from POSTECH and grant No. R01-2006-000-10808-0 from the Basic Research Program of the Korea Science and Engineering Foundation (KOSEF). The authors wish to thank Prof. Hong Z. Tan for valuable discussions and Prof. Gerard Jounghyun Kim for carefully proof-reading this paper.


Perceptualizing a Haptic Edge

393

(a) The surface height map. Higher re- (b) A cross-section of the height map with gions are coded with lighter colors. the trajectory of a haptic probe tip. Fig. 1. “Protein-on-mica” data. See the text for details.

Haptically recreating virtual environment dynamics with high fidelity can be much more difficult in the applications of scientific data perceptualization. Data perceptualization is an emerging interdisciplinary field extended from data visualization so as to utilize other senses such as touch and sound in addition to vision. A pioneering work of data perceptulization was Project GROPE [6] that allowed users to perceptualize the docking of molecules through the use of a force feedback device and a head-mounted visual display. Subsequently, Taylor et al. [7] developed a teleoperation system that interfaced a force feedback display to a scanning tunneling microscope providing both visual and haptic feedback of measured data. Early haptic volume rendering algorithms [8][9] used the local gradient as the surface normal and force transfer functions. Newer systems have incorporated proxy-based haptic rendering techniques for volumetric data rendering [10][11]. Recently, Maciejewski et. al. [12] released a data perceptualization system that included a haptic rendering technique for multiple volumetric data sets. Yano et. al. [13] showed that the perception of stiffness distribution defined on a flat surface may not match to its model and proposed a simple compensation technique that prewarps the stiffness model. Unlike other virtual reality applications that emulate real environments, the attributes to render are frequently collocated in data perceptualization. One such example that has motivated our work is provided in Fig. 1(a). The figure shows a surface height map of patches of bi-lipid membrane with embedded proteins on a mica substrate with lighter colors corresponding to higher surfaces. The topography map was acquired with a scanning probe microscope that can measure various nanometer-scale collocated features on a flat substrate [14]. The image shows a nearly circular protein membrane patch surrounded by a halo (presumably of lipids that have dissociated from the membrane) resting on an atomically flat mica substrate. Because the membrane patch is filled with a periodic array of the transmembrane protein, it should be considerably stiffer than the halo of dissociated lipids, but not as stiff as the mica substrate. The height profile of the data set is illustrated in Fig. 1(b) with a solid line. For convenience, we will refer to this image as the “protein-on-mica” data set (see [15] for further details about the data set).

394

J. Cheon and S. Choi

In our previous study [15], we attempted to render the “protein-on-mica” data set with the traditional penalty-based haptic rendering techniques (e.g., [16][17]). A user could tap on and stroke the resulting virtual haptic surface to gauge both the local stiffness and topography, respectively. However, it was consistently reported that the halo regions felt lower than the surrounding protein and mica regions in opposite to the model. The observation was confirmed by recording probe-tip trajectories during a user’s stroking of the data set. A probe-tip trajectory simplified for clear demonstration is provided in Fig. 1(b). The figure shows that the probe tip was “dipped” in the halo region against the height map. Note that the penalty-based techniques are originally designed to render an object shape with uniform stiffness. Our follow-up study revealed that the inversion of perceived height difference around the halo region occurs due to a human exploratory behavior named force constancy [15]. Force constancy states that a user tends to maintain a constant penetration force while stroking a virtual surface in order to perceive its topography. Applying force constancy to the “protein-on-mica” data set where the mica region had lower stiffness than its surrounding regions, we can confirm that the user maintaining a constant contact force resulted in larger penetration depth in the halo region. If the increase in penetration depth exceeded the modeled height difference between halo and mica, the halo region was incorrectly perceived to be lower than the mica region against their modeled height difference. More detailed analysis using force constancy is provided in Sec. 2. The “protein-on-mica” data set is an example that substantiates the need of new rendering algorithms for correct perceptualization of data sets with collocated surface topography and stiffness. Responding to the need, we have developed a topography compensation algorithm based on force constancy, concentrating on rendering of a haptic edge. By the “haptic edge,” we refer to a region where surface height changes abruptly (just as the halo region of the “proteinon-mica” data set), as opposed to a visual edge where color varies quickly. The method adaptively adjusts surface height topography by estimating user-applied force based on force constancy. The algorithm is very computationally efficient, and does not require any preprocessing of the data. The performance of the rendering algorithm was extensively evaluated in terms of both stimulus characterization and human perception. The results showed that our algorithm is very effective and robust at correctly rendering a haptic edge with varying stiffness, without any perceptual distortions. The performance evaluation has also unveiled a few new research issues that need to be resolved to extend our current algorithm to be applicable to a general case of continuously varying surface topography with nonuniform stiffness. The remainder of this paper proceeds as follows. In Sec. 2, we briefly review the theory of force constancy. The haptic rendering method is proposed in Sec. 3, followed by the results of comprehensive performance evaluation in Sec. 4. We conclude the paper in Sec. 5 with a plan for future work.


2

395

Force Constancy

The theory of force constancy states that users maintain a constant penetration force when they stroke virtual surfaces in order to perceive the surface topography. Our previous psychophysical study showed that the users indeed maintained a constant contact force when a surface was not too soft (surface stiffness = 0.3 – 1.1 N/mm) [15]. Stiffness less than 0.3 N/mm renders a very soft surface and thus does not provide a clear boundary of the surface. The users tended to use smaller penetration forces in this case. We can obtain a mathematical expression for perceived height difference subject to nonuniform surface stiffness based on force constancy. Consider a simple case of two vertical virtual planes depicted as P1 and P2 in Fig. 2. The heights of the two surfaces are denoted by h1 and h2 along the z-axis (Δh = h2 − h1 > 0 in Fig. 2), and the stiffness values are by k1 and k2 , respectively. The higher plane P2 is closer to the user than the lower plane P1 . We also assume that force is rendered with the penalty-based method by fz = k (h − pz ) where pz is the probe-tip position normal to the planes. If the probe tip is under P1 , h = h1 and k = k1 , and if under P2 , h = h2 and k = k2 . As the user strokes the virtual surface from P1 to P2 , the perceived height difference Δpz traced by the probe tip can be computed as: Δpz = Δh + Δhd .

where Δhd = −fp

1 1 − k2 k1

(1)

.

(2)

Here, Δhd represents the amount of distorted height difference due to the stiffness change and fp is the constant penetration force applied by the user. The detailed derivation is provided in [15]. Eqn. 1 shows that the perceived surface height difference has two components. The first term (Δh) is the modeled height difference that we intend for the user to feel. The second term (Δhd ), however, introduces a distortion in perceived surface height difference when k1 = k2 , i.e., the two planes are not equally stiff. Depending on parameter values, we can think of three possible scenarios:

Fig. 2. Top view of the haptic rendering of two vertical planes. The symbol at the bottom represents a user facing the vertical surface.

396


1. k1 = k2 : Δhd = 0 ⇒ Δpz = Δh. This is the case that most haptic rendering algorithms assume. The probe-tip follows a trajectory that is parallel to the modeled virtual surface topography. 2. k1 > k2 : Δhd < 0 ⇒ Δpz < Δh. This is a case similar to that encountered with the “protein-on-mica” data set if we consider P1 to be the lower and stiffer mica substrate and P2 to be the higher and softer halo surface. If |Δhd | > |Δh|, the higher plane P2 is perceived to be lower than the lower plane P1 . 3. k1 < k2 : Δhd > 0 ⇒ Δpz > Δh. In this case, the surface-height difference between P1 and P2 is perceived to be greater than what it should be.

3

Topography Compensation Algorithm

This section presents the compensation algorithm developed to deliver a haptic edge defined on a surface with nonuniform stiffness in a perceptually correct manner. Our approach is adaptive since proper compensation requires the knowledge of user-applied force (fp in Eqn. 2) that is unavailable in preprocessing. The basic idea is to cancel out Δhd (see Eqn. 1) in every haptic update interval. This would make perceived height difference Δpz equivalent to modeled difference Δh. We thus estimate Δhd in each haptic update interval and adjust the surface height using the estimated value. At time instance t, let the probe-tip position of a haptic interface be p(t) = (px (t), py (t), pz (t)). Without loss of generality, we assume that a height map h(·, ·) and a stiffness map k(·, ·) are defined on xy plane, consistently with Sec. 2. We also denote that h(t) = h(px (t), py (t)) and k(t) = k(px (t), py (t)) for simplicity. Consider t = n where n is a discrete time index for haptic update loop. In each loop, we compute Δhc (n) to compensate the height distortion of Δhd (n). Using the compensated height difference Δh + Δhc (n) instead of Δh in Eqn. 1 results in Δpz (n) = Δh(n) + [Δhc (n) + Δhd (n)] . (3) Therefore, setting Δhc (n) = −Δhd (n) ensures that the modeled height difference will be identical to the perceived difference. The critical part of the topography compensation algorithm is estimating Δhd (n) in Eqn. 2. When k(n) = k(n − 1), it is clear that Δhd (n) = 0. When k(n) = k(n − 1), user-applied force fp needs to be estimated. In general, a force sensor is required to exactly compute fp . However, force sensors are not available in many haptic interfaces, especially those of impedance type. Instead, our estimation relies on two fundamental facts on the human motor characteristics. The first one is force constancy. If the user maintains a constant penetration force during stroking a surface, the force that the user is applying to the surface is in equilibrium with the force that the haptic interface is producing. This fact allows to use the force computed by a rendering algorithm for estimating fp . The other consideration is the human motion bandwidth. A haptic update rate (typically 1 kHz) is much faster than the human motion bandwidth (less than 10 Hz), which implies that user-applied force cannot change abruptly within a few


397

haptic update intervals. Taken the two facts together, we estimate user-applied force fp (n) as: fp (n) = fz (n − 1) . (4) where fz (n − 1) is a force command along the z-direction computed in the previous update interval by the haptic rendering algorithm. Then, the estimated ˆ d (n) is height distortion term Δh 1 1 ˆ d (n) = −fz (n − 1) Δh − . (5) k(n) k(n − 1) Finally, our topography compensation algorithm can be summarized as the following three computational steps: (1) Height Compensation Term ⎧ ⎨0 1 1 Δhc (n) = − ⎩ fz (n − 1) k(n) k(n − 1)

if k(n) = k(n − 1) if k(n) = k(n − 1)

.

(6)

(2) Compensated Height hc (n) = h(n) +

n

Δhc (n) .

(7)

i=1

(3) Force Command fz (n) = k(n) (hc (n) − pz (n)) .

(8)

Note that in Eqn. 7, the height compensation terms need to be accumulated in order for the previous height distortions to be accounted for.

4

Experiment

We measured the reliability of the topography compensation algorithm for delivering the information of a haptic edge in a psychophysical experiment. The traditional penalty-based method was also tested for performance comparison. During the experiment, we recorded probe-tip trajectories felt by the subjects’ hand and used them to judge the similarity of rendered height differences to their models. This section reports the detailed design and the results of the psychophysical experiment. 4.1

Methods

This section describes the design of the psychophysical experiment. Apparatus: A PHANToM model 1.0A with a stylus shown in Fig. 3(a) is used in the experiment.

398


(a) A subject performing the experiment. (b) The visual scene shown on the monitor. Fig. 3. The setup of the psychophysical experiment

Subjects: Seven subjects (S1–S7) participated in the experiment. S7 was female, and the other subjects were all male. All subjects self-declared to be right-handed, except for S3. S2 and S4 were experienced users of the PHANToM. S1, S5, and S6 were generally familiar with haptic interfaces, but not as experienced as S2 and S4. S3 and S7 had not used any haptic interfaces prior to their participation to this experiment. The age of the subjects ranged from 21 to 32 years with the average of 25.4. No subjects reported any known sensory or motor impairments with their upper extremity. Stimuli: The haptic stimuli used in the experiment consisted of two vertically adjoined planes facing the user as shown in Fig. 2. A haptic edge between the two planes is modeled using Hanning windows (half-cycle sinusoidal functions) that smoothly connected the two planes in both height and stiffness in order to prevent rendering instability. The equations of the Hanning windows were equivalent to those used in our previous study [15]. The window width was set to 4 mm which resulted in stable transitions across the haptic edges in all experimental conditions. During the experiment, a computer monitor displayed visual information only to provide a spatial reference to the subjects. As shown in Fig. 3(b), the visual scene consisted of two blocks representing the starting and ending regions for stylus stroking, a blue cone showing the stylus-tip position, and texts asking a question to the subjects. Note that the text appeared on the screen only when the subjects were about to enter a response. Experimental Conditions: Table 1 summarizes the parameter values used in each experimental condition. The height of a haptic edge (Δh) was fixed to 3 mm in all conditions. Three sets of stiffness values were tested, resulting in three conditions (C1–C3 in the table). The table also shows expected perceived height differences (Δpz ) for a range of used-applied forces (0.5 – 3.0 N) when the penalty-based rendering rule was employed. The range covers the distribution (0.87 – 2.23 N) of the preferred contact force levels of the participants measured in our previous study [15]. The conditions were designed so that the penaltybased algorithm would result in different levels of distortion (Δpz = 3.0 mm). Considering the human detection thresholds of surface height difference ( 0.66


399

Table 1. Parameter values used in the psychophysical experiment Experimental k1 k2 h1 h2 Δh Δpz (mm) Condition (N/mm) (N/mm) (mm) (mm) (mm) fp =0.5 fp =1.0 fp =2.0 fp =3.0 (N) C1 0.6 0.3 -1.5 1.5 3.0 2.17 1.33 -0.33 -2.00 C2 0.9 0.3 -1.5 1.5 3.0 1.89 0.78 -1.44 -3.67 C3 0.9 0.6 -1.5 1.5 3.0 2.72 2.44 1.89 1.33

mm; see [15]), C3 would produce results most consistent with the height model, C2 most inconsistent, and C1 in the middle. We also expect that the topography compensation algorithm would render the height difference between P1 and P2 to be modeled, i.e., Δpz = Δh regardless the stiffness differences. Throughout the experiment, haptic rendering was updated at 1 kHz. Procedure: A one-interval two-alternative forced-choice paradigm [18] was used to measure the subjects’ sensitivity to height difference between the two planes independent of their response biases. On each trial, P1 was randomly presented on the left (with P2 on the right) or on the right (with P2 on the left) with equal probabilities. We denote these two stimulus alternatives with P1 → P2 and P2 → P1 , respectively. The subjects’ tasks were to stroke the virtual surface from left to right once and to report whether the plane on the right was perceived to be higher or lower than the one on the left using a keyboard (see Fig. 3). Each subject conducted 60 trials in each experimental condition. The half of the 60 trials was rendered with the topography compensation algorithm, and the other half was with the penalty-based method. The order of trials within each experimental condition was randomized. The order of the three experimental conditions was also randomized for each subject. Once data collection began, no correct-answer feedback was provided to the subject. In each trial, the subjects were instructed to move the probe tip toward the left block until the color of the block turned from green to red, indicating the beginning of a trial (see Fig. 3(b) again). Once the probe tip entered a ±2 mm band along the y-axis (centered on the line connecting the centers of the two blocks), its motion was constrained to the zx (horizontal) plane. The subjects then stroked the virtual surface from left to right until the probe tip intersected the right block. The color of the right block turned from green to red to indicate the end of the current trial. The two planes were then turned to have the same height until a response was entered. It follows that one trial consisted of one sweep across the vertical planes. The probe-tip position and the force computed from a rendering algorithm were also recorded on a computer file in every trial. The whole experiment took about 20 minutes for each subject excluding training. Before the subjects started the experiment, they had training sessions each of which was composed of 24 trials. The trials in each session differed in experimental condition and rendering algorithm. Most subjects became familiar with the task with only one training session. Just a few subjects repeated the training session up to two times upon their requests.

400


Data Analysis: We computed sensitivity index d from the confusion matrix of each subject and each experimental condition. Sensitivity index is a measure for discriminability and free of response bias [18]. In our setting, a positive d indicates that a subject judged P2 to be higher than P1 , a d close to zero that the subject could not discriminate the relative height difference between P1 and P2 , and a negative d that the subject judged P2 to be lower than P1 . We also computed the average penetration force of each subject from the recorded data. From the recorded data of a subject in each trial, rendered forces were averaged over the ±100 ms interval whose center corresponded to the middle point of the haptic edge. These mean forces were averaged again for the average penetration force of the subject. 4.2

Results

The results of the experiment are summarized by bar graphs shown in Fig. 4. Figs. 4(a) – 4(c) represent the sensitivity indices of each subject under conditions C1 – C3, respectively. The d values averaged across the subjects are plotted in Fig. 4(d). The average d under each condition was computed from a confusion matrix that pooled the confusion matrices of all subjects under the corresponding condition. In each figure, the left group represents the results of the topography compensation algorithm and the right group those of the penalty-based algorithm. Note that d values that reached ∞ are truncated to 5 and those reached −∞ to −5 for visibility. It is apparent from Fig. 4 that our topography compensation algorithm produced large positive d values under all experimental conditions and for all subjects. This indicates that all subjects perceived P2 to be higher than P1 as regardless stiffness differences between P1 and P2 , which clearly agrees to the height model used in the experiment (Δhz = 3 mm). However, the penalty-based algorithm resulted in unreliable perception of height differences. Under C1 and C2, the d values of the penalty-based algorithms are mostly negative or close to zero. This means that P2 felt lower than, or equally high to P1 , respectively, which is clearly inconsistent with the height model. Under C3, the d values of the penalty-based method are all positive. Note that this was expected in our experiment design (see Δpz in Table 1). However, the d value of each subject under C3 is mostly smaller than or close to the corresponding d value of the topography compensation algorithm. This implies that our algorithm was more robust in delivering the height of a haptic edge. All of these results are consistent with our expectations discussed earlier. The variability of individual discriminability seems to be highly correlated with the average penetration forces of the subjects. Table 2 lists the measured average penetration force of each subject and the corresponding expected perceived height differences under the three experimental conditions computed from the force constancy model when penalty-based rendering is employed. Considering that the detection threshold of surface height change is about 0.66 mm [15], the pairs of subject and experimental condition exhibiting |Δpz | less than 0.66 mm in Table 2 were expected to result in d close to 0. The values of Δpz larger


5

5 S1 S2 S3 S4 S5 S6 S7

4 3 2 1

d’

3 2 1

d’

0

−1

−1

−2

−2

−3

−3

−4

−4 Topography Compensation

S1 S2 S3 S4 S5 S6 S7

4

0

−5

−5

Penalty−based

Topography Compensation

Applied Algorithm

(b) Condition C2.

5

5

4

4

3

3

2

2

1

C1 C2 C3

1

d’

0 −1

−3 −4 Topography Compensation

0 −1

S1 S2 S3 S4 S5 S6 S7

−2

−5

Penalty−based

Applied Algorithm

(a) Condition C1.

d’

401

−2 −3 −4 −5

Penalty−based

Topography Compensation

Applied Algorithm

Penalty−based

Applied Algorithm

(c) Condition C3.

(d) Averages across all subjects.

Fig. 4. Sensitivity indices d measured in the psychophysical experiment Table 2. Average penetration forces measured in the experiment and corresponding Δpz when the penalty-based rendering is used Subject fp (N) C1 Δpz C2 (mm) C3

S1 1.41 0.65 0.13 2.22

S2 1.43 0.62 -0.18 2.21

S3 1.53 0.45 -0.40 2.15

S4 2.57 -1.28 -2.71 1.57

S5 2.60 -1.33 -2.78 1.56

S6 2.69 -1.48 -2.98 1.51

S7 2.72 -1.53 -3.04 1.49

than 0.66 mm would make d larger than 1, and those less than -0.66 mm would d smaller than −1. These predictions can be well confirmed by comparing the d values shown in Fig. 4 and the Δpz values in Table 2, except for the d of S6 under C1. We also carefully examined the trajectories of the PHANToM stylus tip recorded during the experiment. Overall, the measured trajectories when the topography compensation algorithm was used were parallel to the surface height model around the haptic edge. However, those of the penalty-based algorithm were often severely distorted from the height model. Four examples of the measured probe-tip trajectories are provided in Fig. 5. Figs. 5(a) and 5(b) show the

402


6 5 4 3 2 1 0 0

Trajectory of Probe Tip

4000

20 15 10 5 0 −5 −10 −15 −20 0

500

1000

1500

2000 2500 Time (ms) Force

3000

3500

4000

4000

6 5 4 3 2 1 0 0

500

1000

1500

2000 2500 Time (ms)

3000

3500

4000

500

1000

1500

Height (mm)

Surface Height Adjusted Height Trajectory of Probe Tip


3000

3500

Force (N)

Force (N)

Height (mm)

Trajectory of Probe Tip 20 15 10 5 0 −5 −10 −15 −20 0

500

1000

1500

2000 2500 Time (ms)

3000

3500

Surface Height Trajectory of Probe Tip

(a) P 1 → P 2 with the topography com- (b) P 1 → P 2 with the penalty-based algopensation algorithm. rithm.

6 5 4 3 2 1 0 0

Trajectory of Probe Tip

1000


4000

500

1000

1500


3000

3500

4000

5000

6 5 4 3 2 1 0 0

500

1000

1500

2000 2500 Time (ms)

3000

3500

4000

Height (mm)

5000

20 15 10 5 0 −5 −10 −15 −20 0

Surface Height Adjusted Height Trajectory of Probe Tip

Force (N)

Force (N)

Height (mm)

Trajectory of Probe Tip 20 15 10 5 0 −5 −10 −15 −20 0

1000

2000 3000 Time (ms)

4000

Surface Height Trajectory of Probe Tip

(c) P 2 → P 1 with the topography com- (d) P 2 → P 1 with the penalty-based algopensation algorithm. rithm. Fig. 5. Probe tip trajectories delivered to the hand

data measured in trials where S5 stroked from left to right the surface in the P1 → P2 configuration under C2, with and without the compensation algorithm, respectively. In each figure, the trajectory of the probe-tip is represented by a blue solid line in the upper panel along with the surface height model shown in a red dashed-dotted line. Fig. 5(a) also contains the adjusted height model (hc (n)) drawn with a green dashed line. The forces rendered by the corresponding algorithms are shown in the lower panels. One can observe from Fig. 5(a) that the topography compensation algorithm rendered the stylus trajectory to be parallel to the surface model around the haptic edge and thus perceptualized the edge information correctly. However, the penalty-based algorithm in Fig. 5(b) exhibits a probe-tip trajectory inverted from its model around the haptic edge. The same observations can be made in Figs. 5(c) and 5(d) that show the measured data of S5 stroking a surface with P2 on the left and P1 on the right. The height differences measured in Figs. 5(a) and 5(c) were approximately 2.6 mm and 3.3 mm, respectively. These values were close to the modeled height difference (= 3 mm) and thus well demonstrate the robust performance of the compensation algorithm.


5

403

General Discussions

In this paper, we have presented a haptic rendering method that correctly delivers the height information of a haptic edge despite its encompassing stiffness change. The algorithm estimates user-applied force based on the human motor characteristics and uses the information in adaptively adjusting surface height. The experimental results have manifested the robust performance of our topography compensation algorithm. To the best of our knowledge, this is among the first attempts to incorporate the effect of human exploratory patterns into computational techniques for haptic rendering, thereby accomplishing more precise perception. We believe that such approaches would lead computer haptics to advance one step further considering the bidirectional nature of haptic interaction. This work has been our first step towards perceptually correct haptic rendering of collocated surface topography and stiffness. We clarify again that our criterion of “correctness” is whether rendered attributes agree to their models, which is a fundamental requirement in data perceptualization. Investigating what happens on a real object surface with varying stiffness is another research direction that we are pursuing in parallel. Our future work can be laid out as follows. First, in our experiences, changing force direction according to the geometry of a haptic edge (as opposed to the fixed force direction used in the present study) seems to be beneficial for edge detection. In this case, when a probe tip hits a haptic edge, tactile cues can occur due to the resulting force direction change. These initial tactile cues may help the perception of the haptic edge, in addition to the kinesthetic sensory cues resulted from a position change. We plan to quantify performance enhancement of varying force directions in the near future. Second, we plan to elaborate the way of estimating user-applied force. Inside a haptic edge, it was observed that the force generated by the PHANToM was no longer in equilibrium with the user-applied force (see the bottom panels of Fig. 5). It seems that the user cannot maintain constant contact force when the force exerted by the haptic interface changes abruptly. This would result in errors in the user-applied force estimated using the current algorithm. Although this error was not critical in the present psychophysical experiment (recall that the modeled edge height was 3 mm, which is much higher than the edge detection threshold of 0.66 mm), it may affect the performance of our compensation algorithm for height differences close to the detection threshold. Finally, it was found that surface heights perceived with different exploratory procedures1 may not be consistent. For example, the perceived height of plane P2 in Fig. 5(a) by stroking would be one resulted from the adjusted height represented by the green dashed line. However, tapping on P2 without touching P1 before would make the current algorithm position the surface at its original height represented by the red dash-dotted line. 1

Exploratory procedure refers to a stereotypical movement pattern that we employ to perceive a certain object attribute by touch [19]. For instances, tapping is an optimal exploratory procedure for stiffness perception, and stroking is for topography change and texture perception.

404


This inconsistency is because our compensation algorithm was designed solely for the exploratory procedure of stroking. Extending the algorithm to be able to adequately handle all exploratory procedures would be quite a challenging yet interesting assignment for perceptualizing the data of collocated surface topography and stiffness.

References 1. Lawrence, D.A.: Stability and transparency in bilateral teleoperation. IEEE Transactions on Robotics and Automation 9(5) (1993) 624–637 2. Colgate, J.E., Schenkel, G.G.: Passivity of a class of sampled-data systems: Application to haptic interfaces. Journal of Robotic Systems 14(1) (1997) 37–47 3. Choi, S., Tan, H.Z.: Perceived instability of virtual haptic texture. I. Experimental studies. Presence: Teleoperators and Virtual Environment 13(4) (2004) 395 – 415 4. Choi, S., Tan, H.Z.: Perceived instability of haptic virtual texture. II. Effects of collision detection algorithm. Presence: Teleoperators and Virtual Environments 13(4) (2005) 463–481 5. Otaduy, M.A., Lin, M.C.: Introduction to haptic rendering. In: ACM SIGGRAPH Course Notes on Recent Advances in Haptic Rendering & Applications. (2005) A3–A33 6. Brooks, T.L.: Telerobotic response requirements. In: Proceedings of IEEE Conference on Systems, Man and Cybernetics. (1990) 113–120 7. Taylor II, R.M., Robinett, W., Chi, V.L., Brooks, Jr., F.P., Wright, W.V., Williams, R.S., Snyder, E.J.: The nanomanipulator: A virtual-reality interface for a scanning tunneling microscope. In: Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques. (1993) 127–134 8. Iwata, H., Noma, H.: Volume haptization. In: Proceedings of the IEEE Virtual Reality Annual International Symposium. (1993) 16–23 9. Avila, R.S., Sobierajski, L.M.: A haptic interaction method for volume visualization. In: Proceedings of the ACM Conference on Visualization. (1996) 197–204 10. Lundin, K., Ynnerman, A., Gudmundsson, B.: Proxy-based haptic feedback from volumetric density data. In: Proceedings of Eurohaptics. (2002) 104–109 11. Ikits, M., Brederson, J.D., Hansen, C.D., Johnson, C.R.: A contraint-based technique for haptic volume exploration. In: Proceedings of the IEEE Visualization Conference. (2003) 263–269 12. Maciejewski, R., Choi, S., Ebert, D.S., Tan, H.Z.: Multi-modal perceptualization of volumetric data and its applicatiton to molecular docking. In: Proceedings of the World Haptics Conference. (2005) 511–514 13. Yano, H., Nudejima, M., Iwata, H.: Development of haptic rendering methods of rigidity distribution for tool-handling type haptic interface. In: Proceedings of the World Haptics Conference. (2005) 590–570 14. Sarid, D.: Scanning Force Microscopy. Oxford Unversity Press, New York (1991) 15. Choi, S., Walker, L., Tan, H.Z., Crittenden, S., Reifenberger, R.: Force constancy and its effect on haptic perception of virtual surfaces. ACM Transactions on Applied Perception 2(2) (2005) 16. Zilles, C.B., Salisbury, J.K.: A constraint-based god-object method for haptic display. In: Proceedings of IEEE International Conference on Intelligent Robots and Systems. (1995) 146–151


405

17. Ruspini, D., Kolarov, K., Khatib, O.: The haptic display of complex graphical environments. In: Computer Graphics Proceedings (ACM SIGGRAPH Proceedings). (1997) 345–352 18. Macmillan, N.A., Creelman, C.D.: Detection Theory: A User’s Guide. Second edn. Lawrence Erlbaum Associates (1994) 19. Lederman, S.J., Klatzky, R.L.: Hand movement: A window into haptic object recognition. Cognitive Psychology 19 (1987) 342–368

Multi-sensory Perception of Roughness: Empirical Study on Effects of Vibrotactile Feedback and Auditory Feedback in Texture Perception Ki-Uk Kyung1 and Dong-Soo Kwon2 1

POST-PC Research Group, Digital Home Division, ETRI 161 Gajeong-dong, Yuseong-gu, Daejeon, 305-700, Korea [email protected] 2 Human Robot Interaction Research Center, KAIST 373-1 Guseong-dong, Yuseong-gu, Daejeon, 305-701, Korea [email protected]

Abstract. This work investigates effects of vibrotactile stimuli and auditory feedback in perceiving roughness. One of objectives in the research is to derive functional correlations between perceived roughness and principal elements of vibrotactile stimuli. A tactile display device having a 5x6 pin array has been developed as the test bed for the quantitative experiment. Sandpapers were used as specimens of perceived roughness. The experimental results show that there is a tight correlation between perceived roughness and vibrotactile stimuli and their functional relations have been derived. The second objective of this research is to analyze the relation between perceived roughness and auditory feedback. Real sound samples recorded during rubbing surface of sandpaper specimens have been adopted as auditory feedback elements. The experimental results showed human could estimate surface roughness from auditory information if and only if he/she recognized the source of sounds. The results in this papers show possibilities to fortify representing roughness by using a tactileauditory display in a tangible environment beyond conventional approach changing sizes of grits or grooves on a surface.

1 Introduction In efforts to enhance the efficiency of task performance under a virtual environment or a remotely controlled workspace, numerous studies have been conducted to develop meaning of sharing feelings and creating tactile feelings. However, the existing developed apparatuses only utilize simple contact or the intensity of stimuli. Haptic display in an interactive dynamic environment or texture display still remains challengeable problems[22][9]. Particularly, since perceiving texture is a complex fusion process of multimodal inputs such as vision, sound as well as tactile sensation, the effects of multimodality also has to be investigated. Since Weber started investigating human tactile sensitivity[20], many researchers have studied the relationship between perceived texture and the physical quantities of stimuli. Katz researched tactile performance quantitatively[7], and the results showed Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 406 – 415, 2006. © Springer-Verlag Berlin Heidelberg 2006

Multi-sensory Perception of Roughness

407

that vibrational stimuli are necessary in the perception of surface texture. Lederman and Taylor studied measurements of perceived roughness on various grooved surfaces[13]. Lederman et al. also investigated the role of vibration in tactual perception, [12]. Hollins et al. investigated how, on the basis of texture, tactile surfaces are connected with the perceptual space[6], finding that roughness is one of the essential sensations of touch. Recently, there have been several investigations that sought a relationship between vibrotaction and perceived roughness based on the duplex theory[4][5][6]. More recently, based on neuroscientific approaches, Cascio et al. and Yoshika et al. confirmed that temporal cues contribute to the tactile perception of roughness [1][21], while Smith et al. showed friction and tangential force affect tactile roughness[17]. In addition, recent studies have reported that human tactile sensitivity is a function of frequency of vibration [10]. There has been a research multimodal perception of virtual roughness. McGee et al conducted an experiment to investigate the perceived roughness of audio textures with the hypothesis that the frequency of the audio texture (or number of notes) will affect the perceived roughness of the virtual texture[16][20]. To keep pace with the demands of related research, this paper assesses how the frequency and amplitude changes of vibrational stimuli affect texture perception when using a tactile display system and how the auditory feedback affect roughness perception when using recorded sounds of rubbing real surface. In section 2, a pin-array based vibrotactile display for quantitative stimuli is described. In section 3, the question of how much roughness, an essential element composing texture, can be represented using a tactile display is assessed experimentally. Therefore, the objective of the section is to derive the functional correlation between dependant variables and the perceived roughness of an arbitrary surface. The dependant variables are the amount of longitudinal movement of pins within an array, that is, the amplitude of vibrotactile stimuli, and the velocity of the pins’ up and down movement, that is, the frequency of the tactile display. In section 4, effect of auditory feedback on tactile sensation is described. In this section, we adopt recorded real audio textures instead of using composed digital sounds. The participants’ responses to the recorded sound have also been analyzed.

2 Texture Display System 2.1 Planar Distributed Tactile Display A tactile display which has 5x6 tactors array has been developed as a test-bed for psychophysical experiment[10]. The maximum deflection is greater than 700μm and the bandwidth is about 350Hz. The blocking force of each pin is 0.06N. The specifications of the tactile stimulator with piezoelectric bimorphs were verified to ensure that it deforms the user’s skin within 32dBSL (sensation level in decibels above threshold) [3][10][18]. The contact area is 9.7mm×7.9mm- a previous study showed this area is sufficient to discern difference in textures[9]. Since the tactile display unit is small enough to be embedded into a computer mouse, we developed a mouse-type texture display system[11]. The response time of each actuator is a scale of submilliseconds. This texture display system has been adopted as a test-bed in this study.

408

K.-U. Kyung and D.-S. Kwon

2.2 Performance Evaluation Static Pattern Display. In order to use the proposed system as a computer interface, the system should provide some kinds of symbols, icons, or texts in a haptic manner. Therefore the performance of the tactile display was evaluated by asking subjects to discriminate between plain and textured polygons, round figures, and gratings. In these experiments, the actuator voltages were adjusted to set the desired shape, which was then held constant. Subjects were allowed to actively stroke the tactile array with their finger pad. The proportion of correct answers (90-99%, depending on the stimulus) far exceeded chance (10%), indicating that the display provides a satisfactory representation of tactile pattern[10]. Vibrotactile Pattern Display. We investigated how vibrotaction affects the identification of forms in which only passive touch, and not rubbing, is used. Craig has already compared the sensitivity of the scanned mode and static mode in discerning tactile patterns[2], but here we compare the sensitivity of the static mode and synchronized vibrating mode. In order to set the other conditions identical to those in the experiment of static pattern display, except for the vibrotaction, the same texture groups used in static pattern display were deployed with three different low frequencies: static, 1Hz, and 3Hz[10]. The frequencies were selected based on identical sensation levels, since the magnitudes of the threshold value in the frequency band of 0~5Hz are almost the same[19]. The experimental showed that the proportion of correct answers generally increases as the frequency rises from static to 1 Hz to 3Hz. From a psychophysical and physiological point of view, it seems likely that a 3Hz vibration can effectively stimulate the Merkel cells and that the associated SA I afferent provides the fine spatial resolution necessary for the subject to make the required discriminations. And we increased vibrational frequency of the patterns up to 250Hz and observed the change of percentage of correct pattern discrimination[10]. This study reported there are several frequency bands that human can easily discern surface’s patterns. From these results, we expect that the texture display, which has been a test-bed, is capable of providing tactile stimuli quantitatively while the user simply grasps and translates the mouse while exploring the virtual environment.

3 Display of Roughness Using Vibrotactile Stimuli In this section, the question of to what degree roughness, one of the most essential elements composing texture, can be represented using a tactile display is assessed experimentally. Hollins’s research verified that vibrotaction could be the necessary and sufficient condition for roughness perception [4]. The objective of this section is to derive a certain functional correlation between vibrotaction and the perceived roughness. 3.1 Subject Training In this study, the subjects have been trained using pieces of sandpaper before participating in the main experiment. We selected sandpaper grits to be linearized when it is


409

log-scaled. Sandpapers of seven different grits were selected: 24, 32, 60, 100, 220, 600, and 2000. The size of the sandpapers roughly corresponded with that of the contact area of the tactile display. Subjects were asked to respond by giving sensitive strength values of roughness for every specimen on a seven-point scale. Table1 shows averages and standard deviations of the results with changes of the grit sizes during active and passive touches, respectively. Table 1. Roughness Values of Sandpaper Active Touch

Grit of Sandpapers

Passive Touch

Ave.

Stdev.

Ave.

Stdev.

24

6.7

0.48

5.4

1.63

36

6.6

0.73

5.2

2.07

60

6.4

0.82

4.8

0.75

100

6.1

0.91

3.5

1.21

220

5.5

1.22

2.9

1.33

600

3.0

1.91

1.4

0.84

2000

1.4

0.45

1.2

0.41

3.2 Experimental Method In this experiment, all pins of the tactile display move simultaneously, and they gives cutaneous stimuli while a user touches the pin array using his/her finger. In order to observe the effects of both the frequency and amplitude of stimuli, the stimuli are composed of the set shown in Table 2. The stimuli are determined to cover the range of perceivable vibrational frequency, which is 0-250Hz, and the range of amplitude satisfies the antinociception area considering pain threshold. A stimulus among the 34 stimuli determined in this section was generated for 8 seconds, and the subject had to answer within 11 seconds, including the stimulation time. The answer was in the form of the sandpaper number with the perceived roughness most similar to that transmitted by the tactile display. In this case, sandpapers should be chosen among the sandpapers that could be discerned through active touch. Since the noisy sound of the tactile display could affect the experimental results, the subject could hear only pink noise via headphones during operation of the pin array. All stimuli were repeated 4 times, and their order was randomized. The entire procedure was repeated under the condition of passive touch. Table 2. Combinations of Frequency and Amplitude

400 μm 200 μm 100 μm 50 μm 20 μm

0Hz

2 Hz

5 Hz

10 Hz

25 Hz

50 Hz

O O O O O

O O O O O

O O O O O

O O O O O

O O O O O

X O O O O

100 Hz

X X O O O

250Hz

X X X O O

410


3.3 Experimental Results and Analysis The experimental results of section 3.2 are analyzed statistically. The objective of this analysis is to find correlation coefficients and functional relations between perceived roughness and the principle variables of vibrotactile stimuli, frequency and amplitude. Figure 1 shows the perceived roughness as a function of log-scaled frequency. All Subjects - Active Touch

400um 200um 100um 50um 20um

6

6 5 4

400um 200um 100um 50um 20um

3 2 1

All Subjects - Passive Touch

7

1

10

100

1000

Perceived Roughness

Perceived Roughness

7

5 4 3 2 1

1

Frequency Hz (log scale)

(a) Perceiver roughness for active touch

10

100

1000

Frequency Hz (log scale)

(b) Perceiver roughness for passive touch

Fig. 1. Perceived roughness as a function of frequency

From the observation of Figure 1, it appears that perceived roughness has certain relations with the following five variables: frequency, logarithm of frequency, amplitude, logarithm of amplitude, and multiplication of logarithm of frequency and logarithm of amplitude. From the linear regression of the experimental data, the following functional relation could be derived. In the case of equation (3.1), k1=7.79, k2=3.79, and k3=-0.87 for active touch, and k1=6.68, k2=2.93, and k3=-0.47 for passive touch. The unit of frequency is hertz and that of amplitude is millimeters. R = k1 + k2·log(Amp) + k3·log(Freq)

(3.1)

4 Effect of Auditory Feedback on Tactile Roughness Perception Tactile perception process is not only a somatosensory response, but also a combination of multimodal sensory fusion such as visual, auditory, and haptic feedback. Lederman et al experimentally assessed the relative contributions of tactual and auditory information to multisensory judgments of surface roughness using a rigid probe, and their results showed auditory feedback contributed to roughness perception[14]. McGee et al conducted an experiment to investigate the perceived roughness of audio textures with the hypothesis that the frequency of the audio texture (or number of notes) will affect the perceived roughness of the virtual texture[15][16]. In this paper, we adopt recorded real audio textures, which occurred between real finger pad and surface, instead of using digitally sound generation methods such as MIDI, PCM etc. The participants’ responses to the recorded sound have also been analyzed.


411

4.1 Stimuli and Apparatus Sandpaper samples referred to in the previous section have been adopted for the experiment in this section. A trained subject who could maintain gentle pressure and a lateral motion speed of 100-120mm/sec while rubbing the sample surface participated in sound recording. A microphone was located 5 cm away from the sandpaper, and the sound generated by the participant rubbing the sandpaper was transmitted to Microsoft Window’s Recorder. Sandpapers are generally sorted by grit number. 7 sandpaper samples having grit numbers of 24, 32, 60, 220, 600, and 2000 were utilized, and the rubbing sound was recorded for every sandpaper sample. Sandpaper sample 1 has a grit number of 24, constituting the roughest surface. As the sandpaper code number increases, the number of grit increases and the roughness decreases. For instance, sample 3 has a grit number of 60. Figure 2 shows the frequency response of recorded audio texture in the band of audio frequency (20-20000 Hz). The back ground and system noise are uniformly distributed in overall frequencies, except the noise has high intensity at a frequency band of 20-40Hz. Compared to the background and system noise, the frequency response of audio texture for sandpaper sample 1 shows very high intensities in the lower frequency band (30-200Hz). In addition, the audio texture for sandpaper sample 1 shows higher intensities in the overall frequency range compared to background noise. The frequency response of audio texture for sandpaper sample 2 shows a similar response to that of sample 1. However, the intensities in the lower frequencies are lower than those of sample 1. For sample 3, the intensities of low frequency sound are lower than previous samples, but the response of sample 3 shows higher intensities at a frequency band of 200-400Hz. For sample 4, intensities of low frequency band (40-80Hz) are higher than the system noise, and the response of sample 4 shows higher intensities at a frequency band of 500-800Hz. For sample 5, intensities of the low frequency band (40-80Hz) are higher than the system noise, and the response of sample 5 shows higher intensities at a frequency band of 500-800Hz. For sample 6, intensities of the low frequency band (40-80Hz) are higher than the system noise. For sample 7, intensities of the low frequency band (40-80Hz) are higher than the system noise, and intensities at higher frequencies are lower than those of sample 6. From an inspection of sounds recorded during rubbing of the sandpaper, it is found that low frequency is related to the physical roughness of the sandpapers. Since the sound during touching is caused by stick slip between the finger pad and the surface’s grit, and rough sand papers have bigger grits, we can speculate that low frequency sound could be generated when we larger grit sandpapers are rubbed with the same lateral speed. 4.2 Auditory Feedback and Perceived Roughness In this section, an experiment to measure the perceived surface roughness from auditory feedback is described. 5 subjects in their twenties participated in this experiment. Through headphones, the subjects heard one of 7 sound samples delineated above, and they were requested to assess how rough the surface would be if they imagined rubbing it. The playing time was 10 seconds followed by 5 seconds of silence. The subjects were not informed that the sound was recorded from rubbing a surface. They responded according to a seven-point bipolar scale. Each of the 7 sound samples was

412


(a) Background and system noise

(c) Audio texture for sandpaper sample 2

(b) Audio texture for sandpaper sample 1

(d) Audio texture for sandpaper sample 3

(e) Audio texture for sandpaper sample 4

(f) Audio texture for sandpaper sample 5

(g) Audio texture for sandpaper sample 6

(h) Audio texture for sandpaper sample 7

Fig. 2. Frequency analysis of recorded audio texture

suggested four times in random order. Table 3 shows the experimental results. The average score of recorded sound for rubbing sandpaper sample 1 shows the highest value, and the score decreases as the sandpaper’s roughness decreases. These results indicate that humans may discern surface roughness using only auditory information. Since experimental sound samples were not quantitatively determined, we could not derive functional relations between elements of sound and perceived roughness. However, we can conclude that experience and behavioral learning enable estimation of surface roughness from the rubbing sound.


413

Table 3. Combinations of Frequency and Amplitude Sound Sample 1 6.3 1.79

Average Stdev.

Sound Sample 2 5.4 1.91






4.2 Roughness Discrimination from Auditory Information

Percentage of Answers

In section 4.2, the experimental results show that the change of auditory feedback could cause the change of perceived roughness. In order to investigate whether the subjects discern surface roughness using only auditory feedback, an experiment has been conducted using real sandpaper samples and recorded sounds of rubbing these samples. For the experiment in this section, real sandpaper specimens having grit numbers of 24, 36, 60, 220, 600 and 2000 were presented to the subjects. The subjects heard one of the sound samples recorded in section 4.1, and they were guided to match the sound samples with the real specimen. Rubbing and observing real sandpapers were allowed in order to help the subjects estimate their roughness; however, the real sound generated by rubbing the sandpapers was not transmitted to the subjects; this was accomplished by having the subjects wear headphones capable of surround noise cancellation. Each of the 7 sound samples was suggested four times in random order. Figure 3 shows the experimental results in the form of a confusion matrix plot. For sound sample 1, all answers were given as sandpaper of grit number 24, constituting perfect matching. About 65% of the answers were sandpaper of grit number 32 for sound sample 2, and about 30% of the answers were given to neighboring sandpapers. For sample 3 and sample 4, the percentage of correct matches was lower than 50%; however, in the cases of these samples, other answers were concentrated on the neighboring sandpaper samples. Although the total percentage of correct matches is 63.8%, most wrong matches are concentrated on the correct sandpaper’s neighboring samples. From these experimental results, we can conclude that auditory feedback alone enables accurate estimation of surface roughness if the subject knows the source of sound.

Fig. 3. Confusion rate for sandpaper discrimination

5 Conclusion and Discussion The first experimental results in section 3 show that there is a tight correlation between perceived roughness and vibrotactile stimuli and their functional relations have

414


been derived. The results of the analyses show that perceived roughness is proportionally intensified as logarithms of frequency or logarithms of amplitude increase and that amplitude and frequency could complement each other. This research suggests a methodology for roughness representation of an arbitrary surface using a pinarrayed vibrotactile display. In addition, from the experiments described in sections 4.1, 4.2 and 4.3, the following observations could be derived. (1) In the case of sandpapers, intensities of sound in low frequencies are related to physical roughness. (2) The subjects could estimate roughness from the rubbing sound information. (3) If the subject is aware of the source of sound, he/she can discern surface roughness among the various roughness samples. This result means recorded sound can fortify roughness representation in tangible environment. In addition, if the sound samples could be recorded under stricter conditions and analyzed quantitatively, meaningful relations between elements of sound and perceived roughness could be derived. In addition, multimodality combining auditory feedback and tactile feedback will increase human sensitivity of texture perception in a virtual environment.

Acknowledgment This research has been supported by the Ministry of Information and Communication, Republic of Korea (2006-S-031-01).

References 1. Cascio, C.J. and Sathian, K., Temporal Cues Contributes to Tactile perception of Roughness, The Journal of Neuroscience, Vol. 21(14). (2001) 5289-5296. 2. Craig, J.C., Identification of scanned and static tactile patterns, Perception & Psychophysics, Vol.64(1). (2002) 107-120 3. Diller,T., Schloerb, D. and Srinivasan, M.A., Frequency response of Human Skin in vivo to mechanical stimulation, RLE Technical Report No. 648. MIT. (2001). 4. Hollins, M., Bensmaia, S.J., and Roy, E.A., Vibrotaction and texture perception, Behavioral Brain Research, Vlo.135. (2002) 51-56. 5. Hollins, M., Faldowski, R., Rao, S., and Young, F., Perceptual dimensions of tactile surface texture: a multidimensional scaling analysis, Perception and Psychophysics, Vol. 54. (1993) 697-705 6. Hollins, M. and Risner, S.R., Evidence for the duplex theory of tactile texture perception, Perception & Psychophysics, Vol.62(4). (2000) 695-705 7. Katz, D., The World of Touch, In: Krueger, L.E.(Trans.), Hillsdale, NJ, Erlbaum (1989) 8. Kyung, K.U., Choi, H., Kwon, D.S. and Son, S.W., Interactive Mouse Systems Providing Haptic Feedback During the Exploration in Virtual Environment, Lecture Notes in Computer Science, Vol. 3280. (2004) 136-146. 9. Kyung, K.U, Son, S.W., Yang, G.H. and Kwon, D.S., Design of an Integrated Tactile Display System, Proc. of the IEEE ICRA 2004 (2004) 776-781 10. Kyung, K.U., Ahn, M., Kwon, D.S. and Srinivasan, M.A., A compact planar distributed tactile display and effects of frequency on texture judgment, Advanced Robotics, Vol. 20(5). (2006). 563–580


415

11. Kyung, K.U., Kwon, D.S. and Yang, G.H., A Novel Interactive Mouse System for Holistic Haptic Display in a Human-Computer Interface, International Journal of Human Computer Interaction, Vol. 20(3). (2006) 247–270 12. Lederman, S.J., Loomis, J.M. and Williams, D.A., The Role of Vibration in the Tactual Perception of Roughness, Perception & Psychophysics, Vol.30. (1982). 82-89. 13. Lederman, S.J. and Taylor, M.M., Fingertip force, surface geometry, and the perception of roughness by active touch, Perception & Psychophysics, Vol.12(5). (1972).401-408 14. Lederman, S.J, Klatzky, R.L., Morgan, T. and Hamilton, C., Integrating Multimodal Information about Surface Texture via a Probe: Relative Contributions of Haptic and TouchProduced Sound Sources, Proceedings of HAPTICS’02, IEEE. (2002) 97-104 15. McGee, M.R., Gray, P. and Brewster. S., Feeling Rough: Multimodal Perception of Virtual Roughness, Proceedings of Eurohaptics 2001(2001) 16. McGee, M.R., Gray, P. and Brewster. S., Mixed Feelings: Multimodal Perception of Virtual Roughness, Proceedings of Eurohaptics 2002, (2002) 17. Smith, A.M. and Chapman, C.E., Role of friction and tangential force variation in the subjective scaling of tactile roughness, Experimental Brain Research, 144. (2002).211-223 18. Srinivasan, M.A. Surface deflection of primate fingertip under line load, Journal of Biomechanics, Vol.22(4). (1989) 343-349 19. Verrillo, R.T., Fraoli, A.J. and Smith, R.L., Sensation magnitude of vibrotactile stimuli, Perception and Psychophysics, Vol. 7. (1969) 366-372 20. Weber, E.H., E.H. Weber on the Tactile Sense(2nd Edition), In: Ross, H.E. and Murray D.J.,(Edit. and Trans.), Erlbaum(UK) Taylor & Francis (1996) 21. Yoshioka, T., Gibb, B., Dorsch, A.K., Hsiao, S.S. and Johnson, K.O., Neural coding mechanisms underlying perceived roughness of finely textured surface, The Journal of Neuroscience, Vol. 21(17). (2001) 6905-6916 22. Son, W., Kim, K., Jang, B. and Choi, B., Interactive Dynamic Simulation Schemes for Articulated Bodies through Haptic Interface, ETRI Journal, Vol. 25(1). (2003) 25-33

The Research of Spiritual Consolation Object with the Function of Affective Computing Tang Yiping, Sun Hongjie, Gu Xiaokai, and Deng Tao Zhejiang University of technology [email protected], [email protected], [email protected]

Abstract. China is running to the aging society. In China, the number of the elder's families of empty nest presents an ascendant trend. It has caused various kinds of demands of the service for spiritual consolation. This article introduces how to distinguish different emotion form the sound sample by affective computing and brings forward the concept of spiritual consolation object. Elder can communicate with this object when they need spiritual consolation. With this, we can set up an interchange environment considering people first, which is intelligent and understandable between children and the elder. Keywords: Gerontechnology， Affective Computing， Spiritual Consolation Object， Speech Signal Processing.

1 Introduction China is running to the aging society. The national elder's families of empty nest accounts for 36% of all the families in our country. The experts predict that in the following ten years, the elder's family of empty nest will become the main mode when the elder retire to enjoy the life. Its quantity will be more than 80%. The constant increase of the elder's family of empty nest causes many social problems, such as the elder's healthy guarantee, the personal security, life caring, the spiritual consolation, economy supporting, etc. In the elder's demands of service for life, the spiritual consolation ranks the first. Now the nations of the world treat gerontechnology as the focus of the study[1]. It’s better to develop a system, from which the elder can get spiritual consolation instead of their children when the elder feel lonely. The elder can communicate spiritual consolation with the agent first. At the same time the agent can notify their children in the distant place in time. The system can set up an exchange environment, which is people-first, high efficient, automatic, intelligent, fully considering the psychology of the elder.

2 Background At present, children giving spiritual consolation can only rely on teeter paper, telephone, calling at their parents. Today, the young people face enormous existence Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 416 – 424, 2006. © Springer-Verlag Berlin Heidelberg 2006

The Research of Spiritual Consolation Object with the Function of Affective Computing

417

pressure and challenge both in life and work. They have not enough time to accompany the elder. This makes the elder have a mental depression. Therefore, there have such problems as the dissymmetry of the opportunity of the communication between solitary elder and their children. So we develop a spiritual consolation device. The sensors in the device get the elder’s emotion parameter, calculate the emotion through these parameters, and then send the result to their children through the wireless network. Children can get the affective state of their parents in time and do something useful to make the elder feel happy. Affective computing is a computing that relates to emotion, arises from emotion, and deliberately influences emotion. It is a new kind of man-machine exchange. Affective computing mainly includes the expression, the language, the touch and so on. Language, the tool that transmits information and express emotion, is very important to affective computing. So the calculation of the speech emotion is a main content of this research. The research of the speech emotion begins in 1990s. In American, Japan, Britain and other countries, there are many universities and studios that research on affective computing. Speech affective computing has fetched lots of production, and there are many valuable theses. But it has not formed one extensive approved theory and research approach, and it is never used for spiritual consolation to the elder either.

3 Design of the Spiritual Consolation Object The spiritual consolation object includes the hardware system and the software system. 3.1 Design of the Hardware System The spiritual consolation object has the touch sensor, speech sensor, microprocessor, the part for sending and accepting information, the information memory, and the

Temperature and humidity

Touch sensor

The message receives’ device Memory

Microprocessor

Message sending and receiving unit

Fig. 1. The hardware structure

418

Y. Tang et al.

information input part, the pronunciation output part, etc. Spiritual consolation object is instead of the children whom the elder expects to communicate with. This research designs the spiritual consolation agent as their children’s portrait, as Fig. 1 shows. 3.2 Design of the Software System This article mainly researches the speech emotion of the elder. Pronunciation sensor is one of the most important instruments to get the elder’s emotion information. Microprocessor automatically saves information through interruptive style when the elder speak facing the spiritual consolation device. Emotion information will be read from the information memory; characteristic parameters of the speech signal will be calculated by using affective computing. The device can confirm the emotion classification at the time the elder speaks by pattern search, and then does accordingly spiritual consolation to the elder, as Fig. 2 shows. Keyboard

Microphone

Signal deal unit

Info edit unit

Affective computing

Take medicine unit

Emotion judge unit

Take temperature unit

Voice prompt unit

Loudspeak

Memory unit

Message sending and receiving unit

Fig. 2. The software structure

4 Affective Computing of Speech Emotion It is the key to the system about how to judge the elder’s emotion state exactly. The system first gathers the elder’s speech signal, secondly gets the characteristic parameter through pretreatment, then calculates the elder’s emotion state according to the data from the emotion speech database, and then does accordingly spiritual consolation to the elder. The processing is as Fig 3 shows.


Pretreatment

Speech signal input

Calculate the characteristic parameter

Emotion speech database

419

Pattern search

Result of recognition

Fig. 3. The process of emotion classifying

4.1 The Pretreatment of Speech Signal The pretreatment of speech signal is needed before the speech emotion calculating. The goal of endpoint detection is to confirm the endpoint from a speech signal. Effective endpoint detection can reduce the processing time but also can remove the influence of noise, thus makes the judgment result more accurate.

Fundamental frequency

Judge of surd and sonant

Smoothness

The range of frequency

Change rate of fundamental frequency

Pass rate

zero

Affective classify

Short energy

time

All the characteristic

Average amplitude

Change of amplitude

Autocorrelation function

Affective computing

Fig. 4. The speech signal before and after the endpoint detection

Speech is made up of surd and sonant. The experiment results show that sonant includes most of the speaker’s emotion characters. The energy of sonant concentrates below 3kHz, nevertheless the energy of surd majority appears in a higher frequency. Short time energy can be used to separate majority sonant from surd. In order to

420

Y. Tang et al.

prevent the loss of low energy of sonant, the short time pass zero rate is used as a supplement judgment. The experiment result shows that sonant has a more obvious regularity and much lower pass zero rate than surd. Therefore, by using the two standards to endpoint detection, we can examine the effective speech accurately, as the Fig 4 shows. 4.2 Calculate of the Characteristic Parameter The fact that the voice can express the emotion due to the voice includes the parameters that can materialize the emotional feature. It is significant to the recognition of speech emotion that gathers these parameters from the speech signal. Now how to gather these parameters from the speech signal is searched in lots of articles. The main methods include time structure, amplitude structure, fundamental frequency structure, and formant structure. The speed of speaking is deferent when the speaker is at deferent emotion states. For example, when he is angry his speed of speaking is faster than calmness. Therefore we can judge the speaker’s emotion degree by his speed of speaking. The amplitude is correlation with the emotion state also, when the speaker is happy, anger, and fear, the amplitude of his speech is much higher than that he is at the emotion of sad, further more the more discrepancy at amplitude the more discrepancy at emotion state. The important characteristic parameters, which influence the recognition of emotion, are listed as table 1[2]: Table 1. The important characteristic parameter

Characteristic parameter Rate Average fundamental frequency Range of fundamental frequency Intensity Change of fundamental frequency FI Average FI Range In the research we choose six parameters of the speech as the characteristic parameters of the emotion. They are average short time energy, average change rate of amplitude, average fundamental frequency, average change rate of fundamental frequency, average short time pass zero rate, the time ratio of surd and sonant. Fundamental frequency is the best one that materializes the emotional features among these parameters.


421

Fundamental frequency is most important parameter of the speech. Fundamental frequency inspection is a rather complicated problem. As the oscillation of vocal cords is not absolute periodically, it is difficult to determine some frames, which mix surd and sonant belong to periodic or nonperiodic. The method of fundamental frequency inspection may be divided into two parts.（ 1） Method in time domain， gets the fundamental frequency directly form the wave.（ 2） Method in frequency domain, first changes the speech signal to the frequency domain, then calculates its fundamental frequency. In this research we use the improvement autocorrelation function to estimate the fundamental frequency of the speech signal. The short-time autocorrelation function R (x) has a very big peak value in each integral multiple of the fundamental cycle. The principle of autocorrelation algorithm is to estimate the fundamental frequency by seeking a first biggest peak value position, according to short-time autocorrelation function.

Rn (k ) =

∞

∑ [ x(m)w(n − m)][ x(m + k ) w(n − m − k )]

(1)

m = −∞

The arithmetic uses y0， y1， y2， ……yn to denote a frame of speech signal. From the first sample point of each frame, it uses （ y0* y1） +（ y1* y2） +……+（ yn-1*yn) to denote the value of autocorrelation function when the k equal to zero, then add one to k, calculate until k equal to n-2, save all the values. According to characteristic of the wave, when the k moves to the integral multiple of the fundamental cycle, the value of autocorrelation function is the biggest. Therefore, if we find the max value of autocorrelation function means we find the fundamental cycle. According to the fundamental cycle we can calculate the fundamental frequency easily. In the research of this article, according to the frequency characteristic of speech and the range of human speech, K needn’t start from zero, the calculate length needn’t to be as long as one frame either. Because the experiment uses 10kHz sample frequency, and according to the characteristic that human’s speech frequency concentrate on the range of 50 to 500Hz, in the experiment the range of k is calculated form 28 to 200. The computation obtains the K value is not necessarily the fundamental frequency corresponding points, also has the possibility that it is the 2 frequency multiplication or several frequency multiplication point. In order to avoid these mistakes, we take the maximum value method as supplement. When the point moves to the cyclical integral multiple, the value of autocorrelation function is the maximum, although the true fundamental frequency corresponding value is not necessarily a max value, but definitely is a maximum value, therefore after discovering the max value, establishes a threshold value, then use the threshold value to find the maximum value, then pick up a optimal point as the fundamental frequency corresponding point.

422

Y. Tang et al.

4.3 Emotion Pattern Search In the experiments, we divide old person's emotion states as calmness, fear, happy, sad and anger. Respecting to 5 different emotion states, 100 samples are collected, which are used as the samples of emotion speech database. We calculate the characteristic parameters of these samples, and then use the method of weighting Euclidean distance to recognize the emotion states[3]. Under the different emotion states, the contribution of various characteristic parameters to the emotion states is different. Therefore we use the contribution analytic method to determine the weight value when we constructer the emotion template. It is defined as:

wj =

1 n cov(θˆ( y i , φˆ j ( x ij )) ∑ var φ ( y ) n i =1 i

t = 1, L , n

(2)

j = 1, L ,8

(3)

8

θ ( y i ) = ∑ φ j ( x ij ) + ε i j =1

Here, the distance y,

θ ( y i ) is

φ j ( xij )

(4)

the function of t-th emotion speech and template Euclidean

is conic function of

xij

, the J-th emotion characteristic parameter

φˆ ( x )

θˆ( y )

ε

i of t-th sentence, where i is random error, and j ij is the optimal [4] function determined by iteration of ACE ( Ameyded Conditional Expectation). In the experiments, respecting to 5 different emotion states, 20 samples are collected. 6 emotion characteristic parameters are gathered from 20 emotion sentences for each emotion state. Thus, we turn every emotion sentence to six dimension primitive eigenvector. Each dimension element unit is not unified, therefore they need to be turned to a standard unit by the parameter normalization standard when recognition. Twenty sentences are used to train according to the function Euclidean, which defined before, therefore n equal to 20. The weight of emotion characteristic parameters are calculated as the table 2 shows: For each emotion state, the summation of 6 emotion characteristic parameters is 1, 6

∑w

j

=1

w

w

w

w

w

w

that is . In the table, 1 ， 2 ， 3 ， 4 ， 5 ， 6 , Separately corresponding to average short time energy, average change rate of amplitude, average fundamental frequency, average change rate of fundamental frequency, average short time pass zero rate, the time ratio of surd and sonant. When recognition, 6 emotion characteristic parameters of the sentence is gathered, then calculate the normalization eigenvector of each emotion template. Match the normalization j =1


423

Table 2. Emotion state weight starting value assignment

Weight

Emotion state Calmness

Fear

Happy

Sad

Anger

w1

0.21

0.19

0.22

0.20

0.25

w2

0.16

0.15

0.14

0.11

0.16

w3

0.23

0.22

0.25

0.24

0.27

w4

0.11

0.12

0.19

0.18

0.11

w5

0.14

0.11

0.11

0.16

0.12

w6

0.15

0.21

0.09

0.11

0.1

ǃ eigenvector with five emotion templates. Calculate each Euclidean distance of each emotion state. The emotion state that has the least Euclidean distance is the result of the recognition. 4.4 Result of the Experiment In the experiments, with respect to 5 different emotion states, 40 samples are collected, which are then recognized and the results are showed as table 3: Table 3. Result of the speech emotion experiment

Output Input

Calmness

Fear

Happy

Sad

Anger

Recogniti on Rate

Calmness

38

0

0

2

0

95%

Fear

0

30

4

0

6

75%

Happy

0

6

27

0

7

67.5%

Sad

0

0

0

40

0

100%

Anger

0

4

3

0

33

82.5%

From the data of the table, we know that the recognition rate of sad and calmness reach 100% and 95%, because the amplitude and the fundamental frequency of this two emotions are much lower than other three emotions. We can differentiate sad and

424

Y. Tang et al.

calmness by the average change rate of fundamental frequency easily. But the recognition rate of happy only reaches 67.5%. Sometimes happy would be recognized as fear and anger, because the fundamental frequency and the average short time energy of happy are close to the fear and anger. Similarly fear and anger also have the certain probability to recognize to happy. The recognition rate of fear and anger only 75% and 82 %. Result of the experiment shows that the recognition of sad and calmness and anger is more exact than the happy and fear. There are 3 reasons for this phenomenon. First of all, since there are significant similarity features of happiness, horror and anger, and thus the Euclidean distance is not good enough to be used to classify these features. Secondly, emotion feature is heavily related with specific context, so the samplings collected in our experiments may be short of emotion feature. Thirdly, the number of our samplings is limited, and thus the training of voice emotion model is far from perfect.

5 Conclusion This article mainly introduces a design of the spiritual consolation object, and specializes in introduction of the hardware architecture of the system, the principles of pronunciation of signal's emotion. The system realizes how to save speech signal, how to recognize five basic speech emotions and also supplies the service of spiritual consolation. This system can improve the living quality of the elder who live alone; strengthen the connection and communication between solitary elder and their children, relatives and friends. And it offers a kind of new support for establishing harmonious family and harmonious society. With the quick development of the computer science, computers, which have the function of affective computing, will be used more widely. The future research will pay more attention to affective information got through speech, vision and touch, and improve the precision of emotion recognition, which provides better service for the elder who live alone.

References 1. http://www2.convention.co.jp/5isg/english/ 2. Zhao Li,Qian Xiangming, Zhou Cairong, Wu Zhenyang, Pronunciation of signal’s Emotion characteristic’s analyse and research, Communication journal, 2000, volume(21), page 18-24 3. Zan Yongzhao,Cao Peng, Research and implementation of emotional feature extraction and recognition in speech signal, Jiangsu university,2005,1, volume(26), page 72-75 4. COME Roddy, COTNELIUS Randolph R .Describing the emotional states that are expressed in speech[J] Speech Commurzication,2003, volume(40), page 5-32. 5. Cowie. R. Emotion recognition in human-computer interaction. IEEE Signal Processing Magazine .2001, volume (18), page 32-80.

Real Time Dynamics and Control of a Digital Human Arm for Reaching Motion Simulation Huijie Yu1 and Ray P.S. Han2 1

2

Dept. of Mechanics & Engineering Science, Fudan University 200433 Shanghai, China College of Automation Engineering, Qingdao University, Qingdao Shandong 266071 China & Dept. of Mechanical & Industrial Engineering, University of Iowa, Iowa City, IA 52242 USA [email protected]

Abstract. High-fidelity simulations for dynamics and movement control of the human arm is a complex and challenging problem. To realize such high-quality response in real time using today’s computing power is simply not possible. In this paper we have presented a simplified, neural network-based dynamics model of a digital human arm for simulating its reaching movements. The arm is modeled as an assemblage of 2 rigid bodies connected by joints and muscles. A Hill-type model is used to capture the muscle forces. Activation dynamics via a low-pass filtering operation is applied to trigger musculotendon contractions. Since we are interested in real-time macro response, we have replaced the forward dynamics module with a recurrent neural network approach. The results of our simulation show that the model produces fairly realistic reaching motion of the arm and it is able to do so in real-time. Keywords: Digital human arm modeling, real-time dynamics and control, arm reaching movements, arm musculoskeletal model, neural network.

1 Introduction The tremendous increase in computing power and modeling technologies has made it viable to develop high-fidelity models of humans or more aptly, “digital humans” for engineering applications. At this point in time, a truly realistic model of the human arm is quite beyond the capabilities of even the most powerful computers. The reason is our knowledge of the interaction between the human brain and the arm is still quite limited. We do not have a good understanding of the brain and how it activates and interacts with specific muscles of the arm. Further, an accurate fatigue-based model of the human muscles is still unavailable. Therefore, in this work, we will present a macro-model of the digital human arm that emphasizes on the gross reaching motions in real time. The reaching motion has the characteristic of fast movement towards a known target via piecewise straight paths and bell-shaped speed profiles. The movement of the human arm can be analyzed as consisting of the following phases: 1) the brain becomes aware of the target position through the eyes, 2) the central nervous system (CNS) sends appropriate signals to the muscles causing them to contract and produce forces to move the arm, 3) the eyes continue to transmit Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 425 – 436, 2006. © Springer-Verlag Berlin Heidelberg 2006

426

H. Yu and R.P.S. Han

visual information to the brain as it controls and moves the arm until the target is reached. To simulate the reaching movement of the human arm, 3 key issues should ideally be addressed. First， the arm model should not be just an assemblage of rigid bodies; it must have muscles to mimic the real human arm. So, it is necessary to build a model possessing basic anatomical characters. The second consideration is that the real arm is a redundant system consisting of many muscle types, all of which are prone to fatigue. Third, there are several trajectories that the arm can use to arrive at its target position. The trajectory solution is therefore, not necessarily unique and the model needs to generate a realistic path that consumes the least effort. Over the past 2 decades, there has been a significant increase in research on the modeling of the human arm. Dornay et al. [1] presented an arm model that employs a controller programmed with a minimum muscle-tension change criterion. Further, by using inverse models of the arm dynamics for controls, their process of simulating the arm reaching movements became fairly complex and computationally intensive. Jordan and Rumelhart [2] created a controller-based forward dynamics model for the supervised learning paradigm. They showed that the algorithm is capable of learning in multi-layer networks. Massone and Myers [3] described the response of a neural network model of an anthropomorphic redundant arm moving in a horizontal plane with gravity loading. They used a 2-joints model with 6-muscle pairs actuators and subjected the system to various patterns of muscle activations. They investigated several control strategies, plant properties and the resulting motion qualities with the objective of developing guidelines for the design of artificial arms. Karniel and Inbar [4] developed a mathematical model for the control of the human arm to simulate reaching movements. Their model consists of a 2-DOF planar manipulator used in conjunction with a Hill-type, 6-muscle unit. Rectangular pulses with varying amplitudes and timings that are set based on the target, are inputted as control signals. By adjusting a handful of parameters in lieu of the entire trajectory, they managed to drastically reduce the dimensionality of the control problem. Fagg et al. [5] developed a cerebellar learning model for controlling arm movement via muscle synergies. In the event that the cerebellum is unable to move the arm to its target, an extracerebellar module is activated to carryout low-fidelity corrections and the information is fed back to the cerebellum. Their model employed delayed sensory signals together with the generated motor commands to compute new output signals. Ogihara and Yamazaki [6] built an arm model with neuro-control to generate reaching motion without requiring a trajectory. They used 3-rigid links with 8 principal muscles to construct their musculo-arm model. Further, at each joint they attached a nonlinear viscoelastic element. However, their muscle model uses a constant distance for computing the muscle torque. This method overestimates the torque magnitudes and consequently, produces unrealistic reaching movement. Kashima et al. [7] analyzed a muscle control system in elbow joint movements through theoretical and experimental analyses. They used an EMG-based neural input to control the behavior of individual muscles. In their study, they compared their computed trajectories with experimental observations and showed that their model is capable of capturing the essential responses even in multi-joint motion. Lan [8] provided a stability analysis of a 2-joint limb using muscles that are activated electrically. He constructed a planar arm model with 3 pairs of antagonistic muscles in ADAMS and deployed it for studying the local feedback controller performance. By evaluating the eigenvalues at

Real Time Dynamics and Control of a Digital Human Arm

427

a current posture of the model he was able to examine the stability behavior for postural control of the motor tasks. Thelen et al. [9] introduced a “computed muscle control” algorithm that employed static optimization with feedforward and feedback controls to guide the trajectory of their musculoskeletal model towards its target. They demonstrated their algorithm by computing muscle excitations for driving a 30muscle, 3-DOF model to simulate kinematics and kinetics of pedaling. De Sapio et al. [10] presented a task-oriented control strategy for generating feedback controls in their attempt to simulate goal-targeted human motion. They reported a framework to produce musculoskeletal simulations for synthesizing control of human-like motion. In this paper, we will present a model of a digital human arm that is dynamically operated and controlled in real time for its reaching motions. The arm is constructed as a planar assemblage of 2 rigid links connected by 2 joints and 6 muscles. The neuroexcitations are transmitted to the muscles via a simple arm controller that is trained to generate typical reaching movements. To enhance the computational efficiency, a recurrent neural network is used in lieu of the forward dynamics calculations. In doing so, we are able to simulate in real-time the goal-directed motion of the human arm fairly realistically.

2 The Arm Musculoskeletal Model As depicted in Fig. 1, the arm musculoskeletal model consists of a rigid multi-body system connected by shoulder and elbow joints, and spanned by a set of musculotendon actuators [3]. Motion in the model is restricted to the vertical plane that takes into consideration the effects of gravity. The 6 muscle-like actuators are located as follows: 2 antagonist actuators that act across the shoulder joint (shoulder muscles), 2 antagonist actuators that act across the elbow joint (elbow muscles) and 2 antagonist actuators that act across both the shoulder joint and the elbow joint (double-joint muscles).

Fig. 1. Arm musculotendon model: (a) arm model with 2 joints (shoulder and elbow); (b) muscle model with the following muscles: a-Pectoralis Major, b-Deltoidus, c-Biceps L-H, d-Triceps S-H, e-Biceps S-H, f-Triceps L-H

428


2.1 Arm Musculotendon Dynamics The arm is modeled as a system of constrained 2-DOF mechanical-like manipulator whose configuration is described by a set of generalized coordinates q . The generalized coordinates consist of joint angles defining the orientation of the limb segments. The equation of motion can be written in its standard form as,

M (q)q + h(q , q) + g (q) = T ,

(1)

where M (q) is a ( 2 × 2 ) inertia matrix, q is a ( 2 × 1) vector of joint angles, h( q , q ) is a ( 2 × 1) vector of torques arising from Coriolis and centrifugal forces, g (q) is a ( 2 ×1) vector of torques due to gravity and T is a ( 2 ×1) vector of joint torques computed from the musculo-dynamics forces. To ensure good fidelity of the modeled arm, the inertial and other parameters of each segment are sourced from Atkeson [11]. Their values are listed in Table 1. Table 1. Inertial parameters of the limb segments

Segment

Mass (kg)

Length (m)

C.G. (m)

Inertia (kg-m2)

Upper arm

1.82

0.326

0.145

0.0201

Forearm

1.43

0.444

0.302

0.0287

2.2 Neuromuscular Dynamics

The time evolution of muscle states in terms of their length and force magnitudes in response to neural excitation is considered next. As sketched in Fig. 1(b), a system of musculotendon actuators is defined to drive the arm. Each arm segment is driven by a pair of muscles acting in unison. The joint torque is then computed by the net effect of the agonist-antagonist muscle pair as each muscle produces torques that pull the arm in the opposite direction. Further, the torque at each joint is a weighted sum of the torques of the muscles that act on that joint. Following Massone and Myers [3] we can write Ti the torque at joint i as T1 = Tsh _ fl − Tsh _ ex + λ1 ⋅ (Td _ fl − Td − ex ), T2 = Tel _ fl − Tel _ ex + λ2 ⋅ (Td _ fl − Td − ex ).

(2)

Additionally, Tsh _ fl and Tsh _ ex are respectively, the flexor and extensor torques for the shoulder. Similarly, Tel _ fl and Tel _ ex are the flexor and extensor torques for the elbow, and Td _ fl , Td − ex are the flexor and extensor torques for the muscles that act across both joints. The connection strengths λ1 and λ2 can either be arbitrarily set or learned at the time that the arm controller is programmed. In this paper, we adopted the following values: λ1 = 0.6, λ2 = 0.4 [3].


429

A Hill-type nonlinear mechanical representation is used to model each muscle [1213]. As schematically illustrated in Fig. 2, the model consists of a contractile element with a force generator F0 , 2 damper elements of viscosity B and B p , and 2 spring elements with stiffness K s and K p connected together. The force generator is modulated by neural input ni . Activation dynamics is then applied to bring about musculotendon contractions in the Hill-type model [14-15]. In our model this process is described by a low-pass filtering operation, which can be captured by the following state-form equation written in terms of the activation a a =

1

τ ne

(ni − a ) ,

(3)

where ni ∈ [0,1] is the neural input and τ ne refers to the time constant of the filter.

ni F0

x0

Ks

x

B

Bp

Kp

Fig. 2. The general mechanical model of the muscle: ni : neural input; F0 : hypothetical tension;

B : damper; K s : series elasticity; Bp : parallel damper; x : muscle length; x0 : length of contractile element

The activation dynamics result in the following force F0 given by 1 F0 = (a − F0 ) .

τa

(4)

Note that τ a is the time constant of the process. The actual force is obtained by calibrating F0 using the constant Fmax . Table 2 lists the maximum forces for the six muscles in our arm model [8, 12]. The result is then multiplied by a constant mean moment arm d to yield a theoretical torque T0 . T0 = F0 ∗ Fmax ∗ d

(5)

430

H. Yu and R.P.S. Han Table 2. Maximum force for various muscles

Fmax

(N )

Muscle Type

Function

a: Pectoralis Major

Shoulder Flexor

800

b: Deltoidus

Elbow Flexor

700

c: Biceps L-H

Shoulder Extensor

800

d: Triceps S-H

Elbow Extensor

700

e: Biceps S-H

Double-joint Flexor

1000

f: Triceps L-H

Double-joint Extensor

1000

The relationship between the theoretical torque and the length of the contractile element is described by

T0 + Bx0 = K s ( x − x0 ) .

(6)

Finally, the torque produced by the muscle Tm can be determined from

Tm = B p x + K p x + K s ( x − x0 ) .

(7)

The values of various muscle parameters and their corresponding units are tabulated in Table 3 [3]. Table 3. Values of the muscle parameters used

Parameter

Value

Units

Bp

0.2

N.sec/rad

B

4.5

N.sec/rad

Kp

1.5

N/rad

Ks

30

N/rad

τ ne

0.015

sec

τa

0.005

sec

d

0.03

m


431

Further, the relationship between the muscle length x and the joint angle q are listed in Table 4. Table 4. The relationships between each muscle length and the joint angle (m) xShoulder _ Flexor (

3π − q1 ) d 2

xShoulder _ Extensor (

π 2

+ q1 )d

xElbow _ Flexor

x Elbow _ Extensor

xdouble _ Flexor

xdouble _ Extensor

(π − q2 )d

(π + q2 ) d

3π ( −q1 − q2)d 2

( +q1 +q2)d 2

π

3 Dynamics Model The simulation involves moving the arm from its initial position to a defined target position as depicted in Fig. 1(a). The main characteristics of such a movement are piecewise straight-path with bell-shaped speed profiles. To achieve this, the following potential U ( X ) is introduced for computing the minimum traversed distance U ( X ) = ( X − X T )T K X ( X − X T ) + X T KV X ,

(8)

where X , X T are respectively, the initial and target positions of the arm’s tip in Cartesian coordinates and X is the velocity of the arm tip. Further, K X , KV are the gain errors for position and tip velocity. They are ( 6 × 2 ) matrices and their values are

chosen to closely reflect the physiological properties of the arm so that the reaching movement appears to be smooth and natural. In Eq. (8) the first term refers to a penalty for not satisfying X − X T = 0 constraint. It decreases as the tip position X (q) approaches the target position X T . The second term refers to the arm stopping criterion, namely X = 0 when the arm-tip reaches the target and U ( X ) approaches

zero. Hence the object of the control is to minimize so that U ( X ) → 0 .

KV | f |≤ TOL?

ni

f

KX

T Tm

q q

X

X − XT

XT

Fig. 3. Controller block diagram for computing the reaching movement of the arm

X

432


Modeling the reaching movement of the arm can be a complex operation that requires several steps as schematically illustrated in the block diagram of Fig. 3. The muscles are activated by their neural input ni and this generates generalized muscle torque Tm ( ni , q, q ) . From the arm geometry the muscle torque is then converted into

joint torque T. This information is fed into the forward-dynamic block to compute angular acceleration q as a function of the current T , q, q values. An integrator is applied to determine new values of the velocity and position q, q . Upon attaining convergence in the local iteration, the forward-kinematic block is then activated to transform the angular coordinates into Cartesian coordinates of the endpoint X. To guide the iterative process, the error ( X − X T ) and X are sampled to repeatedly adjust the neural input ni until a predefined tolerance is achieved at convergence. 3.1 Forward Dynamics

As shown in Fig. 4, the forward dynamics of the arm involves mapping the computed joint torques to angular accelerations and then, applying an integrator to obtain their corresponding angular positions and angular velocities. The process involves repeated local iterations until converged values of the kinematics are attained.

T

q

q

q

Fig. 4. Forward dynamics integrator for mapping joint torque T to q, q , q

Eq. (1) can be re-written to compute the angular acceleration q

q = M ( q ) ⎡⎣T ( q, q , q) − h ( q, q ) ) − g ( q ) ⎤⎦ . −1

(9)

Since the kinematic quantities on the right-hand-side of Eq. (9) are known only at the previous time-step, Eq. (9) must be solved repeatedly until converged values of q, q , q are obtained for computing M ( q ) , T ( q, q , q) , h ( q, q ) , g ( q ) . The forward dynamics simulation can be time consuming as the process requires local iterations at every time-step. To accomplish the simulation for real-time control we have developed a recurrent neural network technique to cope with the computational complexities as well as the inherent nonlinearities in the model.


433

3.2 Neural Network

Neural network has become a powerful proven tool for controlling complex nonlinear dynamic systems. The basic idea is to employ an appropriate neural network estimator to identify the unknown and/or complex nonlinear dynamics and to compensate for it. To increase the computational efficiency of our model, a recurrent neural network is deployed to replace the forward-dynamics module. As shown in Fig. 5, the recurrent neural network consists of 3 layers; the first is the input layer, which has 6 units, followed by the second and hidden layer, which has 10 units and the third is the output layer, which has 2 units. The output function of the hidden layer is sigmoid-type, while all others are linear. Training the network is most important, as well as, most time consuming. Thus, for our work it is best to train the network off-line. The weights for learning are from the input layer to the hidden layer, and from the hidden layer to the 2 units that output the angular accelerations. We chose to construct the training set for the forward dynamic simulation of the reaching motion whereby, the endpoint starts out an a predefined initial position of the workspace and moves via a number of straight-line paths towards the given target position.

T1

T2

q1

q1

q2

#

q2

q1 q2 N

N

N

Fig. 5. Structure of the recurrent neural network: the input layer with 6 units is first, followed by a hidden second layer with 10 units and the third layer is the output containing 2 units

There have been many reported cases [16] where the shoulder angle operates in the range of ( -0.817, 2.789 ) while the elbow angle in the range of ( 0.000, 2.489 ) in the vertical plane. Our technique involves partitioning the 2 angles into 20 parts and sampling the tip coordinates of each of the 21 movements (that is, inclusive of the start and end points) in an attempt to generate a smoother motion. This is illustrated in

434


Fig. 6 with 200 samples (with a sample period of 0.005s) in each direction for an assumed bell-shaped velocity profile. With kinematics solved, that is, q, q , q are known, the joint torque T for 21× 200 = 4200 samples can be readily determined via Eq. (1). Prescribing the computed torque values, together with the known angular positions and angular velocities as inputs to the training set, we get as outputs, the angular accelerations. Further, in employing an analytical approach to compute the time derivatives of the angular quantities, our training set should be free of the associated numerical errors. We then trained the 3-layer network using the prescribed training set in conjunction with the standard back-propagation (BP) algorithm [17]. The network was able to learn quite nicely by generating acceptable new values of torques, angular positions and angular velocities that are different from those stored in the training set. We were able to obtain an error less than 10 −4 during our training of the network 664 epochs. The error curve of the training is plotted in Fig. 7.

Fig. 6. The reaching movements selected for constructing the training set. Note that the symbol “ × ” denotes target samples.

Fig. 7. The error curve of the training. The training of the 664 epochs is stopped when an error ≤ 10−4 is attained.

3.3 Control Method and Simulation

The coordinates of the arm-tip X can be computed via forward kinematics involving the angular position q that in turn, is integrated from angular acceleration q fed by the neural network module. To ensure that the system stays robust even in the presence of disturbances, a feedback control scheme for the model is adopted. The following control response as input to the decoupled system is designed for its use f = −∇U = − K X ( X − X T ) − KV X .

(10)

Simulation experiments are performed to test the model. The arm-tip is initially located at ( 0, 0.32 ) and it is required to move to the target position at ( -0.10, 0.67 ) .


435

The simulation results are displayed in Fig. 8. Observe that the arm-tip moves via a series of piecewise straight-line path with a bell-shaped speed profile to the target position. Furthermore, unlike other control models [18], our approach does not make use of the reference trajectory.

(a)

(b)

Fig. 8. Simulating the reaching movement of the arm: (a) trajectory of the arm-tip as it moves from the initial position to the target position, (b) velocity profile

4 Concluding Remarks We have developed a 2-DOF planar arm model spanned by 3 antagonist muscle-pairs: shoulder-muscle pair, elbow-muscle pair and double-joint muscle pair to simulate in real-time the reaching movement of a digital human arm. A recurrent neural network approach is used in lieu of the arm forward dynamics module to accomplish the real time movement control. The muscle model used is based on the Hill-type muscle unit. The simulation results show that the digital arm moves in a natural and human-like motion toward a given target position using a series of piecewise straight-line path with a bell-shaped speed profile. Our next step is to incorporate fatigue in the muscle model. Acknowledgment. Funding for this work from the National Science Foundation of China through Grant No. 50475048 is gratefully acknowledged.

References 1. Dornay, M., Uno, Y., Kawato, M., Suzuki, R.: Simulation of optimal movements using the minimum-muscle-tension-change model. Adv. Neural Info. Processing Syst. 4 (1991) 627-634. 2. Jordan, M.I., Rumelhart, D.E.: Forward models: supervised learning with distal teacher. Cognitive Sci. 16 (1992) 307-354.

436


3. Massone, L.L.E., Myers, J.D.: The role of plant properties in arm trajectory formation: a neural network study. IEEE Trans. Syst. Man Cybern. B 26 (1996) 719-732. 4. Karniel, A., Inbar, G.F.: A model for learning human reaching movements. Biol. Cybern. 77 (1997) 173-183. 5. Fagg, A.H., Sitkoff, N., Barto, A.G., Houk, J.C.: A model of cerebellar learning for control of arm movements using muscle synergies. 1997 IEEE Int. Symp. Computational Intelligence in Robotics and Automation (CIRA ’97) (1997) 6-12. 6. Ogihara, N., Yamazaki, N.: Generation of human reaching movement using a recurrent neural network model. Syst. Man Cybern., IEEE SMC ’99 Conf. Proc. 2 (1999) 692-697. 7. Kashima, T., Isurugi, Y., Shima, M.: Analysis of a muscular control system in human movements. Biol. Cybern. 82 (2000) 123-131. 8. Lan, N.: Stability analysis for postural control in a two-joint limb system. IEEE Trans. on Neural Syst. Rehab. Eng. 10 (2001) 249-259. 9. Thelen, D.G., Anderson, F.C., Delp, S.L.: Generating dynamic simulations of movement using computed muscle control. Jour. Biomech. 36 (2003) 321-328. 10. De Sapio, V., Warren, J., Khatib, O., Delp, S.L.: Simulating the task-level control of human motion: a methodology and framework for implementation. Visual Comp. 21 (2005) 289 – 302. 11. Atkeson, C.G.: Learning arm kinematics and dynamics. Ann. Revw. Neurosci. 12 (1989) 157-183. 12. Winters, J.M., Stark, L.: Analysis of fundamental human movement patterns through the use of in-depth antagonistic muscle models. IEEE Trans. Biomed. Eng. 32 (1985) 826-839. 13. Zangemeister, W.H., Lehman, S., Stark, L: Simulation of head movement trajectories: model and fit to main sequence. Biol. Cybern. 41 (1981) 19-32. 14. Schutte, L.M.: Using musculoskeletal models to explore strategies for improving performance in electrical stimulation-induced leg cycle ergometry. PhD. thesis, Stanford University (1992). 15. Zajac, F.E.: Muscle and tendon: properties, models, scaling, and application to biomechanics and motor control. Critical Reviews Biomed. Eng. 17 (1989) 359–411. 16. Zhang, A.: Motion simulation of the human upper limb based on MATLAB. Master thesis, Tianjin University, Tianjin, China. (2003) (in Chinese). 17. Pineda, F.J.: Dynamics and architecture for neural computation. Jour. Complexity 4 (1988) 216-245. 18. Katayama, M., Kawato, M.: Virtual trajectory and stiffness ellipse during multijoint arm movement predicted by neural inverse. Biol. Cybern. 69 (1993) 353-362.

Emotion Recognition Using Physiological Signals Lan Li1 and Ji-hua Chen2 1

School of Electrical and Information Engineering, Jiangsu University, Zhenjiang, 212013, P.R. China [email protected] 2 Institute of Biomedical Engineering, JiangSu University, ZhenJiang, 212013, P.R. China

Abstract. The ability to recognize emotion is one of the hallmarks of emotional intelligence. This paper proposed to recognize emotion using physiological signals obtained from multiple subjects without much discomfort from the body surface. Film clips were used to elicit target emotions and an emotion elicitation protocol, verified to be effective in the preliminary study, was provided. Four physiological signals, electrocardiogram (ECG), skin temperature (SKT), skin conductance (SC) and respiration were selected to extract 22 features for recognition. We collected a set of data from 60 female undergraduates when experiencing the target emotion. Canonical correlation analysis was adopted as a pattern classifier, and correct-classification ratio is 85.3%. The research indicated the feasibility of user-independent emotion recognition using physiological signals. But before emotion interpretation can occur at the level of human abilities, there still remains much work to be done.

1 Introduction Nowadays affective computing has become the hotspot in computer science. Recording and recognizing physiologic signatures of emotion has become an increasingly important field of research in affective computing and human-computer interface [1]. Traditional investigation, which has made considerable achievements, is based on the recording and statistical analysis of physiological signals from Autonomic nervous system [2]. Some researchers have been doing their best to develop wearable devices, while others devoting themselves to implementing a physiological signal-based emotion recognition system [3,4,5]. In 1999, researchers at IBM developed an emotion mouse about 75 percent successful in determining a user's emotional state [3]. In 2001, Picard and colleagues at MIT Media Laboratory developed pattern recognition algorithms which attained 81% classification accuracy [4]. Because of data acquired from only one subject, these emotion recognition methods can only measure one subject’s emotion. In 2004, Kim and his group developed a multiple-users emotion recognition system using short-term monitoring of physiological signals. A support vector machine (SVM) was adopted as a pattern classifier, and correct-classification ratio for 50 subjects is 78.4% [5]. This paper discussed how to recognize emotion using four physiological signals obtained from multiple subjects. It is novel and different from previous research. Film clips were used to arouse the inner feelings of the subjects, and an emotion elicitation Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 437 – 446, 2006. © Springer-Verlag Berlin Heidelberg 2006

438

L. Li and J.-h. Chen

protocol, verified to be effective in the preliminary study, was provided. 22 features were extracted from 4 physiological signals. Canonical correlation analysis was adopted to find the relationship between three emotions and extracted features. And the recognition accuracy is 85.3%.

2 Method 2.1 Emotion Elicitation Protocol Compared with other emotion elicitation techniques such as images, sounds, facial and body movement, scripted and unscripted social interactions, music, and so on, films are more reliable and more naturalistic to induce internal feelings of the subjects[6]. In order to evoke specific target emotion statuses effectively, we chose over three film clips (3~8 minutes in length) as stimulus for each target emotion. 89 (male 36; female 53) undergraduate students, aged from 18 to 23, in Jiangsu University took part in the preliminary test. Through completing post-film questionnaire, the subjects were encouraged to report the state of the emotion elicited by the film clips, and to use a five-point scale (0-very low， 1-low， 2-high， 3- median high， 4-very high) to report the intensity of the emotion . Validity (the percentage of subjects who report that the given stimulus properly induced the intended motion) and the average intensity of the emotion were used to evaluate the quality of the emotion stimulus. After the preliminary test, we drew conclusions as following: • The stimulus clips for fear, neutral, and joy were chosen successfully, while others for anger, disgust, sadness, embarrassment, surprise, anguish, contempt, stress, interest, satisfied, were not. • The most effective stimulus for three emotions was picked out, and showed in Label 1. Validity of fear, neutral and joy is 86.67%, 93.33% and 90.0% respectively; while the average intensity is 2.85, 2.63 and 3.25 respectively. • Given the same stimulus, female students felt much stronger than male students. • That three stimulus were shown sequently in a way, ‘fear-joy-neutral’, was proved to be more effective for the subjects to transition from one emotion state to another emotion state than in the way, ‘fear-neutral-joy’. Table 1. Summary of emotion-elicitation protocol Target Emotion Fear Joy neutral

Decscription

Film

Clip length

Frightened Uplifting happiness Relaxation, vacancy

The Doll Tom and Jerry Noncommercial screen saver

5’12’’ 6’45’’ 3’26’’

2.2 Acquisition of Emotion-Specific Physiological Signals Subjects. Since target emotion state is easier to be elicited from female students, our target subjects were 60 female undergraduate students aged from 18 to 23. All

Emotion Recognition Using Physiological Signals

439

subjects were healthy volunteers without any history of medical, neurological or psychiatric illness. Before the test, they have not done exercise heavily in four hours and have not taken any medicine in a week. Otherwise, the subjects were chosen randomly, while the order of the subjects is arranged randomly. Selection of the physiological signals. To develop a practical recognition algorithm, the physiological signals which we can choose are very important as well as very limited at the same time. Although electroencephalogram (EEG), facial electromyograms, and blood pressure may be helpful for the research, the attachment of electrodes to the scalp or face seems not to be tolerable for practical use, thus they are left out of consideration. Nowadays almost all wearable devices developed can measure four major physiological signals: skin temperature (SKT), skin conductance (SC), heart rate (HR), and electrocardiogram (ECG). SKT is an important and effective indicator of emotion states. The variation in SKT due to emotional stimulus was studied by Shusterman, Barnea (1995) and Kataoka(1998) [7]. It reflects autonomic nervous system activity. Skin conductance (SC), sometimes called the Electrodermal activity (EDA) or Galvanic skin response (GSR), is another important signal to represent the activity of the autonomic nervous system. It characterizes changes in the electrical properties of the skin due to the activity of sweat glands and is physically interpreted as conductance. Sweat glands distributed on the skin only receive output from the sympathetic nervous system, and thus SC is a good indicator of arousal level due to cognitive stimulus. Heart rate (HR) is dually controlled by the sympathetic (increase) and parasympathetic (decrease) branch of the ANS that may act independently [8]. In addition, time-domain features, such as mean and standard deviation (SD) of the HRV (heart rate variability) time series have also been considered to be significant for the exploration of autonomic nervous system in many previous studies for cardiac function assessment and psychophysiological investigation. They have been frequently used as features [9]. Since ECG signal can be obtained relatively easily, and both time-domain features of HRV and HR can be computed from it, ECG signal seems to be very important in this study. In addition, respiration is very important in the emotion research. Specific emotion expressions, such as crying, laughing, or shouting, have unique respiratory signatures. A detailed quantification of volume, timing and shape parameters in the respiratory pattern waveform can map into different emotional states along the dimensions of calm-excitement, relaxation-tenseness, and active vs. passive coping [10]. Initial evidence indicates that respiratory parameters also map into the ‘affective space’ dimensions of valence (aversive vs. appetitive stimulus quality) and arousal (activating vs. calming stimulus quality) [11]. So, in this research, four physical signals--SKT, SC, ECG and Respiration are selected to extract features for emotion recognition. Experiment devices. Good devices are essential to gather good data. The equipments used in this research are:

440


World-renowned PowerLab Data Acquisition System with Chart software. It is developed by AD Instruments Company in Australia, including: ML870 Powerlab 8 channel data acquisition system MLT409/A SKT probe ML309 Thermister Pod ML116F GSR finger electrodes ML116 GSR AMP MLT1132 Piezo respiratory belt transducer MLA700 Reusable ECG clamp electrodes ML132 Bio AMP A USB camera 800*600 pixels Two computers. One is used to show the film clips to elicit the target emotion of the subject; while the other, connecting with PowerLab ML870 and a USB camera in covert, is used to record, analyze, preprocess the physiological data, and monitor the facial or body expression of the subject. Experimental method Preparation. Before the experiment, the subject was requested to report the personal information and sign a volunteer grant. By talking, we could make the subject feel relax, not curious and be a calm state. Then, attach the sensors to the subject. The MLT409/A SKT prob was placed on the tip of the thumb of the left hand. The bipolar MLT116F GSR finger electrodes measured SC from the middle of the three segments of the index and middle fingers of the left hand. And based on lead I, three reusable ECG clamp electrodes of the MLA700 were attached to wrists and right ankle. While the MLT1132 piezo respiratory belt transducer was placed around the body at the level of maximum respiratory expansion. Instructions. The test will begin soon. Please make yourself be as relaxed and comfortable as possible. After 1 minute, film clips will be showed. Please watch the films carefully; don’t move casually; let yourself experience whatever emotions you have as fully as you can, and don’t try to hold back or hold in your feelings. Thank you for your cooperation! Data collection. Acquisition of high-quality database of physiological signals is vital for developing successful emotion recognition algorithm. The emotion stimulus was showed in a way, ‘fear-joy-neutral’. One emotion session took approximately 5~8 minutes. The first 1 min was taken to measure the baseline without any stimulus. Subsequently, emotional stimulus was applied. The physiological signals were recorded, at the sampling rate of 400 HZ for all the channels, by using PowerLab 8-channel physiological Data Acquisition System with Chart software. At the same time, a trained graduate monitored the subject with a hidden PC camera. When there were sudden changes in the facial or body expression, special marks and annotations would be added to the record to provide supplementary information for later data-processing. After one session, the subject was requested to


441

complete the post-film questionnaire. Then there would be 5 minutes interval for the subject to be calmed down before another emotion session. Two segments of raw data (SC and Respiratory signal) under three different emotion states were shown in Fig. 1.

(mv)

(mv)

(c) Respiratory signal

1200

RR Intervals (ms)

1200

RR Intervals (ms)

1200

(f) HRV 1000

1000

1000

800

800

800

600

10 30 50 Interval Number

600


600

RR Intervals (ms)


Fig. 1. Examples of physiological signals measured from a user during the period of experiencing the target emotion (Fear, Joy, Neutral). From top to bottom: (a) skin conductivity (MicroSiemens), (b) the difference of skin conductivity, (c) respiratory signal (raw digital signal obtained with the respiratory belt), (d) respiratory rate, (e) Heart rate , (f) HRV time series. The sampling rate is 400/s. The segment shown here are visiblely different for the three emotions, which was not true in general.

442


2.3 Data Processing Since each emotion session would last around 5~8 minutes, and signals were sampled at 400Hz, there would be about 120 to 192 thousand samples per physiological signal. It is necessary to extract a short time significant segment of the signals from the raw data. The only criterion of extraction is whether the signals were sampled during the period of experiencing strong feelings of the target emotion. According to the emotion theory, SC is a good indicator of arousal level. Base on the comprehensive consideration of SC differentiation and the marks & annotations made during the period of experiments, data segments (1 min length) were taken for features extraction from the raw pattern waveform of SKT, ECG, Respiration for each of the three emotions. SKT. No special signal processing was necessary for SKT. Although frequencydomain analysis of the time-varying SKT has been reported [7], here we used the mean and the difference between the maximum and the minimum within 1 min as the features of SKT. SC. The raw SC signal was shown in Fig. 1a, while differentiation signal, computed by chart software, shown in Fig. 1b. In this research, the SC differentiation signal was used to be one important basis to choose samples from raw waveform. And the first difference of it was used as an important feature [4]. Respiration. Respiration parameters can be computed from the changes in thoracic or abdominal circumference during respiration, which can be measured easily by a piezo-electric device contained by the Respiratory Belt Transducer MLT1132. Here, we took respiratory rate (see Fig. 1d) and peak inspiratory amplitude as features. ECG. Heart rate can be computed by R-peak detection. Fig. 1e illustrates the heart rate waveform detected from raw ECG signals. Mean of the heart rate and the difference between the maximum and the minimum are taken into account. The time-domain features of HRV can be calculated by Chart conveniently and accurately. These are : HF power____high-frequency heart rate variability spectral power [0.15~0.4HZ] LF power____low-frequency heart rate variability spectral power [0.04~0.15HZ] LF/HF____ratio of low-to high-frequency power Mean NN____ the mean of the normal cardiac cycle SDNN____the standard deviation of the normal cardiac cycle SD Delta NN____the standard deviation of the delta of the normal cardiac cycles Mean T____the mean of T-wave amplitude SDT____the standard deviation of T-amplitude SD Delta T____the standard deviation of the delta of T-amplitude Mean R____the mean of R-wave amplitude SDR____the standard deviation of R-wave amplitude SD Delta R____the standard deviation of the delta of R-amplitude PNN50____(NN50 count) / (total NN count), the fraction of consecutive NN intervals that differ by more than 50 ms’ RMSSD____the square root of the cumulate of the square of the delta of the normal cardiac cycles Ratio____ SDNN/SD Delta NN


443

The features listed above were adopted in this research. Some of them were considered to correlate with emotion from the literature, and some we supposed to. 2.4 Pattern Classification Using Canonical Correlation Analysis Feature vectors extracted from multiple subjects under the same emotion state form a distribution in high-dimension space. Duda had projected them onto two-dimension space for visualization by a Fisher project. The research showed that the projected feature vectors from the same emotion state formed a cluster with a large amount of variation, and the clusters of feature vectors from different emotion state significantly overlapped. Kim proposed to solve this difficult high-dimension classification problem with the support vector machine (SVM) classifier[5]. SVM is based on the property that separation by a linear classifier becomes more promising after nonlinear mapping onto high-dimensional space. The linear classifier can be obtained with maximum generalization performance derived from the statistical learning theory of Vapnik. Unlike SVM, CCA, known as multivariate multiple regression analysis, can find two sets of basis vectors in which the correlation matrix between the variables is diagonal and the correlations on the diagonal are maximized. Given two random variables, X and Y, the basic CCA model is: CVX1 = a1X1 + a2X2 +...+ apXp. , CVY1 = b1Y1 + b2 Y2 + ... + bm Ym .

(1)

The goal is to describe the relationships between the two sets of variables. The canonical weights (coefficients) a1, a2, a3, ... ap are applied to the p X variables, while b1, b2, b3, ... bm applied to the m Y variables in such a way that the correlation between CVX1 and CVY1 is maximized. It is a combination of predictor and dimension-reducer. More details can be found in [12]. In this research, considering the consistent success in previous evaluations of feature selection algorithms, CCA was adopted as a classifier.

3 Results The classification results are shown in Table 2. 85.3% of test cases can be correctly classified. The classification rates for fear, neutral, joy were 76%, 94%, 84% respectively. And One combined-group figure was shown in Fig. 2, which can visualize the three emotions’ territorial map. In the figure, we can see there is somewhat overlaped between the emotion ‘fear’ and ‘joy’, Which may be the reason of low classification rate on ‘fear’ and ‘joy’. Table 2. Classification rates for three emotions Initial emotion fear neutral joy

Predicted Group Members fear neutral 38 2 1 47 6 1

joy 10 2 43

Total 50 50 50

444

L. Li and J.-h. Chen 6

Emotion ●Fear △Neutral ★Joy ■Group centroid

Fuction2

4 2 0 -2 -4 -4

-2

0 2 Fuction1

4

Fig. 2. Combined-group figure

4 Discussion Like previous research, Our study demonstrated the feasibility of a physiological signal-based emotion recognition again. Although we don’t know what emotion is, how it happens and how it reacts, there does lie clues, pehaps facial or body expression, tone of speech, etc., among which the physiological signals are more natural and more belivable ones (see Fig. 1), to estimate the subject’s emotion state. Although there are some clues in physiological signals, no one know exactly which signal will work best, so signals adopted are somewhat different. Furthermore, there is no standard database. All these make the comparitive study of different method impossible. Expecting to share opinion with researchers in this field, I will analyze why the classification accuracy of our method can reach up to 85.3%. First, data is good and effective. Our data was obtained under emotion states without any externel disturbance or interference. To achieve this goal, the subject was left single in a typical room, electrically shielded and soudproof inside. At the same time, the mobile would be turned off. Sequently, carefully sellected film clips were shown to arouse the inner feelings of the subject. In general, target emtion can be elicited, thus our collecting data really corresponding to real emotion states. Second, emotion category is limited to three kinds, which makes the classification simplified. Since Picard got 88.3% accuracy for three emotion recognition in 2001 [4], it seems that less kinds of emotion, higher accuracy of classification. Third, all 60 subjects were female undergraduates. Since there have been reports of sex differences in several aspects of emotional responding, and in the preliminary study, we did find the problem, we decided to perform research only for female subjects to avoid the sex differences. In addition, a good classifier, CCA, was adopted to find the relationship for the high-dimensional classification problem. The shortcomings of our method is about 3 aspects: First of all, our database is far from complete. Secondly, only three kinds of emotion are concerned. And thirdly, signal processing is relatively simple for SKT and respiration signal. For pactical application, future work shuld be done according to the three aspects.


445

5 Conclusion Emotion recognition is one of the key skills of emotion intelligence for adaptive learning systems. The sheer difficulty in this field is how to gather data corresponding to real emotion states and how to find the relationship between the emotion and the physiological signals in high-dimension space. After 2 years research, we have developed a novel method for a user dependent emotion recognition based on the processing of physiological signals. In our research, the database used for emotion recognition is obtained from multiple subjects when they were experiencing the specific feelings, so the bio-signal database can represent the nature emotion state. To arouse the inner feelings of the subject, film clips, the more effective technique, were used as stimulus. It is different from most of the previous studies, in which the emotion was intentionally ‘tried and felt’ [3], or ‘acted out’[4]. After the preliminary test, three clips that can successfully elicit the target emotion were selected. A good emotion elicitation protocol for the emotion research on Chinese undergraduates is provided. For practical application, we selected four significant physiological signals including ECG, SC, SKT, and respiration, which is easy to obtain relatively, to recognize emotion. Based on the data-processing, 22 features were obtained from the raw signals. To overcome the high-dimension classification difficulty, CCA, a combination of predictor and dimension-reducing technique, was adopted. Recognition accuracy is up to 85.3%, which is much higher than previous studies. The classification results were quite encouraging, and showed the feasibility of a user-independent emotion recognition based on physiological signals. We expect to develop a simple and accurate affect estimation system for machine to understand the users’ feelings. Future work will be aiming at two aspects: • Build a good database, which is most arduous, difficult, but foundational and significant as well. Since there is no standard database, there is no standard to evaluate the researchers’ work. A standard database is a matter of great urgency for the study. • Featrues extraction. For us, what emotion is, how it happens and how it reacts remain unknown till now, which makes it very difficult to select the most corresponding physiological features for classification. Great foundational work should be done to explore the relationship between emotion and the physiological signals. It is key of the emotion recognition. In a word, it is feasible to classify emotion with physiological signals. But before emotion interpretation can occur at the level of human abilities, there still remains much work to be done. Acknowledgement. The work is supported by Academy Natural Science Foundation of Jiangsu Province (04KJB310171), and Advanced Technologist Research Start Foundation of Jiangsu University (05JDG029).

446


References 1. Picard, R. W.: Affective Computing, MIT press, Cambridge, MA, (1997) 2. Andreassi, J. L.: Psychophysiology: human behavior and physiological response, Lawrence Erlbaum Associates, New Jersey, (2000) 3. Ark, W., Dryer, D. C., and Lu, D. J.: The emotion mouse, 8th Int. Conf. Human-computer Interaction, (1999) 453-458 4. Picard, R. W., Vyzas, E., Healey, J.: Toward machine emotional intelligence: analysis of affective physiological state. IEEE Transactions Pattern Analysis and Machine Intelligence, (2001), 23(10) 1175-1191 5. Kim, K.H., Bang, S.W., and et al.: Emotion recognition system using short-term monitoring of physiological signals, Med. Biol. Comput., (2004) 42: 419-427 6. Gross, J.J., Levenson, R.W.: Emotional suppression: Physiology, self-report, and expressive behavior, Journal of Personality & Social psychology, (1993) 64: 970-986 7. Shusterman, V., Barnea, O.: Analysis of skin-temparature variability compared to variability of blood pressure and heart rate, IEEE Ann. Conf. Engineering Medicine Biology Society, (1995) 1027-1028 8. Berntson, G.G., Cacippo, J.T., et al.: Autonomic determinism: the modes of autonomic control, the doctrine of autonomic space, and the laws of autonomic constraint, Psychological Review, (1991) 98: 459~487 9. Mccraty, R., Atkinson, M., et al.: The effects of emotions on short –term power spectrum analysis of heart rate variability, AM. J. Cardiol., (1995) 76:1089-1093 10. Grossman, P., Wientjes, C. J.: How breathing adjusts to mental and physical demands, Respiration and emotion, spring, New York, (2001) 43-53 11. Ritz, J., Nixon, A., et al.: Airway response of healthy individuals to affective picture series, International Journal of Psychophysiology, (2002) 46(1): 67-75 12. Hotelling H.: Relations between two sets of variates, Biometrika, (1936), 28: 321-377

Animation Preserving Simplification for MPEG Based Scalable Virtual Human HyungSeok Kim Department of Internet & Multimedia Engineering, Konkuk University 1, Hwayang-dong, Kwangjin-Ku, Seoul, Korea [email protected] http://home.konkuk.ac.kr/∼ hyuskim

Abstract. In this paper, a new method to build an adaptable body shape considering the motion parameters. The proposed method constructs MPEG conforming scalable virtual human by simplifying the body parts which have similar motion parameters while preserving details with characteristic motion parameters. Characteristic motion is classified by its high deformation over time which is extracted from skinning parameter distribution. The proposed method is adaptable with in MPEG framework and is designed to minimize load of adaptation server which keeping compactness as much as possible. It is also targeted to cover devices with small computational power such as PDAs. Example of body simplification is given to show the effectiveness of the proposed method. Keywords: Multiresolution model, Mesh simplification, Virtual human.

1

Introduction

The virtual human is one of key elements in virtual reality applications including interactive game, virtual fashion, and virtual heritage. The virtual human is applied to give more realism in applications which requires high quality presentation of shape and animation. To give high quality rendering and simulation, the shape of a virtual character is usually too complex to be processed in realtime. For example, laser scanned body has more than millions of polygons, while the model itself need to be animated in conventional PCs or even at light-weight mobile phones. Thus, those complex representation of virtual human should be adapted to the requirements of real-time rendering and simulation. MPEG-21 is a standard for managing multimedia contents including 3D shapes and animations. In this paper, a method to generate MPEG-21 based animatable virtual human is presented. With respect to regular solid objects in a virtual environment, the most important feature of the virtual character is its animation. The surface of a virtual character is deformed by its facial and body animation. In this paper, both the shape and animation representation of the virtual character is considered to generate real-time renderable representation. Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 447–456, 2006. c Springer-Verlag Berlin Heidelberg 2006

448

H. Kim

There have been many approaches to generate simplified shapes for realtime rendering since the first introduction of multiresolution modeling [2]. Those methods evaluates representations using error metrics with respect to the original models. Most common error metrics are based on geometric and color space distances for the static shape. For the polygon mesh, vertices/edges/faces are removed to generate simplified shape while minimizing errors [6,5,3]. An edge collapsing is a widely adopted simplification operation by merging two vertices into one vertex [7]. In addition to preserve static shape features, dynamic features such as view and light silhouettes are also considered. Hoppe et. al proposed vertex tree which can easily accommodate view silhouettes [7,12,11]. Kim et al. proposed a structure to organize directional relations to be accessed in real-time considering both view and light silhouettes [10]. Although existing simplification methods give good results, applying these methods to animatable object with deformation still has limitations. The animation of the virtual character consists of a facial animation and of a body animation. The simplification process to generate an LOD model should consider these animation parameters so that the low complexity model can be animated as close as possible to the original model. Kim et. al proposed MPEG-21 based facial animation parameter preserving multiresolution method [8]. In this paper, we extend the method for the body animation. Body animation can be represented in joint-based movement of solid segment or skinning style deformation. For body animation, it is important to preserve surface detail around deforming parts such as surface parts near joints. For example, a surface part which is close to a joint should have more detail, to preserve overall shapes even when the part is highly deformed by joint rotation [9]. DeCoro et al. proposed a pose-independent simplification method to simplify a shape considering animation parameters [4]. Their method minimizes errors for different body poses in simplification. Importance of each pose can be modeled such that most frequent pose will have less errors. Although the method give high quality simplification, it needs an extension for multiresolution modeling [4]. Also, it does not effectively consider different level-of-articulation mentioned in MPEG-4/21 LOA scheme [9]. In this paper, a multiresolution extension of pose-independent simplification is presented with LOA model. Ahn et. al proposed a method to simplify skeleton structures based on analysis of pre-recorded motions [1]. Each motion is analyzed by its overall shape and a hierarchy of skeleton structure is generated such that the least number of joint is animated along to its motion complexity and rendering complexity. In this paper, this approach is adopted to create LOA and a link to continuous multiresolution model. In this paper, a new method to build an adaptable body shape considering the motion parameters. The proposed method simplifies the body parts which have similar motion parameters while preserving details with characteristic motion parameters. This paper consists of four parts. In the next section, an overview of MPEG-21 based adaptation and discussion on motion consideration is presented. Section 3

Animation Preserving Simplification for MPEG

449

presents proposed simplification method and structure. An experimental result is followed in the next section with a few simplification cases.

2

Scalable 3D Model in MPEG Framework

Adaptation within the MPEG framework for the 3D shapes, Kim et al. proposed a generic framework for scalable virtual human [8].

Fig. 1. MPEG-21 Adaptation Engine [8]

The adaptation process of MPEG is basically passing information through the adaptation engine. As illustrated in Figure 1, 3D contents are adapted in the adaptation engine by given description of the binary stream and adaptation information. The structure description should be retained even after the simplification thus the adaptation methods must be inline with the lowest level defined in the schema [8] . To make a scalable virtual human in MPEG framework, the model should be constructed in multiresolution manner so that arbitrary detail resolutions can be created by MPEG-21 adaptation mechanism. Due to the nature of MPEG-21 based adaptation, the multiresolution model for the MPEG-21 is required to satisfy following requirements. – Compact representation The multiresolution model should be able to be encoded by standard stream formats such as MPEG-4 BIFS or LASER, etc. Also it is desired that the encoded representation is in compact form. – Binary adaptable The standard MPEG-21 adaptation is performed for the binary encoded contents. Instead of letting the adaptation server to decode and re-encode for the process, the binary adaptable data makes the server be free of additional heavy computation. – Heterogeneous terminal devices In common mobile environment, highly heterogeneous devices participate in the network. For the least capable devices, which has very limited memory size, memory access bandwidth and computation power, a simple decoding scheme of the adapted contents should be devised to conform to the limitation of memory and computation power in those devices.

450

3 3.1

H. Kim

Simplification Considering Body Motion Clustered Multiresolution Representation

The shape of virtual human is optimized by mesh simplification methods. In this research we adopt a simple decimation method based on quadric error [5]. For a multi-resolution model, we adopt and extended the concept of clustering representation [9]. In this subsection, a summary and extension of Kim et al.’s approach is illustrated. The multiresolution model is clustered so that a specific complexity can be obtained by simply choosing a set of clusters. From the complex mesh Mn (Vn , Fn ) where Vn is a set of vertices and Fn is a set of faces, it is sequentially simplified to Mn−1 , , M1 , M0 . The multiresolution mesh M is presented by V and F which are the set of vertices and faces over all levels [9]. V and F can be partitioned into set of clusters. The first type is a set of vertices and faces that are removed from a mesh of the level i to make a mesh of the level i − 1, denoted by C(i). The other type is a set of vertices and faces that are newly generated by simplification, denoted by N (i). The simplification operator is sequentially applied to generate multiresolution model. In each simplification step, corresponding C(i) and N (i) can be generated. For more efficient adaptation, C(i) is again clustered into C(i, j) for corresponding N (j) such that C(i, j) is a part of C(i) which is subset of N (j) where j > i and C(i, i) which does not belongs to any N (j). Hence a level i mesh is as follows [9]: Mi =

i j=0

(C(j, j) +

n

C(j, k))

k=i+1

In the previous work, a half-edge collapsing operator is used with modified QEM considering relation to joint animation [9]. To have better result DeCoro’s pose-independent quadric can be used [4]. The focus of this work is not in the presenting better error measurement but presenting an integrated method to create efficient clustered representation. The last process is the concatenation of clusters into a small number of blocks to reduce the number of selection or removal operations during the adaptation process. By ordering vertices of C(i, i) by the order of i, the adaptation process of vertex data for level i is a single selection of continuous block of data, v0 , v1 , , vi . For the indexed face set, each C(i) is ordered by C(i, j) in the ascending order of j. Thus, an adaptation to level i, consists of at most 3i + 1 selections or at most 2n removals of concatenated blocks [9]. 3.2

Cluster with Animation Hierarchy

To make real-time presentable human body, it is needed to make plausible simplification over different postures give by motions. The previous work on scalable virtual human in MPEG-21 framework does not effectively consider the differences given by motion [9]. So that, the mesh simplification does not considers


451

effects caused by different LOA by constantly preserving unmoving joint details. Also, LOA architecture presented in their work does not consider the skinning parameter so that only segmented body animation is supported while the skinning based body animation is quite popular for the higher quality animation. The method presented in this paper is extension of fore-mentioned motion independent simplification. Instead of covering all motions, only the necessary motion part is considered. For example, when a virtual human is placed in distance so that only knee movement is significant for walking motion, it is not necessary to preserve motion parameters for other parts of body. Ahn et. al. presented an interesting method to construct motion hierarchy so that only necessary joints and motion parameters are used in crowd animation [1]. It is assumed that the motion is created from pre-captured set of motions. It is blended and re-targeted for each virtual human. Following Ahn’s method, a generic motion hierarchy is constructed for all motions. In this framework, it is assumed that every motion will be appeared with the same probability but it could be easily extended to put different weight for each clips. The most simplified motion parameter is given as a static figure considering that no motion is given to all joints. The standard posture is calculated from average position from the given set of motions. Starting from the static figure, it is sequentially refined by adding joints by joints according to given set of motions. Figure 2 shows an example of motion hierarchy used in this paper. In the hierarchy, each bone in coarse level is refined into a set of bones (lines in the figure) by adding corresponding joints (dots in the figure).

Level of Articulation: Coarse to Detail

Fig. 2. Example of Level of Articulation: Motion Hierarchy

The motion hierarchy is a structure of bones such that the simplest model has only one bone b0 . For each refinement bi is refined into bj and bk such that bi+1 is always higher or the same detail level of bi . Once the bone hierarchy is constructed, polygon clusters are assigned to the bone structure. At first, each polygon is assigned to the highest level of bones. Polygons around joints which have relations to more than one bones are assigned most relevant one according to skinning weight. Cluster C(i, j, k) is polygons in the cluster C(i, j) related to bone bk . This process is continued until the shape simplification error exceeds the motion simplification error using maximum QED and maximum positional error estimation of Ahn’s work.

452

H. Kim

By the ith simplification which exceeds the error calculated by motions of bone level l with bk , the bk is merged into bm along with adjacent bones following the pre-constructed hierarchy. From this level, simplified polygons are allocated in C(i, j, m) instead of C(i, j, k) and it is continued until the b0 . Following this procedure, M is again clustered into C(i, j, k) by bones such that part of polygons from Msl to Mel where sl < el are assigned to bk . The previous equation is extended with bone structures. Mi =

i

(C(j, j, l) +

j=0

n

C(j, k, l))

k=i+1

where bl is the selected bone level. M0 LOA0 M0 … M2 meshes are generated using LOA0 bone structure

LOA1 M3 … M5 meshes are generated using LOA1 bone structure

. . . M3 . . . M6



LOA4

. . . M9 . . . M12 M13

M12 … M14 meshes are generated using LOA4 bone structure

M14

Fig. 3. Example of Level of Articulation: Motion Hierarchy

Figure 3 is an example of combined LOA and shape levels. Each shape levels are constructed with using the corresponding bone structures. In addition to the vertices and face information, additional vertex properties such as normal, color, and texture coordinates are also clustered by C(i, j, k). The clustering method is basically literal adoption of Kim et al.’s method [9] as those are not related bone levels but only to polygon levels. Only the skinning information need to be reconstructed for different bone levels. As the cluster of skinning parameter resides on each bones, similar method can be applied to the skinning parameter as the vertex. When C(j, k, l) is simplified C(j −1, k, m), newly created skinning parameters belongs to C(j −1, k, m) is just


453

added by the end of previous clusters so that corresponding skinning parameters can be selected. Skinning parameter for the simplified bones are reconstructed by average of previous parameters so that overall motion remain unchanged. For exceptional cases, mapping from the vertex to faces is considered for clusters as for the normal, color or texture coordinates. 3.3

Adaptation Using Clusters

In this work, an extended mechanism for scalable virtual human is presented which considers skinning-based animation and LOA for the shape simplification. The constructed scalable representation contains scalable shape and body animation data. For body animation parts, the shape is mapped onto different LOA such that a single level of articulation representation to have range of shape details. It is processed in the adaptation server which selects the shape complexity by the function of desired level of articulation and shape complexity. In the adaptation server, at first, the desired shape complexity is selected. The device contexts such as rendering power acts as constraints for the shape representation. The selected representation is the most complex representation within the complexity bound for the desired context. Every representation with less complexity than the selected one conforms to the context constraints. After the selection of shape complexity bound, LOA complexity bound is selected. LOA complexity bound is selected by constraints on terminal device capability for animation computation along with view parameters. Following the clusters, each level of shape has unique correspondence to the LOA. If the corresponding LOA of the selected shape complexity is the same or less level than the desired, it makes both animation and shape complexity satisfies the condition of device. If the corresponding LOA is higher than the desired, the most highest complexity shape for the desired LOA is selected. M0 . . . M3

LOA0

. . .

LOA1

Selected LOA

Shape (polygon) budget

LOA2

LOA4

Selected Mesh Level

. . . LOA budget

LOA3

M6

M9 . . . M12 M13 M14

Fig. 4. Adaptation Example with motion and polygon hierarchy

454

H. Kim

Figure 4 illustrates the selection example. From the given LOA and polygon budget described in device context, the shape and level of articulation is selected by finding the best matching correspondence while keeping everything under the budget. Selection of level is selection of clusters for the shape level. Each cluster has a set of vertices and vertex properties such as vertex normal, colors, texture coordinates and skinning parameters. Along with vertex information, the cluster has a set of indexed faces, normal faces, color faces and texture faces. Also each cluster can consist of sub-segments with their own material and texture. Each level is selected by choosing blocks of clusters.

4

Experiments

The proposed method constructs different levels of detail to be adapted in MPEG framework. The strong point of propose method lies on its conformity with standard and its compactness. In terms of compactness, the progressive mesh (PM) approach is known as a method using near an optimal storage for multi-resolution models. In other extreme, the discrete mesh (DM) is a representation of a set of discrete levels of details which is still quite common in most of real-world applications because of its simplicity in adaptation. Table 1. Size of Multiresolution Model with Different Methods Number of polygons 71k

Original Proposed PM DM ˜ 16.9M / 5.9M 25.8M / 9.7M 18M 132.5M / 29.2M

The table 1 illustrates size of multiresolution model for different methods. The numbers are the size of the VRML and BIFS files respectively. PM stands for progressive mesh representation and DM is for discrete levels of detail. Since the PM cannot currently be encoded in the BIFS format, only the approximated size for the VRML file is noted. As a result of the adaptation, the highest details have a number of polygons of 71K and 7K for each model whilst the lowest details have 1K and 552 polygons each. The models are constructed to have 5 different levels in LOA and 3 shape levels in each LOA. Compared to original or PM model, the proposed model has around 50% more data in both VRML and BIFS representation but it is about 20% of full discrete model. The number of operations for adaptation is at most selecting 3 clusters for each bones and vertex properties, which is at maximum 270 clusters for 15 bones and 6 properties (position, color, normal, texture coordinate, skinning parameters and polygons). It is quite small when compared to operations for a PM, which requires at least n/2 operations, where n is number of vertices (28K for the body model). Furthermore, the method uses a simple selection, whilst PM requires relatively


455

complex substitutions. The proposed method is located in-between of these approaches, and is flexible and simple enough to allow adaptation with relatively small file size and encodable to standard codec and is able to be transmittable via standard MPEG streams. It also utilizes a simpler adaptation mechanism, which is very similar to the simplest discrete level selection.

5

Conclusion

In this paper an extended framework to construct multiresolution 3D virtual human model that can be encoded and adapted using standard MPEG-21 framework. The method is extended from the previous work such that it effectively incorporates body motion hierarchy to preserve more details over animation. It inherits strong points of previous approaches most importantly adaptable with no additional information at the client and can be rendered immediately. In addition, the representation used is suitable for clients with low computational power and provides significant advantages over progressive mesh techniques of devices such as PDAs. In this work, each mesh representation corresponds to the single LOA in order to make compact representation of skinning parameters. It makes the control of LOA is strongly constrained by the given shape complexity. To make more flexible adaptation for various cases, it is necessary to be able to control both complexities independently. Further more, devising unified error metric for both motion simplification and shape simplification will help to construct more efficient representation of multiresolution model.

Acknowledgements This work was supported by the faculty research fund of Konkuk University in 2006.

References 1. Junghyun Ahn, Seungwoo Oh, and Kwangyun Wohn. Optimized motion simplification for crowd animation: Research articles. Comput. Animat. Virtual Worlds, 17(3‐4):155–165, 2006. 2. James H. Clark. Hierarchical geometric models for visible surface algorithms. Commun. ACM, 19(10):547–554, 1976. 3. Jonathan Cohen, Amitabh Varshney, Dinesh Manocha, Greg Turk, Hans Weber, Pankaj Agarwal, Frederick Brooks, and William Wright. Simplification envelopes. In SIGGRAPH ’96 Proc., pages 119–128, Aug. 1996. 4. Christopher DeCoro and Szymon Rusinkiewicz. Pose-independent simplification of articulated meshes. In SI3D ’05: Proceedings of the 2005 symposium on Interactive 3D graphics and games, pages 17–24, New York, NY, USA, 2005. ACM Press. 5. Michael Garland and Paul S. Heckbert. Surface simplification using quadric error metrics. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques, pages 209–216. ACM Press/Addison-Wesley Publishing Co., 1997.

456

H. Kim

6. Hugues Hoppe. Progressive meshes. In SIGGRAPH ’96 Proc., pages 99–108, Aug. 1996. 7. Hugues Hoppe. View-dependent refinement of progressive meshes. In SIGGRAPH ’97: Proceedings of the 24th annual conference on Computer graphics and interactive techniques, pages 189–198, New York, NY, USA, 1997. ACM Press/AddisonWesley Publishing Co. 8. HyungSeok Kim, Chris Joslin, Thoman Di Giacomo, Stephane Garchery, and Nadia Magnenat-Thalmann. Multi-resolution meshes for multiple target, single content adaptation within the mpeg-21 framework. In Proceedings of the IEEE ICME Conference, pages 1699–1702, 2004. 9. HyungSeok Kim, Chris Joslin, Thomas Di Giacomo, Stephane Garchery, and Nadia Magnenat-Thalmann. Device-based decision-making for adaptation of threedimensional content. The Visual Computer, 22(5):332–345, 2006. 10. HyungSeok Kim, Soon Ki Jung, and KwangYun Wohn. A multiresolution control method using view directional feature. In VRST ’98: Proceedings of the ACM symposium on Virtual reality software and technology, pages 163–169, New York, NY, USA, 1998. ACM Press. 11. David Luebke and Carl Erikson. View-dependent simplification of arbitrary polygonal environments. In SIGGRAPH ’97: Proceedings of the 24th annual conference on Computer graphics and interactive techniques, pages 199–208, New York, NY, USA, 1997. ACM Press/Addison-Wesley Publishing Co. 12. Julie C. Xia and Amitabh Varshney. Dynamic view-dependent simplification for polygonal models. In VIS ’96: Proceedings of the 7th conference on Visualization ’96, pages 327–ff., Los Alamitos, CA, USA, 1996. IEEE Computer Society Press.

A Virtual Reality Training System for Robot Assisted Neurosurgery Da Liu and Tianmiao Wang Robotics Institute, Beijing University of Aeronautics and Astronautics, 100083 Beijing, China [email protected]

Abstract. Surgical training systems based on virtual reality simulation techniques offer a cost-effective and efficient alternative to traditional training methods. This paper describes a virtual reality system for training robot assisted neurosurgery. The training system undertakes the task of building the 3D environment of the virtual operation room and the surgical robot. The 3D model of the patient’s brain is reconstructed based on the 2D images. After the virtual robot move to the entrance pose that is planned by the surgeon, the user can control the virtual needle to insert into the virtual brain to verify his surgical plan. 3-D display device and home-made humancomputer interactive devices increase the immersion and telepresence of this system and establish a robot assisted surgical training demo system oriented clinical application. Keywords: Robot assisted neurosurgery, Training, Virtual Reality, Humancomputer interactive device.

1 Introduction Robot assisted surgery is rapidly becoming an active interdisciplinary field, which has already begun to develop potential practical applications in minimally invasive neurosurgery [1]. Compared to conventional surgery, robot assisted neurosurgery offers less trauma and quicker patient convalescence. However, in robot assisted neurosurgery, the restricted vision and limited mobility of surgical instrument can easily cause unexpected injuries to patients. Virtual reality training systems provide an elegant solution to these problems. The advantage of using a VR-based simulator stems from the ability to repeat procedures as many times as needed, without hurting the patient. As opposed to cadaver training, the surgeon’s actions and outcomes are sampled in real-time and analyzed by the computer running the simulation. At the same time, VR-based simulators allow surgeons to practice new and unusual surgical procedure [6]. So many training systems have been developed [2-5], but these systems are not tailored for robot assisted neurosurgery. In this paper, a training system for robot assisted neurosurgery is presented. This system can not only greatly improve the training and education quality to doctors but Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 457 – 466, 2006. © Springer-Verlag Berlin Heidelberg 2006

458

D. Liu and T. Wang

also increase the understanding for doctors to the robot assisted neurosurgery, which is beneficial to the technique spreading.

2 System Architecture 2.1 Overview of the System In the real robot assisted neurosurgery, the procedure starts with fixing four markers on the skin in the skull of the patient. When medical images are input to the system, the surgeons identify them in the model of the brain. Then the robot is used to touch the pre-defined four markers on the patient’s head, the system get the tip location in the robot local reference system. With the visualization of the brain model, the surgeon can define the route of the incision to avoid puncturing the critical vessel or nerves. Thus we use the robot to locate at the pre-defined incisive site with correct orientation. Surgeons can use the probe installed on the robot to implement the operation. The training system is established on a PC using WTK software. It provides the surgeon with a virtual brain, virtual robot and virtual operation room. The structure of the system is shown as Fig. 1. A surgical intervention is complicated and requires several activities. The CT/MRI images of the patient’s head are input into the system as the original data, and they are segmented to reconstruct the three-dimensional model. The surgeon can make a pre-operative planning to decide the entrance position and direction. The motion of the virtual robot with 5 degree of freedom is controlled by the surgeon. The surgeon can interface with the virtual environment, such as revising the puncture path of surgery (Fig. 2).

Fig. 1. The structure of the training system

A Virtual Reality Training System for Robot Assisted Neurosurgery

459

Fig. 2. The virtual training environment

2.2 The Hardware Structure of the System Considering the practicability of this system, we adopted not the distributed multicomputer system but single computer with human-computer interactive devices. In the meanwhile, in order to improve presence sense, interaction and autonomy， the system integrates many interaction instruments. The main property of this structure is: to add a data processing model between the interaction instrument and the center computer, taking the corresponding algorithm to process the collected data and then send the data to the center computer. In contrary, the output data of the center computer are also sent to the interaction instrument for processing. That can decrease as much as possible the computation of the center computer and increase the simulation plotting speed of the system. The hardware structure of this system is shown in Fig. 3.

Fig. 3. The hardware structure of the system

This system consists of the several parts: the control model of the robot arm and data glove, HMD（ Head Mounted Display） , and transmitter of Fastrak. This system fixes

460

D. Liu and T. Wang

the two receivers to HMD and data glove to implement the visual feedback and bearing tracking. 2.3 The Software Structure of the System In our system, we use WTK (World Tool Kit) to construct our virtual environment. WTK can be used in WINDOIWS platform, so it provides the possibility of carrying out graphics simulation in the PC with cheap hardware. The software structure consists of database, models of program state setting, sensor data reading, global Begin

Models of program state are set

Database

Devices and scene objects are initialized To enter simulation loop Sensors are read

The universe's action funtion is called Objects are updated with sensor input

Objects perform tasks

Paths in record or playback mode are stepped

The universe is rendered To exit simulation loop End

Fig. 4. Flow chart of the system


461

function performing, simulation object updating, assignment prosecuting and path processing. In all of them, the models of database, object updating (according to the sensor data) and assignment prosecuting are the three parts that affect the program efficiency and simulation effect most. In order to improve the program efficiency, we utilize the scene graph to manage the scene object and introduce switch node to optimize scene tree. The scene display takes the Level of Detail (LOD) technique. Thus can promise the display effect and avoid the waste of system resource. Flow Chart of the system is shown in Fig. 4. All of the current elements of the scene, such as geometry, lights, fog, and positional information are hold by the scene graph. The scene graph is an ordered collection of nodes, in the form of a directed acyclic graph, which holds hierarchical scene data. The scene graph provides a very powerful scene structure for real-time 3D simulation and enhances performance of the WTK’s rendering stage (drawing the scene) because it facilitates spatial culling of the scene. During model plotting, all new view sub-domains based volume-rendering algorithm are presented. Taking advantage of the separability of the view domain and the equation of color composition, this algorithm separates the whole data set in view domain into several smaller data set in view sub-domains. Thus the rendering task can be implemented by combining the rendering results of all view sub-domains. Because the size and the complexity of the data set in view sub-domain are reduced, the rendering of the data set can be accelerated by the hardware of the normal workstations and desktop machines. This algorithm can also implement the rendering of the hybrid model. Based on the view sub-domain volume-rendering algorithm, a new fast volumerendering algorithm for walk through is proposed. This algorithm takes advantage of the correlation of the texture images between two adjacent frames. Some information in the last texture images can be used in the new texture images for rendering. Thus, this algorithm can reduce much calculation time and improve the speed of rendering algorithm.

3 Human-Computer Interactive Device As the human-computer interactive device, robot arm and data glove are used in our system. So far The Robotics Institute of BUAA has researched a type of robot arm and two types of data glove. They are BH1 robot arm, BHG-I data glove and BHG-II data glove. To enhance the ability and let operators feel more visual and realistic, we use WTK to produce the visual environment, connecting the system with robot arm and data glove. The arm and the glove can control the training system. In this section, we integrate BH1 robot arm and BHG-II data glove as the VR equipment. 3.1 BH1 6-DOF Robot Arm BH1 6-DOF robot arm is used as human-computer interaction device, measuring the location of space point (Fig. 5).

462

D. Liu and T. Wang

We use none-power method to measure the point’s location. It looks like human’s arm, which is composed of forearm and elbow. The 2-DOF of the hand and the rotating joint of the forearm ensure the arm can get to the target in different status. The maximum error of the arm is 1.83mm, the average error is 1.28mm, covariance is 0.658mm.

Fig. 5. BH1 robot arm overview

3.2 BHG-

Ⅱ5 Finger 11-DOF Data Glove

BHG-II data glove is designed for VR programming. It is constructed with mechanical main body, A/D data collecting card and simulation software, suiting for different human’s hand. It detects the micro movement of five fingers, and use graphics simulation software to realize human-computer interaction at the same time. The range of the joint movement that it can detect is 20°- 90°, the average resolution ratio is 0.49°, the average excursion is 0.045v(4.8°). Fig.6 is BHG-II glove sensor distribution. BHG-II glove is cheap and reliable, which can fit for the requirement of simulation. The connection of arm and glove with the training system depends on RS232 communication, using SCM to control the data collection and treatment under Windows NT platform.

Ⅱdata glove

Fig. 6. BHG-


463

4 The Virtual Three-Dimensional Model Reconstructed 4.1 Segmentation and Surface Mesh Generation We generate surface meshes to represent organs, such as bone and pathological tissue. When the original image data are input into the system, a semiautomatic seed-based method is used to obtain a two-dimensional (2-D) contour from each slice. From the result of segmentation on a series of CT/MRI images, surface mesh is created from the series of 2-D contours using a three-dimensional (3-D) reconstruction algorithm. The algorithm is to tile the triangles from planar contours [7]. All contours are tiled with their adjacent contours by the triangles. The volumetric model of brain is reconstructed by linear interpolation between CT/MRI image slices. 4.2 The Dissection of 3-D Brain Model Through the dissection of the brain model, the change of the inner structure of the brain model will shown to surgeons during the needle insertion. When the insertion begins and collision between the needle tip and the model is detected, a judgment about which triangle faces should be hidden will be made automatically. The origin O of the XYZ coordinate is the needle tip. The dissection planes are designed as XOY plane, XOZ plane and YOZ plane. At each sampling time, the dissection plane intersects with the triangle faces. There are eight dissection directions according the pose of needle. The triangle faces that out of the dissection area are not hidden, otherwise they are hidden.

5 Implementation The robot arm gives the surgeon a possibility to use surgical instrument to perform a simulation on the virtual environment. For example, the surgeon can plan the surgery path by operating on the robot arm. When the tip of the arm is punctured into the virtual brain model, the model cut automatically according to the depth of puncture. The doctor can observe the adjacent tissues around the tip of the arm in the interior of the brain and analyze if the path is correct. For detecting collisions of tip and tissues, the model of the Denavit-Hartenbergmatrices needs to be set up. With the help of the Denavit-Hartenberg-matrices, which respect the geometry of the robot arm, the cartesian coordinates are computed. The robot arm is a serial structure. Using the Denavit Hartenberg convention configuration, the joint parameters are indicated in the table below. Table 1. The joint parameters of our robot arm Joint 1 2 3 4 5 6

θi θ1 θ2 θ3 θ4 θ5 θ6

αi -90o 0o -90 o 90 o -90 o 0o

ai(mm) 0 455 0 0 0 0

di (mm) 0 -55 55 388 55 179

464

D. Liu and T. Wang

Rotate around the x axis by an angleθi Translate along the x axis by a distance ai Rotate around the z axis by an angleαi Translate along the z axis by a distance di The convert’s coordinates from frame i-1 to frame i can be written using the following matrix: i −1

Ai = Rot ( Zi − 1,θi )Trans(0,0, di )Trans(ai ,0,0) Rot ( Xi − 1, ai )

(1)

The position and orientation of the tip with respect to the base is given by:

⎡ nx ox ax ⎢ny oy ay T6 = 0 A1 1 A2 2 A3 3 A4 4 A5 5 A6 = ⎢ ⎢ nz oz az ⎢ ⎣0 0 0

px ⎤ py ⎥⎥ pz ⎥ ⎥ 1⎦

(2)

The tip coordinate is ( Px, Py, Pz). Since the robot arm can be mapped to the model in the computer, when the surgeon moves the robot arm, the correct relative location can also be displayed on the screen. At the same time, because the robot arm has the same structure as the real surgical robot, it can help the surgeon to improve the ability of operating on the surgical robot. Another challenging part is the simulation of tissue behavior caused by the instrument constraint. During neurosurgery operation the surgeon inserts patient brain into probe and absorbs the tumor with injector. To simulate these procedures different approaches are implemented and tested. The tissue specific characteristics should be taken into account, the realism of the simulation should be as high as possible, the real time condition should be as high as possible and the real-time condition doesn’t be lost.

Fig. 7. Operating on the robot arm to simulate the surgery puncturing


465

For increasing the immersion and telepresence of the system, other virtual reality equipment, such as HMD and data glove is used in our system. The surgeon can wear HMD and data glove, selecting the operation instrument, roaming in and interfacing with the 3-D virtual environment (Fig. 7). We combine the data glove (BHG-2) and Fastrak to implement the puncturing simulation in the neurosurgery. The data glove is used to measure the joint value of the finger and Fastrak is used to measure the position and posture of the hand. The surgeon can wear the data glove, taking the probe to puncture the human brain in virtual environment. The probe in the virtual environment punctures into the brain along with the proceeding of surgeon’s hand and the brain is split along with the proceeding of probe. The surgeon can select other instruments to simulate the corresponding operations.

6 Conclusions In this paper, a structure of a virtual reality training system for robot assisted neurosurgery is proposed, setting up from the concrete practical background of neurosurgery. Combining 3-D display devices and home-made human-computer interactive devices, this system structure can also improve the speed of displaying interaction of 3-D operation model and establish a robot assisted surgical simulation and training demo system oriented clinical application. The training system represents an advanced training system incorporating VR and multimedia for training in robot assisted neurosurgery. The system not only simulates neurosurgical interventions as realistic as possible, but also tries to train robot assisted surgical skills in several levels. The tasks in level one are established for the training of hand-eye coordination. In the second level some diagnostic tasks with 3D models of the brain have to be performed. Using robot arm the surgeon navigates through the virtual sites identifying landmarks in some exemplary physiologic and pathologic cases. The third level of the system gives surgeons the possibility to use surgical instruments to perform a therapeutic intervention on the virtual model. Therefore not only a model of the surgical instruments has to be generated, but also the interaction of instruments and tissues have to be taken into account. Surgeons are able to practice various techniques without having to advance their learning curve on humans. These medical simulations are on the way to find an educational base that will be perhaps as important to surgery as flight simulators is to aviation. Acknowledgments. The authors gratefully acknowledge the support of this work by National Natural Science Foundation of China Project under contract 60525314. The authors also express cordial thanks to Prof. Zengmin Tian and Dr. Hongbo Lv who have also participated in the project.

References 1. Dario P.: Robotics for medical applications. IEEE Robotics & Automation Magazine, Vol.3. (1996) 44-56 2. Downes M., Cavusoglu M. C., Gantert W., at al.: Virtual environment for training critical skills in laparoscopic surgery. In: Medicine Meets Virtual Reality, (1998) 316–322

466

D. Liu and T. Wang

3. Y. Michiwaki, et al: Simulation training system for undergraduate students of dental school, Journal of Japanese Association for Dental Education, Vol.17, (2002) 311-320 4. C. Basdogan, C. H. Ha, M. A. Srinivasan: Virtual environments for medical training: graphical and haptic simulation of laparoscopic common bile duct exploration. IEEE ASME Transaction on Mechatronics, Vol. 6, No. 3. (2001) 269-285 5. G. Megali, O. Tonet, M. Mazzoni, at al.: A new tool for surgical training in knee arthroscopy. In: Proceedings of the Medical Image Computing and Computer-Assisted Intervention, vol. 2489. Tokyo Japan (2002)170–177 6. R. Satava: Virtual Reality Surgical Simulator-The First Steps. In: Proceedings of VR systems'93, New York, (1993) 41-49 7. D. Meyers, S. Skinner and K. Sloan: Surface from contours, ACM Transaction on Graphics, vol. 11, no.3. (1992) 228-258 8. Wang Z., Tang Z., Wang T.: VR based Computer Assisted Stereotactic Neurosurgery System. Jisuanji Xubao, Vol.23, (2000) 931-936

The Experiment of Improving Students’ Spatial Ability by Using VGLS Ruwei Yun1, Haixu Xi2, and Yi Li1 1

Educational Technology Department, College of Educational Science Nanjing Normal University, Nanjing, 210097, China [email protected] 2 Information Science & Media School, Jinggangshan College, 343009

Abstract. In order to compare the effect of VGLS (Virtual Geometry Learning System) with that of the traditional teaching method in improving students’ spatial ability, we selected two classes (total 106 students) in Jingqiao middle school as experimental class and control class, and pre-tested their basic level of spatial ability. The VGLS was applied in the experimental class for four weeks, while the traditional teaching method was applied in the control class. Some conclusions can be drawn from the result of the post-test: Compared with traditional teaching method, VGLS approach is more significant effective in the improving student’s mental folding, unfolding as well as rotation ability of students, without obvious superiority in improving student’s pattern recognition; VGLS is significant effective in improving females’ mental rotating ability and males’ mental folding and unfolding ability. Keywords: spatial ability, Virtual Geometry Learning System (VGLS), Virtual Reality.

1 Introduction Spatial ability in geometry has been identified as the imagination ability of shapes, structures, size as well as relation of location of objects [1]. Many researchers have used spatial ability to benchmark mathematics [2], science education [3], engineering [4], and so on. Primary education researchers have realized the importance of cultivating students’ spatial ability. At present, many teachers use some multimedia software produced by 2D animation tools, such as Flash, with the limitation in presenting three-dimension information. Some teachers would like to use model. With the limitation of time and space， it contributes little to students in grasping the concept geometry and related drawing skills. In order to compare the effect of Virtual Reality technology with that of traditional teaching method on improving students’ spatial ability, we specify six knowledge spots in the learning system, which are “the composition of basic geometry object”, “the section shape of basic geometry object”, “the motion change of basic geometry object”, “the three views maps of basic geometry object” and “the folding and unfolding of basic geometry object”. Then we designed the detailed instructional plan, built a VGLS and putted it to practice and experiment. Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 467 – 473, 2006. © Springer-Verlag Berlin Heidelberg 2006

468

R. Yun, H. Xi, and Y. Li

2 Methods 2.1 Participant Two classes with a total of 106 students from Jingqiao middle school of Nanjing volunteered to participate in this experiment. The experimental class includes 26 girls and 29 boys, while the control class 21 girls and 30 boys. 2.2 Test Material This paper uses the test material presenting in the dissertation of master (Zhen Zhou 2000) [5]. 2.3 Design Before this instructional experiment, all participant classes took a spatial ability test so that we could obtain a baseline and background information of their prior ability. In order to exclude the disturbance of different teaching experience and teaching plan from different teachers, the same teacher was consistent in teaching schedule in two classes. Some studies have documented that only after students have participated in spatial activity for at least three weeks, their basic spatial ability will significantly be improved [6]. After four consecutive weeks, all participants took a post-test with the same spatial ability test to determine any significant improvement. Internal consistency reliability has been computed for the pre-test and post-test and they revealed coefficient of reliability of .67 and .64, respectively. These values are seemed to be satisfactory since all values are above .60[7]. Since the same spatial ability test was adopted, we had to avoid the influence of pre-test on the post-test. The gap time between pre-test and post-test was about two or three months in order to avoid the influence of memory. In addition, during the pretest time, we played with students and then tested them with no cue. After the pre-test, we continue the play. In another day, we just told students that they did well in play and yesterday’s test. No any more information about the test was mentioned afterward. 2.4 Procedure In the experiment group, instruction includes teacher’s teaching and students’ selfstudying in the information technology classroom. Teacher explained the basic concepts by using the VGLS courseware. For example, in the explanation of “the composition of basic geometry”, teacher controlled the formation animation of column and tapered to summarize the situation of movement forming of point, line and surface. In the information technology classroom, students operated the courseware themselves, did exercises, finished creative activities designed in the VGLS. During the process, teacher tutored for students’ learning difficulties. In the control group, teacher explained the basic concept by using models and other instruction media. Students took part in activities such as designing models to review the knowledge in order to avoiding “extra teaching” in the experiment group.

The Experiment of Improving Students’ Spatial Ability by Using VGLS

469

Fig. 1. Main page of VGLS

Fig. 2. The control and display of geometry folding and unfolding

Teachers designed some question situations for students’ operating exercises, which are analogy to those in VGLS. VGLS links the six topics in the main page (Fig.1). There are demo, explanations, and exercises in every topic (Fig.2). Students read the introduction of the topic and enter a topic to review or do exercise. After that, they can acquire feedback and suggestion from this system.

470

R. Yun, H. Xi, and Y. Li

3 Data Analysis Independence-Samples z-tests for mental folding, unfolding, mental rotation and pattern recognition for training groups revealed no significant group differences at p>0.05, t-tests for males and females revealed no significant gender differences at p>0.05. These initial results revealed that both groups and genders were indeed equivalent at the outset. The means and standard deviations of scores across the two groups are demonstrated in Table 1 and Table 2. Table 1. Mental folding and unfolding accuracy and std. Deviation Mean(M) Conditions

Mental folding（ S）

Mental unfolding（ S）

Before

After

Before

After

Experimental

2.83（ 1.09）

3.65（ 0.92）

2.11（ 1.00）

2.95（ 0.75）

Males

3.03（ 0.93）

4.10（ 0.84）

2.21（ 1.00）

3.28（ 0.70）

Females

2.62（ 1.21）

3.20（ 1.20）

2.00（ 1.00）

2.62（ 0.90）

Control

2.84（ 0.85）

3.00（ 0.90）

2.29（ 0.94）

2.54（ 0.82）

Males

2.86（ 0.88）

3.10（ 0.95）

2.38（ 0.99）

2.61（ 1.00）

Females

2.81（ 0.79）

2.90（ 0.86）

2.19（ 0.85）

2.47（ 0.66）

Table 2. Mental rotation and pattern recognition accuracy and std. Deviation Mean(M) Conditions

Mental rotation（ S）

Pattern recognition（ S）

Before

After

Before

After

Experimental

1.71（ 0.97）

2.54（ 1.01）

3.56（ 1.58）

4.40（ 1.50）

Males

1.62（ 1.00）

2.52（ 0.80）

3.48（ 1.59）

4.01（ 1.65）

Females

1.81（ 0.92）

2.56（ 1.24）

3.65（ 1.57）

4.79（ 1.55）

Control

1.66（ 0.98）

1.88（ 1.24）

4.01（ 1.99）

4.26（ 1.84）

Males

1.83（ 0.97）

2.02（ 1.30）

4.07（ 2.21）

4.35（ 2.10）

Females

1.48（ 0.96）

1.74（ 1.20）

3.95（ 1.62）

4.17（ 1.68）

The Experiment of Improving Students’ Spatial Ability by Using VGLS

471

After the training was complete, there were significant group differences for mental folding at Z=3.68, p ws

(5)

where nR is the number of pixels in R . We denote ℜts as the union of the selected regions. Then we remove the background pixels from ℜts by using background spatial-color probability. A five-dimensional histogram is built on the domain that concatenates the pixel locations and the RGB colors using the background seeds in Bt. We define PB as the background probability induced by the histogram and the set of estimated object seeds p∈Ot is finally determined if

580

J.-H. Ahn et al.

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 4. Human silhouette extraction using the estimated trimap. (a) input left image, (b) cropped disparity map (c) object region likelihood LR, (d) region selection, (e) trimap; white, black, and gray pixels indicate Ot, Bt, Ut, correspondingly, (f) segmented object by the proposed graph cut.

− log PE ( p) > mCt + K C σ Ct

(6)

and p∈ℜt s where mtC and σtC are the mean and standard deviation of −log PB(p)’s of p∈ℜt s and the parameter KC is related to the level of confidence for the estimated object seeds. Figure 4(e) shows the estimated trimap with the input images in Figure 4(a) and Figure 4(b). 3.2 Graph Cut Framework with the Estimated Trimap

Given an image we try to find the labeling f that minimizes the energy[3] E ( f ) = Σ p D p ( f p ) + Σ{ p ,q}∈N δ ( f p , f q ) V p ,q ( f p , f q ) ,

(7)

where N is a standard 4-neighborhood system and δ( fp, fq) denotes the delta function defined by 1 if fp≠ fq and 0 otherwise. The smooth term Vp,q is defined by V p ,q = exp(− || z p − z q || 2 / β )

(8)

where β is chosen by the expectation of 2 exp(−||zp − zq||2/) over all {p, q}∈ N . The data term Dp measures how well label fp fits pixel p given the observed data zp. We model the object and background color likelihoods of P(⋅|1) and P(⋅|0) using Gaussian mixtures in the RGB color space, learned from image frames labeled from earlier in the sequence. In every frame we estimate a trimap that consists of O, B, and U. The set U =℘−(O ∪ B). Then Dp is defined as ⎧− log P ( z p | f p ) ⎪ D p ( f p ) = ⎨ ( K − c) f p + c ⎪ (c − K ) f + K p ⎩

if p ∈ U t if p ∈ O t , if p ∈ B

(9)

t

where K = max{p,q}∈N Vp,q(fp, fq) and c is a small number(usually accepted as 1 or 2). Minimization of the energy (3) is done by using a standard min-cut/max-flow algorithm[7]. The estimated trimap may have been wrong in some pixels. The role of c is to give the min-cut algorithm little chance to assign different labels from their pre-estimated labels of the trimap. This effect is shown in Figure 4. For the pixels in the estimated region, the data term Dp is determined by either of the constant K or c

An Integrated Robot Vision System for Multiple Human Tracking

581

no matter what the colors or disparities are. This is the reason why trimap estimation improves robustness to changes in illumination and camera movements. Figure 5 shows some human segmentation results for gesture recognition under the camera movements and illumination changes.

4 Experimental Results The proposed system was implemented in C/C++ and was run on Pentium IV-3.0 GHz PC with 1G RAM. The test videos were acquired with a the pan-tilt stereo camera and multiple people entered an indoor scene. Each video contained 700 frames of 320×240 images. The test video KC1 and SS1 were taken by a fixed stereo camera and one person entered the scene. The other five test videos were taken by the pan-tilt stereo cameras. In theses videos two or three people entered the scene, moved around, occluded with each other, and assumed natural poses. Excessive illumination changes occurred in the test video SY3. The average running time of the proposed system was about 9 fps. Performance Evaluation of the Detection and Tracking Algorithm: Performance was evaluated by using the ground-truth bounding boxes. We segmented the silhouettes of moving people manually at every 10th frame and determined their ground-truth bounding boxes. Table 1 shows the results. In the table Entrance refers to the total number of people entering the scene. Detected refers to the total number of detected objects, and True+D and False+D represent the number of correct detections and incorrect detections, respectively. If the detected bounding box overlapped by more than 90 percent with the ground-truth bounding box, we counted it as a correct detection. Tracking rate shows the average of the ratio of the number of correct tracking instances to that of the correctly detected tracks. If the center point of the tracked bounding box fell in the middle half area of the ground-truth bounding box we counted it as correct tracking. The average detection rate of the seven test video sets ws 98.31 percent. Table 1. Performance of the detection/tracking and silhouette extraction algorithms

Detection/ Tracking

Silhouette extraction

Entrance Detected True +D False +D Tracking Rate(%) Error rate(%)

CM2 4 4 4 0

CM3 6 8 6 2

JH3 9 10 9 1

KC1 1 1 1 0

KC3 10 10 10 0

SS1 1 1 1 0

SY3 7 7 7 0

98.96

99.63

98.83

99.71

99.88

99.91

91.72

1.06

2.55

1.25

1.36

1.26

0.61

0.42

Performance Evaluation of the Silhouette Extraction Algorithm: Performance was evaluated with respect to the ground-truth segmentation of every 10th frame in each of seven test stereo sequences. Each pixel of each ground truth image was labeled as object, background or unknown. The unknown label occurred when there

582

J.-H. Ahn et al.

were one plus pixel and one minus pixel along the object boundaries to mark the mixed pixels. Ignoring the mixed pixels, the errors were counted. Table 1 shows the averages of the errors rates within the bounding boxes. Some results obtained by the proposed vision system are shown in Figure 5.

Fig. 5. The results of human tracking and silhouette extraction under an active camera. The images are taken from CM2 and SY3.

5 Conclusions In this paper we presented a new integrated robot vision system for multiple human detection, tracking and silhouette extraction when using an active stereo camera. A prime application of this system in gesture, gait and event recognition when working with HRI systems. The background subtraction methods presented easy ways to detect and track people and extract their silhouettes but they were not applicable to mobile robot environments. The proposed system focused on robustness to camera movements and presented several novel vision techniques, such as a fast histogrambased tracking and graph cut segmentation with an estimated trimap, etc. The proposed system showed very good results when working with real sequences at near real-time speed. Further research will involve human identification since the detection module presently only detects moving objects. Mean shift segmentation takes about 60% of the processing time of the human silhouette extraction module so we are also developing faster image segmentation techniques.

An Integrated Robot Vision System for Multiple Human Tracking

583

Acknowledgement This work was supported by the Korea Science and Engineering Foundation (KOSEF) through the Biometrics Engineering Research Center (BERC) at Yonsei University. This research was supported by the Ministry of Information and Communication, Korea under the Information Technology Research Center support program supervised by the Institute of Information Technology Assessment, IITA-2005(C1090-0501-0019).

References 1. M. Bennewitz, W. Burgard, and S. Thrun: Using EM to learn motion behaviors of persons with mobile robots: Int. Conf. on Intelligent Robots and Systems (2002) 502-507. 2. Y. Boykov and M. Jolly.: Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images. Int. Conf. on Computer Vision, CD-ROM, 2001. 3. Y. Boykov and V. Kolmogorov.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision: IEEE PAMI.: 26. No. 9. (2004) 1124-1137. 4. R.T. Collins, A.J.Lipton, T.Kanade, H. Fujiyoshi, D. Duggins, Y. Tsin, D. Tolliver, N. Enomoto, O. Hasegawa, P. Burt and L. Wixson: A system for video surveillance and monitoring: CMU-RI-TR-00-12 (2000) 5. D. Comaniciu and P. Meer. Mean Shift: a robust approach toward feature space analysis. IEEE Trans. on Pattern Analysis and Machine Intelligence: Vol. 24. No. 5. (2002) 603619. 6. D. Comaniciu, and V. Ramesh.: Kernel-Based Object Tracking. IEEE Trans. Pattern Anal. Mach. Intell. Vol. 25. (2003) 564-577 7. A. Hampapur, L.M. Brown, J. Connell, M. Lu, H. Merkl, S. Pankanti, A.W. Senior, C.F. Shu, and Y.L. Tian.: Multi-scale tracking for smart video surveillance: IEEE Tran. on Signal Processing: Vol. 22. No. 2. (2005) 38-51. 8. I. Haritaoglu, D. Harwood, and L. S. Davis.: W4 :Real-time surveillance of people and their activities: IEEE Trans. Pattern Anal. Mach. Intell..: Vol. 22. No. 8. (2000) 809-830. 9. B.Jung and G.S.Sukhatme.: Detecting Moving Objects using a Single Camera on a Mobile Robot in an Outdoor Environment: Int. Conf. on Intell. Auto. Sys. (2004) 980–987. 10. K. Konolige.: Small Vision Systems: Hardware and Implementation: Eighth International Symposium on Robotics Research, (1997) 111-116. 11. C. Cruz-Neira, D.J. Sandin, and T.A. DeFanti: Surround-screen projection-based virtual Reality: The Design and Implementation of the CAVE: Computer Graphics ACM SIGGRAPH, (1993) 135-142 12. V. Pavlovic, R. Sharma, and T. Huang: Gestural interface to a visual computing environment for molecular biologists: Proc. Int’l Conf. Automatic Face and Gesture Recognition, (1996) 13. S. Shafer, J. Krumm, B. Brumitt, B. Meyers, M. Czewinski and D. Robbins: The new easyLiving project at Microsoft Research: Proc. Joint DARPA/NIST Smart Space Workshop (1998). 14. W. Kruger, C. A. Bohn, B. Frohlich, H. Schuth, W. Strauss and G. Weche: The responsive workbench: A virtual work environment: IEEE Computer, (1995) 28(7) 42- 48.

Disparity Weighted Histogram-Based Object Tracking for Mobile Robot Systems Cheolmin Choi, Jungho Ahn, Seungwon Lee, and Hyeran Byun Dept. of Computer Science, Yonsei University Shinchon-dong, Seodaemun-gu, Seoul 120-749, South Korea {wxman, jungho, swlee, hrbyun}@cs.yonsei.ac.kr

Abstract. A vision-based real-time human detection and tracking capability is one of the key components of surveillance systems, human computer interfaces and monitoring systems. In this paper, we propose a method which uses color and disparity information obtained with a stereo camera. In order to achieve optimal performance with respect to detection or tracking of objects, it is better to consider multiple features together. We have developed a tracking method in which color and disparity information can be combined in a histogram. We used skin color and disparity distribution information to distinguish between different people. For human tracking, we propose a color histogram that is weighted by the disparity distribution of the target. The proposed method is simple and robust for moving camera environments and overcomes the drawbacks of conventional color histogram-based tracking methods. Experimental results show the robustness of the proposed method in environments with changing backgrounds and the tracking capabilities of targets which have similar color distributions as backgrounds or other targets. The proposed method can be used in real-time mobile robot applications. Keywords: Real-time, color histogram, mobile robots.

1 Introduction Recently, there have been remarkable developments in mobile robot systems, which have enabled these robots to provide professional, industrial, and personal services. Personal care services include natural interaction with humans. Visual interpretation of human movements and gestures can be useful in accomplishing natural HumanRobot Interaction (HRI). Vision-based HRI systems have surveillance modules which automatically identify and track people, objects, or events of interest in different kinds of environments. To obtain better performance, it is necessary to consider the robustness and reliability of these surveillance modules. Moreover, intelligent robot systems must be capable of robust object detection and tracking, which are efficient ways of performing real time services. In this paper, we focus on a robust, simple detection and tracking system which can be used with intelligent humanoid robots. A vision-based human detection and tracking capability is one of the key components of surveillance systems. Generally, these surveillance systems consist of stationary cameras and a background subtraction module, which separates foreground Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 584 – 593, 2006. © Springer-Verlag Berlin Heidelberg 2006

Disparity Weighted Histogram-Based Object Tracking for Mobile Robot Systems

585

regions from background regions to detect humans. This background subtraction method is robust and simple in static camera environments [5], but it is hard to extract foreground regions when working with moving cameras (such as in mobile robot environments) because there is a large variation of pixels. There are other approaches which do not perform background subtraction. Typically, these approaches are based on skin color, motion, or disparity models. However, various limitations in robustness and accuracy arise when using only a single approach. For example, the use of skin color alone cannot detect a human body or the back part of a person’s head, and the use of motion alone makes it difficult to identify stationary humans with respect to camera motion. Thus, a robust human detection and tracking system must consider multiple approaches simultaneously. Integration of disparity and color was introduced by Yang et al. [1] and Darrell et al. [2]. Yang et al. built a map from disparity features and subsequently fused it with color and motion information. However, it was difficult to track occluded humans with this method and a static background was required to obtain proper motion information. Darrell et al. presented a multi-modal integration system for unknown environments which combined stereo, color, and face detection modules. However, the approach encountered difficulties in recovery when a person was occluded by another person. Also, it was difficult to implement the approach in moving camera environments. This is because the human detection module was highly dependent on foreground disparity segmentation and a sufficiently large face area was required to perform tracking. Several other approaches have been developed. These approaches include utilizing the statistical information of an object, the learning of scenes, and the evaluation of a hypothesis [3], [4] as well as creating a model of the target and candidates and then solving optimization problems in the feature space [7], [8], [9]. Recently, the mean shift-based object tracking method [9] was proposed, which used a color histogram of the object region and then a mean shift-based optimization method [10], [11], [12] was performed to find a similar local candidate. This method proved to be simple and robust for variations in color and camera environments. However, it often failed to track targets that were of similar color to the background or other targets. In the present study, we introduce a simple, robust detection and tracking method that uses color and disparity information. The proposed method can be used in moving camera environments such as mobile robot systems.

2 Overview A flowchart of the proposed method is shown in Fig. 1. A face detection module using color images obtained by the left camera operates in the region of interest and then a body detection module operates with the output of the face detection module and the morphologically-smoothed disparity image. After successful detection of the human body, output of the detection module is the target of the tracking module. The human tracking module performs tracking using the optimization method which is based on the disparity-weighted histogram.

586

C. Choi et al.

Stereo Camera Color Disparity

Detection Module Face Detection

Human Detection

Tracking Module Human Tracking

Occlusion Detection & Recovery

Fig. 1. Flowchart of the proposed method. The detection and tracking modules are classified into two separate sub-modules. The face detection sub-module uses only color information. The body detection and human tracking sub-modules use color and disparity information.

After a successful tracking procedure, an occluding event is checked using the distance between the targets and the similarity measure. The following section describes the detection module. Section 4 describes a module which performs tracking using a disparity-weighted color histogram that is weighted by the probability-based disparity of the target. Experimental results are analyzed in section 5 and section 6 describes the conclusions.

3 Detection Module One of the most commonly used features for human detection is the color of a human face [5], [6]. In the proposed detection module, we used color information first, and then we used heuristics and a simple geometric model for body detection. 3.1 Face Model For face detection, we used a face color model which was produced from four differently illuminated scenes of one person. We built a normalized red and green color histogram of the face model and then performed the mean shift algorithm on the region of interest. If a face was present, the position of the face-like area could be found using the mean shift algorithm and the threshold of histogram similarity. Fig. 2 shows the face color model (with its histogram equalized for visualization). Because of the quantization nature of the spatial color histogram, the face model is a sufficient way of finding different faces in a region of interest. The number of mean shift iterations for face detection can be small because the mode position of the face itself is not needed. In the experiment only three iterations were used.

Fig. 2. The face color model


587

3.2 Human Detection Using a face-like area coordinate and its disparity value, we can detect the body area by using the following steps. − − − −

Step 1: Find a point of the face-like region. Step 2: Find the horizontal head scale using horizontal edge points. Step 3: Find the vertical head scale using vertical edge points. Step 4: Detect a human body using the scale of the head.

In step 4, we can use heuristics which the disparity value of the body part shows a similar value. Fig. 3 shows the result of each detection step. The detected body area is the target for the subsequent frame.

h

w Face model (a)

(b)

(c)

(d)

Fig. 3. The human detection module. (a) Find a face-like region (search over the ROI using the face color model and mean shift tracking). (b) Find the scale of the head (the white circle means the detected point of a face-like region). (c) Detect the human head area. (d) Detect the human body area.

4 Tracking Module In this section, we propose a simple disparity-weighted histogram method which is based on a mean shift tracking algorithm. At each successful tracking procedure we retained the histogram similarity of the candidate target and the distance between targets for occlusion detection and the recovery sub-module. Mean shift based color tracking method is a simple and robust algorithm that climbs the gradient of a probability distribution to find the best local candidate. This method can be divided into three steps: target and candidate representation, similarity measurement, and optimization using the mean shift algorithm. Short descriptions of these steps are presented below. − Target and Candidate Representation PDF models of targets and candidates can be defined as in Comaniciu et al. [9] n

Target :

Candidate :

qˆ u = Cq ∑ k ⎛⎜ xi i =1 ⎝

2

nh ⎛ y − xi pˆ u = C p ∑ k ⎜ ⎜ h i =1 ⎝

⎞⎟δ [b( x ) − u ] i ⎠ 2

⎞ ⎟δ [b( xi ) − u ] ⎟ ⎠

(1)

(2)

588

C. Choi et al.

where k is a profile kernel for spatial weight, h is the kernel bandwidth with respect to the bandwidth of the target model, Cq and Cp are normalization factors, δ is the Kronecker delta function, and u is the index of the histogram bin. − Similarity Measurement In order to measure the similarity of two histograms, a metric-based Bhattacharyya coefficient can be utilized as follows:

ρˆ ( y ) =

m

∑

pˆ u ( y ) qˆ u

(3)

u =1

Using Taylor expansion, (3) can be reduced to: ρˆ u ≈

⎛ y − xi C nh 1 m pˆ u ( y0 )qû + p ∑ wi k ⎜ ∑ ⎜ h 2 u =1 2 i =1 ⎝ m

wi = ∑ δ [b( xi ) − u ] u =1

2

⎞ ⎟ ⎟ ⎠

(4)

qû pˆ u ( y0 )

(5)

where m is the number of the histogram bins, nh is the number of pixels that fall in a spatial kernel, and y0 is the previous target position. − Optimization Using the Mean Shift Algorithm The target tracking problem can be reduced so as to maximize similarity, which can be solved by the mean shift algorithm. A new target position, which is based on the previous position, can be calculated according to the following relationship. nh

New position :

yˆ 1 =

⎛ yˆ 0 − x i h ⎝

2

∑ x w g ⎜⎜ i =1

i

i

⎛ yˆ − x i wi g ⎜ 0 ∑ ⎜ h i =1 ⎝ nh

2

⎞ ⎟ ⎟ ⎠

⎞ ⎟ ⎟ ⎠

(6)

where, g ( x) = −k ' ( x) . 4.1 Human Tracking Using Disparity-Weighted Histogram The mean shift based color tracking approach has difficulties when separating objects with similar color distributions. We introduce a disparity-weighted color histogram method to overcome this weakness. The disparity obtained with a stereo camera is an important feature that can be used to separate different objects or foreground regions from background regions. We assume that the disparity distribution of the detected human body shows small variations with respect to the background regions or other objects. Frame-to-frame differences in disparity values are also small even if the scene is dynamic. In this section, we propose a simple disparity-weighted histogram which combines disparity information and color information. Let {vˆv }v=1...k and vM be the normalized histogram representation of the target area and its maximum probability. If we choose a target area that is sufficiently restricted in terms of size, it can be assumed that pixels that


589

fall in bin M, which has vM probability, are the most likely target pixels. Let {Wv }v=1...k be the weight value for the disparity histogram bin, which is defined as: ⎧ M −v⎫ ⎨Wv = 1 − ⎬ M ⎭ v =1... k ⎩

Disparity bin weights :

(7)

where v is the bin index of the histogram. A new target and candidate representation with these weights can then be computed as follows: n

New Target :

Qˆ u = C q ,w ∑ Wi ,v k ⎛⎜ xi ⎝ i =1

New Candidate :

nh ⎛ y − xi Pû = C p ,w ∑Wi ,v k ⎜ ⎜ h i =1 ⎝

2

⎞⎟δ [b( x ) − u ] i ⎠

(8)

⎞ ⎟δ [b( xi ) − u ] ⎟ ⎠

(9)

2

where Cq,w and Cp,w are the normalization factors for the target and candidate values, respectively, and Wi,v is the disparity weight for the pixels which fall in the target region. The Bhattacharyya coefficient of (8) and (9) can be derived as shown in (10). Thus, to maximize the similarity measure, the data being weighted by (11) has to be maximized. A new target position based on the previous target position is derived by the mean shift algorithm. ρˆ u ≈

C p ,w 1 m ˆ Pu ( y0 )Qˆ u + ∑ 2 u =1 2

⎛ y − xi wi ,D k ⎜ ∑ ⎜ h i =1 ⎝ nh

m

wi ,D = ∑Wi ,vδ [b( xi ) − u ] u =1

Qˆ u ˆ Pu ( y0 )

2

⎞ ⎟ ⎟ ⎠

(10)

(11)

If the kernel is chosen as the Epanechnikov profile [12], the derivative of the profile, g(x), is constant and a new target position can be reduced to (12). Fig. 4 shows the three steps of the tracking procedure. nh

yˆ1 =

∑w i =1 nh

i ,D

xi

∑ wi , D

(12)

i =1

4.2 Occlusion Detection and Recovery Using spatial information of targets and histogram similarities, it is possible to detect occlusion events. If the distance between the targets approached the threshold value, we checked the similarity of each target to find the occluding target with no significant difference in similarity value and the occluded target with a significant difference. To recover the occluded target information, we performed search process on each side of the occluding target. Occlusion detection and recovery is shown in Fig. 5.

590

C. Choi et al.

Fig. 4. Three steps of the tracking procedure. (a) Targeting step to obtain the coordinates of the area of interest. (b) Information-combining step wherein color and disparity information are combined with the spatial kernel for mean shift based tracking. (c) Mean shift based tracking step.

Fig. 5. Occlusion detection and recovery. (a) No occlusion in the 150th frame of the test data CM3. (b) Before occlusion (checking similarity and distance) in the 163rd frame. (c) Occlusion (checking similarity on both sides of the occluding target) in the 166th frame. (d) Recovered target in the 170th frame (e) No occlusion in the 182nd frame.

5 Experimental Results We used well-calibrated stereo cameras (Videre, STH-MDCS2) and the Small Vision System [13] for disparity computation. The computing platform was a Pentium 4, with 3 GHz CPU and 1G RAM. We used a 16 × 16 histogram in the normalized RG color space for face detection. The targets were represented by a 30 × 30 × 30 histogram in the RGB color space with a 50-disparity histogram.

Fig. 6. The Wash sequence. Frames 30, 190, 260, and 460 are shown. The top row (blue rectangles) shows the conventional color histogram method, and the bottom row (red rectangles) shows the proposed tracking method.


591

Fig. 7. The Turn sequence. The first row shows the results of the conventional color histogram method. The second row shows the results of the proposed method. The first column shows the target area that was provided manually. Because of color similarities, the labels of the targets changed after each crossing event (see the red and yellow rectangles in the first row).

Fig. 8. Output of the detection module in the Two_Man sequence. The yellow rectangles in the top row denote the detected head area, and the blue and white rectangles denote the detected body. Frames 76, 133, 275, and 327 are shown (from left to right). Note that the scenes are dynamic.

Fig. 9. The Two_Man sequences. Frames 0, 76, 133, 149, 153, 183, 275, 327, 387, 408, 426, and 558 are shown ( from top left corner to bottom right, left images of stereo camera).

C. Choi et al.

Similarity

592

Person_1 Person_2 Event

Frame index

Fig. 10. The value of the similarity function of the frame index for the Two_Man sequence. Every event in The Two_Man sequence was identified. Person_1 appeared in the FOV at the 76th frame, occluded by Person_2 at the 149th frame, and disappeared at the 246th frame. Person_1 reappeared at the 275th frame, disappeared at the 361st frame, and reappeared at the 399th frame (see the blue rectangles and red circles). The occlusion event happened twice.

We used three different video sequences. The first, two sequences, Wash and Turn, where the target area was given manually, were utilized for comparison. Then, the results of the proposed method were compared with a conventional color-based histogram tracking results. The third sequence, Two_Man, was utilized for analysis of the proposed method. In the Wash sequences, the target showed similar color distribution with the background initially, and it changed as the target moved across the room (because of the opened door). This illumination change produced inaccurate tracking results with the conventional color histogram, but the proposed method was able to overcome this drawback (see Fig. 6). In the Turn sequence, two people were walking around each other and they showed a similar color distribution (see Fig. 7). In the Two_Man sequence, there were two people who are walking cross from each other and in and out of the camera FOV independently. There were four different kinds of events in the Two_Man sequence, which included the appearing and disappearing of each person and the occlusion and otherwise of each person. Output of the proposed method is shown in Fig. 8. It can be seen that the detected head and body scale is dependent on the sparseness of disparity distribution. Each detection and tracking module performed well. Also, every occlusion events was detected, and the occluded persons were recovered successfully (see Fig. 9). Fig. 10 shows that each event in the Two_Man sequence can be identified using the histogram similarity function of the frame index.

6 Conclusions In this paper, we have proposed a real-time human detection and tracking system. The proposed method uses color information to find a face area and then detects a body by using the coordinates of the face area and disparity information. The tracking module uses a disparity-weighted color histogram that combines spatial color and disparity information. The proposed method overcomes the weaknesses of conventional mean shift based color tracking methods and can be applied in applications like mobile robots, where there are moving camera environments. One drawback of the proposed


593

method is a lack of scale adaptation to different scales. In the future, we plan to build a model of the disparity-scale, which will then provide the relationship between the object disparity and the scale. Using this disparity-scale model, we will develop the method into a more flexible system. Acknowledgement. This work was supported by the Korea Science and Engineering

Foundation (KOSEF) through the Biometrics Engineering Research Center (BERC) at Yonsei University. This research was supported by the Ministry of Information and Communication, Korea under the Information Technology Research Center support program supervised by the Institute of Information Technology Assessment, IITA2005-(C1090-0501-0019).

References 1. M. T. Yang, S. C. Wang, and Y. Y. Lin.: A Multimodal Fusion System for People Detection and Tracking. Int. J. Imaging Systems and Technology, Vol. 15 (2005) 131–142 2. T. Darrell, G. Gordon, M. Harville, and J. Woodfill.: Integrated Person Tracking Using Stereo, Colour, and Pattern Detection. Int. J. Computer Vision, Vol. 15 (2000) 175–185 3. S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp.: A Tutorial on Particle Filters for On-Line Non-Linear/Non-Gaussian Bayesian Tracking. IEEE Trans. Signal Processing, Vol. 50 (2002) 174-189 4. M. Isard, and A. Blake.: Condensation-Conditional Density Propagation for Visual Tracking. Int. J. Computer Vision, Vol. 29 (1998) 5. M. Soriano, B. Martinkauppi, and S. Huovinen.: Skin Detection in Video under Changing Illumination Conditions. IEEE Conf. on Computer Vision and Pattern Recognition, Vol. 1 (2000) 839-842 6. J. Yang and A. Waibel.: A Real-Time Face Tracker. IEEE Work. on Applic. CV. Sarasota. (1996) 108-121 7. P. Viola and W. Wells.: Alignment by Maximization of Mutual Information. Int. J. Computer Vision, Vol. 24 (1997) 137-154 8. C. Olson.: Image Registration by Aligning Entropies. Proc. IEEE Conf. Computer Vision and Pattern Recognition, Vol. 2 (2001) 331-336 9. D. Comaniciu and V. Ramesh.: Kernel-Based Object Tracking. IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 25 (2003) 564-577 10. K. Fukunaga.: Introduction to Statistical Pattern Recognition. 2nd edition. Academic Press. 1990 11. D. Comaniciu and P. Meer.: Mean Shift: A Robust Approach toward Feature Space Analysis. IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 24 (2002) 603-619 12. M. P. Wand and M. C. Jones.: Kernel Smoothing. Chapman & Hall. 1995 13. K. Konolige.: Small Vision Systems: Hardware and Implementation: Eighth International Symposium on Robotics Research. (1997) 111-116.

Towards Robust 3D Reconstruction of Human Motion from Monocular Video Cheng Chen, Yueting Zhuang, and Jun Xiao College of Computer Science, Zhejiang Univ., China {happyrobbie, yzhuang, junx}@cs.zju.edu.cn

Abstract. A robust, adaptive system for reconstructing 3D human motion from monocular video is presented. Our system takes a model based approach. To save computation time, we fit the skeleton instead of full body model to the silhouette. End sites positions identified from the silhouettes serve as an extra constraint. The alleviation of computational burden then makes the use of simulated annealing practical to get rid of local minima. The identifiable end sites also serve as the criterion to judge the reconstruction reliability of single frames. According to different reliabilities, the video is segmented into sections, which are reconstructed using different strategies. Our system is robust, getting rid of error accumulation in tracking, and adaptive, being able to tell the user when and where more information is needed. Keywords: video motion tracking, model based 3D reconstruction.

1 Introduction 3D tracking/reconstruction of human motion is an important subject in fields such as computer animation, virtual reality and so forth. Optical motion capture is widely used nowadays, but there are some applications where the expensive and intrusive optical MoCap systems cannot be employed. Compared to the optical counterpart, video based motion reconstruction is low cost, non intrusive and easy to use and is particularly suitable for a group of applications: 1. In a video teleconferencing environment, 3D poses of the presenter can be reconstructed and shown to the audience to improve the reality and clarity. 2. 3D gestures from video can be used in human-computer interface to allow intuitive control. For example, players can control 3D characters in computer games through gestures instead of keyboards, which greatly improves the experience. 3. Video surveillance systems can benefit from video based motion tracking by automatically tracking and recognizing dangerous actions. 4. There are huge numbers of existing video materials from which motion patterns and styles can be tracked and learned and then used in animation. However, video based 3D motion reconstruction is also a challenging task. [1]. The difficulty comes from the inherent ambiguity from 2D to 3D, frequent self occlusion of human body, noise in video, and so on. Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 594 – 603, 2006. © Springer-Verlag Berlin Heidelberg 2006

Towards Robust 3D Reconstruction of Human Motion from Monocular Video

595

Current human motion reconstruction techniques roughly fall into two categories— learning based and model based. In learning based methods, example 3D poses and corresponding images (features extracted from images) are the sources of learning. Then poses for new images are recovered. [2], [3], [4], [5]. For example [2] regresses a function by RVM using the examples silhouettes’ local shape context distribution and corresponding poses. [3] uses AdaBoost to select from the enormous amount of possible local features the most discriminative ones and use them to retrieve poses for new silhouettes from database. The main limitation of learning based methods is that they usually enforce a strong model on poses. Most methods only work on specific types of motions (walk, gulf swing [6]…). And they often need a database that is suitable for the problem domain, which also means an enormous training process [3]. Our method falls into the model based approach, which tries to estimate the pose that minimizes some metric that reflects the difference between the pose and the image. [7], [8], [9], [10], [11]. For example [9] precomputes several offline subsidiary databases and then use them to select best match from a captured 3D pose pool using silhouette and edge map. [10] uses genetic algorithm to estimate a pose that best fits the extracted features on images; however, the method heavily relies on successful tracking of feature points, which cannot be robust. Our work is close to [8], where a body is rendered at different poses and compared with silhouette using core-weighted XOR and image features. Our work differs from [8] in that we fit the skeleton instead of full body model to the silhouette to save computation time, that we use discernable end sites (head, hands, feet) as a metric of frame’s reconstructing difficulty as well as a constraint for the object function, and that our system has the ability to determine where manual information (key frames) are needed. Model based approaches tend to have two obstacles: high computational burden and spurious local minima. Some other methods try to reconstruct poses from images taken by multiple cameras. [12] calculates the 3D voxel from 5 synchronized images. These methods are more accurate when the number of cameras increases, but this also enforces severe limitations to the system’s application field. Also, some methods take an optical-hybrid way. [13] puts optical markers (far fewer than normal optical MoCap systems) on the body and recover 3D motions in real time. The markers’ locations can be reliably determined and serve as indices into a pre-computed pose database. Then online pose synthesis is conducted locally. In this paper we present a system reliably generating visually pleasing 3D motions from uncalibrated monocular video. No prior knowledge of the motion pattern (such as database) is used. The system is easy to setup and easy to use. It is free from common problems in motion tracking such as error accumulation/propagation and difficulty in recovering from errors. The system is also adaptive. It can tell the user where it needs more information to complete the task. We’ll see that with limited intuitive user interaction, complex motions can be successfully reconstructed. The novelties and contributions of this paper include: 1. The use of skeleton instead of full body in pose estimation greatly reduces the object function’s computational burden and therefore allows the use of simulated annealing, which can solve the local minima problem. 2. We make novel uses of end sites positions as a component of pose estimation object function as well as the criterion of reconstruction reliability.

596

C. Chen, Y. Zhuang, and J. Xiao

3. We delicately combine two detectors with different properties in hands detection. The skin color detector learned on the fly adds power to the curvature detector without introducing severe robustness problem. 4. We develop a reconstruction methodology that divides video into sections based on the reconstruction reliability and then uses different strategies for different sections. This eliminates error accumulation and brings the system the ability to automatically decide where and when it needs more information from the user. Fig. 1 gives an overview of the system workflow. Videos are transformed to silhouettes, from which end sites are identified. Then poses are estimated for frames. Finally, smooth motions from the estimated poses are generated as the final results.

Fig. 1. System workflow

2 Silhouette Extraction and End Sites Identification The first step of our system is extracting silhouettes from video frames. We choose a silhouette way because it has several advantages: 1. Silhouettes can be relatively robustly extracted, at least when we are in a controlled environment and hence the background model is accessible. 2. Silhouettes are insensitive to irrelevant factors. Many tracking methods based on color or texture can be defeated by variations in clothes texture or even cockle. 3. Simple as they look, silhouettes do reserve much information about the pose. In this paper the silhouette extraction method presented in [14] is employed. The method models the background in a statistical approach and is robust to global and local illumination changes (shadows). Noises are further removed as described in [15]. After silhouettes are extracted, we try to identify end sites on them. By “end sites” we mean hands, feet and head. By “identify” we mean both “detect”—finding out the positions of end sites, and “label”—distinguish hands from feet and left from right. The most intuitive detection method might be directly looking at curvatures on the silhouette contour. Fig. 2 shows some results. To be more robust, we employ different strategies for different end sites. Head is the easiest one, because it is almost never occluded by other body parts, it always lies at a local highest position, and its corresponding section on the contour consists of points with not very sharp but steady positive convex curvatures. For feet, they usually lie in the bottom of the silhouette, and since we are not using high resolution cameras, we don’t make further


597

distinctions between the ankles and toes. Hands are vital to the pose comprehension, yet they are most difficult to detect, mostly because they are sometimes “shaded” by the body. Also, sometimes the arms bend so heavily that the high curvatures at the elbow can confuse the detector. In light of these, we set the parameters of the curvature detector for hands stricter. We are trading recall for precision here.

Fig. 2. Sample results for the curvature detector. Red on contour stands for large convex curvatures; blue stands for large concave curvatures. Green dots represent the detected location.

Note in Fig. 2 that only one hand of the 2nd and 3rd silhouettes is detected, respectively. The other one is “shaded” by the body. We have to turn to things beyond silhouettes to deal with this. Since the precision of curvature detector is very high, we build a color model based on the detected hands on the fly and then used the color model to detect remaining hands missed by our high-precision-low-recall curvature detector. In this way we delicately combine the two hand detectors. We use a standard Gaussian color model. Note the hands might be physically occluded by the body and still not discernable. The use of color might cause robustness problem and might seem contradictory to our initial purpose of using silhouette. However, we constrain our searching area to be only in the corresponding silhouette. This greatly reduces the robustness risk, since we don’t search on the background, where most robustness problems come from. The automatic end sites labeling is based on the temporal smoothness. User is required to label the first frame where all 5 end sites are detected. Then the system uses the temporal information to automatically label successive frames. To be more robust, we predict the end sites positions using Kalman filter before labeling.

3 Pose Estimation First we formulize the representation for poses. Each joint in the human skeleton model has 3 rotational DOFs. The root joint has 3 additional DOFs representing global translation. In pose estimation step, we consider pose as the vector of all joints’ rotations. The root’s translation is not included since it reflects the figure’s global position. We postpone the consideration of root’s translation until motion generation, where feet-ground contact is taken into account. Videos are essentially 2D projections of the 3D real world. In other words, given sufficient parameters, a projection function projects a pose vector p to a 2D silhouette s.

598


Our problem is actually the reverse one: we now have s, and we want to recover p. This is mathematically ill posed, since the projection function doesn’t have a reverse counterpart. But due to constraints such as the anatomy and the temporal smoothness, it’s still probable to find a pose p that best explains silhouette s. We take a searching approach—searching for the pose that minimizes some object function E that measures the goodness of the pose according to the silhouette. 3.1 Object Function Design The object function E is made up of three terms. 1. The core area term Ecore-area. The 2D projection of the skeleton at pose p, after alignment and scaling, should lie in the core areas of the silhouette, i.e., the limbs should lie in the median axis of the limbs area on the silhouette. To quantify the core area, we do Euclidean distance transformation to the silhouette. (Fig. 3).

Fig. 3. (1) The alignment to core area. (2) The Euclidean distance transform.

Stating in detail, we calculate the positions of the 16 joints projected on the 2D silhouette, connect adjacent joints to by straight lines to form the limbs, sample some points on the limbs, and calculate the average value on the distance transformed silhouette at locations of the sampled points.

Ecore − area = −∑ s '(t )(point (m)iT(t , p)) / M .

(1)

m

In equation (1), s’(t) is the distance transforms of silhouette at time t, point(m) is the mth sampled point on the skeleton, T(t,p) is the t- and p- dependent rigid transform that converts the local coordinate vector of point(m) to 2D location on the silhouette, and M is the total number of sampled points. Note that if an end site is not identified, equation (1) does not compute the contribution made by the corresponding limb. 2. The coverage term Ecoverage. The core-area term alone is not sufficient to determine the pose, because the limbs might be not sufficiently stretched to cover the limbs in the silhouette and still remain a low Ecore-area. To solve this, we make sure that the locations of the end sites projected on the silhouette are close to the locations identified in Sec. 2.


Ecoverage =

∑

|| loc sil − loc ske ||2 / n .

for all identified end sites

599

(2)

In equation (2), locsil is the end site’s location identified on the silhouette in Sec. 4, locske is the projected skeleton’s end site’s location, and n is the total number of end sites identified at the current frame. 3. The smoothness term. This term comes from the temporal smoothness. The pose should not change abruptly. We use the two previous poses p(t-1) and p(t-2) to measure the smoothness.

Esmoothness =|| p (t ) − 2p(t − 1) + p(t − 2) ||2 .

(3)

Combining equations (1), (2) and (3), we get the final object function E.

E = α Ecore − area + β Ecoverage + γ Esmoothness .

(4)

3.2 Video Optimization Strategy The determining of the desired pose p can be formulated as an optimizing problem of minimizing E, by varying p in the valid range. Note due to biomechanical limitation, the valid range for p is a closed sub region in the pose vector Euclidean space. Due to the complexity of E, derive based optimizing methods cannot be used. Worse is that E has many spurious local minima. We use simulated annealing (SA), which helps a lot in avoiding local minima. By setting the parameter such as the initial temperature and annealing rate, it is very easy to control how much ability of getting out of local minima is allowed. That is important, since for most frames we have an initial guess of the pose and we need only conduct partial SA to save time. In Sec. 1 we have pointed out the two challenges of model based approaches: high computational burden and spurious local minima. Now let’s see how we cope with those. To alleviate object function’s complexity, we fit the skeleton instead of full body model (such as in [8]) to the silhouette. This significantly reduces complexity since only the 2D projections of joint points need to be calculated. The sampled points on the limbs are interpolated directly from the 2D positions of the joints. Our computationally efficient object function then allows the use of SA to help get rid of local minima. If we fit the full body and still use SA, the object function would be so expensive that the estimation of one pose would possibly takes hundreds of seconds. It turns out that if we estimate poses based only on single frames, some frames can not be correctly reconstructed, partly because they don’t have their end sites completely identified and so information is insufficient. In such cases, temporal information should be made full use of. First we classify frames as reliable frames, if all their end sites are identified, or unreliable frames if otherwise. Due to the continuity, reliable and unreliable frames tend to cluster into interleaved reliable sections and unreliable sections. (Fig. 4). Clearly, reliable frames have larger chances to be correctly estimated. Also, due to temporal smoothness, unreliable frames near the section boundaries have larger hope than the ones deep inside the unreliable section. Therefore, if an unreliable section is longer than a preset length L, we label it as a dangerous section, meaning that frames

600


Fig. 4. Segmentation of a real video clip into interleaved reliable and unreliable sections. The number shows the location (in frame sequence) of the section boundaries.

deep inside it are unlikely to be correctly estimated. User is required to specify a key frame in the middle of the dangerous section. Then the key frame splits the long unreliable section into two short ones. This procedure is repeated until no dangerous section is present, and then the following reconstruction algorithm is taken: 1. The algorithm starts at every reliable section. For each reliable section, we choose a base frame and do a full SA to estimate its pose. Full SA means high initial temperature and slow annealing rate. Here the smoothness term of the object function is set to 0. This is the key to getting rid of error accumulation as base frames for each reliable section are estimated independently. 2. For each base frame, we successively estimate frames around it in both directions, until frames in its containing reliable section and two surrounding unreliable sections are covered. 3. Now we have two poses for each unreliable frame from the two surrounding reliable sections. We blend the two poses, using the inverses of the distances to the two surrounding section boundaries as the weights. Up to now, we have estimated poses for all frames. The next stage is to combine these poses to generate a satisfying motion.

4 Motion Generation The motion generation stage has two tasks: smoothly combine the poses, and determine the root’s translations according to feet-ground contact. We apply a simple Gaussian low pass filter on the poses to get smoothness. To solve the contact problem, we make use of the estimated feet height. For most motions where at any time at least one foot is on the ground and no slide occurs, the feet’s height could be used to determine. For motions possible to have two feet flying or where slide exists, such as jumping and skiing, user has to manually adjust the root’s translation. If needed, the generated motion can be further fined by 3D animation tools.

5 Results We use a skeleton with 16 joints in our experiment. Therefore our pose vector is a 48D. As we discard the root’s translation in pose estimation, when calculating the object function, we have to automatically adjust the scaling and alignment of the


Fig. 5. Representative results for reconstructing walking motions

Fig. 6. Representative results for reconstructing fence motion

Fig. 7. Representative results for reconstructing Latin dance

601

602


skeleton according to the silhouette. The alignment is straightforward since we already have head positions detected on the silhouette and calculated on the skeleton by projection. We do the automatic scaling using the method presented in [16]. Most of the computation occurs at pose estimation. It typically takes 5~6 seconds to estimate a base frame and slightly less than 1 second for a non base frame. Usually the proportion of base frame in a video clip is less than 10% (typically around 5%). We show representative results for walking, fence and Latin dance (Fig. 5~7).

6 Conclusion and Future Work We’ve presented and implemented a system that robustly reconstructs 3D motion from monocular video. Instead of tracking features directly on the image, which is sensitive to noise and irrelevant factors, we take a model based approach using silhouettes. In hands detection, we gracefully combine the power of two detectors to achieve excellent results. In pose estimation, we fit the skeleton instead of full body model to the silhouette and it significantly reduces the computational burden and allows the use of SA to cope with the spurious local minima. Our algorithm restarts at every reliable section and avoids error accumulation. Our reconstructing strategy also enables the system to tell when and where more information is needed from the user. Our system has some limitations. It is difficult to extract silhouettes from videos with highly clustered background and/or unpredictable camera movement. Also, since temporal information is employed, our system might fail on abrupt or very fast motions. Cameras with high sampling rate can be used to solve this problem, but this enforces a serious limitation on the usage. Those are some of the potential future work. Acknowledgments. This work is supported by National Natural Science Foundation of China (No.60533090, No.60525108), 973 Program (No.2002CB312101), Science and Technology Project of Zhejiang Province (2005C13032, 2005C11001-05), and China-US Million Book Digital Library Project (www.cadal.zju.edu.cn).

References 1. Gleicher M., Ferrier N.: Evaluating Video-Based Motion Capture. Computer animation 2002(CA02). Los Alamitos: IEEE Computer Society Press; 2002. p. 75--81. 2. Ankur Agarwal and Bill Triggs: Recovering 3D Human Pose from Monocular Images. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 28, No. 1, January 2006 3. L. Ren, G. Shakhnarovich, J. Hodgins, H. Pfister, and P. Viola: Learning Silhouette Features for Control of Human Motion. Proceedings of the SIGGRAPH 2004 Conference on Sketches & Applications, August 2004 4. Ahmed Elgammal and Chan-Su Lee: Inferring 3D Body Pose from Silhouettes Using Activity Manifold Learning. 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'04) - Volume 2 pp. 681-688 5. V. Athitsos and S. Sclaroff: Estimating 3D Hand Pose from a Cluttered Image. Proc. Int’l Conf. Computer Vision, 2003.


603

6. R. Urtasun, D. Fleet, and P. Fua: Monocular 3--D Tracking of the Golf Swing. Conference on Computer Vision and Pattern Recognition, San Diego, CA, June 2005. 7. C. Bregler and J. Malik: Tracking people with twists and exponential maps. Proceedings of Computer Vision and Pattern Recognition (CVPR'98), 1998 8. Yisheng Chen, Jinho Lee, Rick Parent, Raghu Machiraju: Markerless Monocular Motion Capture Using Image Features and Physical Constraints. Computer Graphics International 2005, Stony Brook, USA, June 2005. 9. Antonio S. Micilotta, Eng Jon Ong and Richard Bowden: Real-time Upper Body 3D Pose Estimation from a Single Uncalibrated Camera. In Eurographics 2005 10. Jianhui Zhao and Ling Li: Human motion reconstruction from monocular images using genetic algorithms. Computer Animation and Virtual Worlds 2004; 15:407-414 11. D.E. DiFranco, T. Cham, and J.M. Rehg: Reconstruction of 3-D Figure Motion from 2-D Correspondences. Proc. Conf. Computer Vision and Pattern Recognition, pages 307.314, 2001. 12. M. Cheung, T. Kanade, J.-Y. Bouguet, and M. Holler: A real time system for robust 3D voxel reconstruction of human motions. CVPR00, V2, P714--720. 13. Jinxiang Chai and Jessica K. Hodgins: Performance Animation from Low-dimensional Control Signals. ACM Transactions on Graphics (SIGGRAPH 2005) 14. T. Horprasert, D. Harwood, and L. Davis: A statistical approach for real-time robust background subtraction and shadow detection. IEEE ICCV FRAME-RATE Workshop, 1999. 15. Ronald Poppe, Dirk Heylen, Anton Nijholt, Mannes Poel: Towards real-time Body Pose Estimation for Presenters in Meeting Environments. WSCG (Short Papers) 2005: 41-44 16. Jerome Vignola, Jean-François Lalonde and Robert Bergevin: Progressive Human Skeleton Fitting. Conference on Vision Interface, 2003

Wavelet Scalable Video Coding with Minimal Average Temporal Distance for Motion Estimation Changhoon Yim1 , Hojin Ha2 , Bae-Keun Lee2 , and Woo-Jin Han2 1

Dept. of Internet and Multimedia Eng., Konkuk University, Seoul 143-701, Korea [email protected] 2 Digital Media R&D Center, Samsung Electronics, Suwon, Kyeonggi-Do 442-742, Korea

Abstract. Wavelet scalable video coding (WSVC) can be used efficiently for scalable mesh compression for virtual reality applications over IP networks. This paper proposes efficient methods to improve the coding efficiency of WSVC through the minimization of average temporal distance (ATD) for motion estimation. The ATD is closely related to the peak signal-to-noise ratio (PSNR) performance of WSVC which is based on motion compensated temporal filtering. Simulation results show that the proposed methods with minimum ATD give about 1 to 2dB PSNR improvements compared to the conventional WSVC.

1

Introduction

Three-dimensional (3-D) meshes are used intensively for interactive graphics applications. These applications require 3-D mesh compression algorithm to reduce the transmitted data size [1]. Considerable researches have been performed for 3D mesh compression [1], [2], [3]. Since single-level compression techniques do not consider network characteristics, the received content cannot be displayed until the whole bitstream is fully downloaded, which introduces additional delay in low-bandwidth networks [2]. Hence scalable mesh compression techniques have been investigated to account for variable bandwidth in networks [1], [3]. The scalable 3-D compression can be used efficiently for virtual reality applications over IP networks with variable bandwidth. Scalable video coding has received considerable attention for multimedia content applications with variable bandwidth environments [4], [5]. Wavelet scalable video coding (WSVC) inherently provides very flexible scalable bitstream through wavelet transform [6], [7], [8], [9]. Even though scalability is an attractive feature for multimedia content applications in variable bandwidth environments [5], the performance improvement of WSVC is an important issue for its viability in practical applications. In this paper, we propose efficient methods to improve the coding efficiency of WSVC through the minimization of overall temporal distance for motion estimation. The conventional WSVC method uses only forward direction motion Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 604–613, 2006. c Springer-Verlag Berlin Heidelberg 2006

WSVC with Minimal Average Temporal Distance for Motion Estimation

605

estimation. There can be many combinations for the directions of motion estimation. If there are 16 frames in a group-of-pictures (GOP), then there are 15 frame pairs for motion estimation through levels 0, 1, 2, and 3. If forward or backward direction is selected for each frame pair, then 215 cases are possible for the directions of motion estimation in 15 frame pairs. The conventional WSVC, which uses only forward direction motion estimation, is a very special case among the 215 possible cases. To quantify the overall temporal distance, we define average temporal distance (ATD) for motion estimation, which will be introduced in Section 2. The conventional WSVC method has a large ATD value, which is one of the reasons for quality degradation. This paper proposes two methods which minimize the ATD for motion estimation to improve the coding efficiency of WSVC. This paper is organized as follows. Section 2 presents the overview of WSVC and the conventional method for the directions of motion estimation in WSVC. In Section 3, we propose methods which minimize the ATD for motion estimation in WSVC. Section 4 shows simulation results for performance comparison with the conventional WSVC. Finally, we present conclusion in Section 5.

2

Wavelet Scalable Video Coding

Fig. 1 represents the encoder block diagram of wavelet scalable video coding (WSVC) [7], [8], [9]. Motion compensated temporal filtering (MCTF) block decomposes image sequences in a GOP using the motion vector information from motion estimation (ME) block. The temporally decomposed images are spatially decomposed in the spatial wavelet decomposition block. This paper presents efficient methods for the directions of motion estimation relating the ME and MCTF blocks in Fig. 1. Fig. 2 shows the directions of motion estimation in the conventional WSVC [7], [8], [9]. There are 16 frames in a GOP. In Fig. 2, low and high sub-band frames are shown as white and shaded rectangles, respectively. In the motion compensated temporal filtering (MCTF) in Fig. 2, the directions of motion estimation are all

Fig. 1. Encoder block diagram of wavelet scalable video coding

606

C. Yim et al.

Fig. 2. Directions of motion estimation in the conventional WSVC at each level

forward. For example, the direction for motion estimation between frame 1 and frame 2 at level 0 is from frame 1 to frame 2. Then the temporal high sub-band is located at frame 1 (H,1) and the temporal low sub-band is located at frame 2 (L,2). In this case, the temporal low sub-band L,2 at level 1 is similar to the frame 2 at level 0, and the temporal high sub-band H,1 at level 1 is similar to the edge image of frame 1 at level 0. At level 1, the temporal low sub-band frames are decomposed into low and high sub-bands at level 2. For example, motion estimation is performed from frame L,2 to frame L,4 for temporal decomposition. Then the temporal high sub-band is located at frame location 2 (LH,2) and the temporal low sub-band is located at frame location 4 (LL,4) at level 2. At level 2, the temporal low sub-band frames are decomposed into low and high sub-band at level 3. At level 3, the temporal low sub-band frames (LLL,8 and LLL,16) are decomposed into low sub-band (LLLL,16) and high sub-band (LLLH,8) at level 4. The temporal distance for motion estimation becomes larger as the level becomes higher in Fig. 2. At level 0, the temporal distance for motion estimation is 1. For example, motion estimation is performed between frame 1 and frame 2, and the temporal distance is 1. At level 1, the temporal distance for motion estimation is 2. For example, motion estimation is performed between frame L,2 and frame L,4, and temporal distance is 2. Similarly, the temporal distance at level 2 and at level 3 is 4 and 8, respectively. In Fig. 2, we can see that there are 8, 4, 2, and 1 frame pairs for motion estimation at level 0, 1, 2, and 3, respectively. Hence there are 15 frame pairs for motion estimation through levels 0, 1, 2, and 3.


607

Table 1. The number of frame pairs for motion estimation and the temporal distance for motion estimation at each level in the conventional WSVC

Level Level Level Level

0 1 2 3

Number of frame pairs Temporal distance for motion estimation for motion estimation 8 1 4 2 2 4 1 8

Table 1 shows the number of frame pairs for motion estimation and the temporal distance for motion estimation at each level in the conventional WSVC. As the temporal distance becomes larger, the magnitude of motion vectors becomes larger, especially in video sequences with high motions. In the conventional WSVC method in Fig. 1, the temporal distance becomes larger as the level becomes higher. The large temporal distance at higher levels is one of the reasons for coding efficiency degradation in the conventional WSVC. To quantify overall temporal distance for motion estimation, we define average temporal distance (ATD) for motion estimation as the average value of temporal distances in 15 motion estimations. In the conventional WSVC in Fig. 1, the ATD can be calculated as: AT D =

8×1+4×2+2×4+1×8 = 2.13 15

In this paper, we propose methods which minimize the ATD to improve the coding efficiency of WSVC. The proposed methods would give better coding efficiency through smaller ATD values compared to the conventional WSVC.

3

Proposed Methods for Minimizing ATD for Motion Estimation

We propose two methods for the direction of motion estimations as Method 1 and Method 2. We argue that these two methods have minimal ATD values among possible ATD values when there are 16 frames is a GOP. Fig. 3 and Fig. 4 represent Method 1 and Method 2, respectively. In these figures, the solid arrow represents the forward direction motion estimation and the dotted arrow represents the backward direction motion estimation. At level 0, the directions for motion estimation are forward and backward. The direction for motion estimation between frame 1 and frame 2 is from frame 1 to frame 2, which is forward. The temporal high sub-band is located at frame 1 (H,1) and the temporal low sub-band is located at frame 2 (L,2). However the direction for the next pair of frames is different. The direction for motion estimation between frame 3 and frame 4 is from frame 4 to frame 3, which is backward. The temporal high sub-band is located at frame 4 (H,4) and the temporal low sub-band is located at frame 3 (L,3).

608

C. Yim et al.

Fig. 3. Method 1 for the directions of motion estimation at each level

Fig. 4. Method 2 for the directions of motion estimation at each level


609

At level 1, motion estimation is performed between frame L,2 and frame L,3. Note that the temporal distance for motion estimation is 1 at level 1 in Method 1 (Fig. 3), while the temporal distance for motion estimation is 2 at level 1 in the conventional WSVC method (Fig. 2). By changing the directions for motion estimation to forward and backward at level 0 as in Fig. 3, the temporal distance for motion estimation at level 1 is reduced to 1. There can be two variations in Method 1. The direction for motion estimation at level 3 can be either forward or backward. We call these two variations as Method 1-a and Method 1-b. In Method 1-a and Method 1-b, the LLLL frame is located at 10-th and 7-th frame, respectively. The Method 1-a and Method 1-b have exactly the same number of frame pairs for motion estimation and the same temporal distance for motion estimation as in Table 2. The directions for motion estimation at level 0 are the same in Method 1 (Fig. 3) and in Method 2 (Fig. 4), while the directions for motion estimation at level 1 are different. The direction for motion estimation between L,6 and L,7 is forward and the direction for motion estimation between L,10 and L,11 is backward in Method 1 (Fig. 3). On the other hand, the direction for motion estimation between L,6 and L,7 is backward and the direction for motion estimation between L,10 and L,11 is forward in Method 2 (Fig. 4). There can be two variations in Method 2. The direction for motion estimation at level 3 can be either forward or backward. We call these two variations as Method 2-a and Method 2-b. In Method 2-a and Method 2-b, the LLLL frame is located at 11th and 6-th frame, respectively. The Method 2-a and Method 2-b have exactly the same number of frame pairs for motion estimation and the same temporal distance for motion estimation as in Table 3. Table 2. The number of frame pairs for motion estimation and the temporal distance for motion estimation at each level in Method 1


0 1 2 3


Table 2 shows the number of frame pairs for motion estimation and the temporal distance for motion estimation at each level in Method 1. In Method 1, the average temporal distance (ATD) value can be calculated as 8×1+4×1+2×4+1×3 = 1.53 15 Table 3 shows the number of frame pairs for motion estimation and the temporal distance for motion estimation at each level in Method 2. In Method 2, the ATD value can be calculated as 8×1+4×1+2×3+1×5 AT D = = 1.53 15 AT D =

610

C. Yim et al.

Table 3. The number of frame pairs for motion estimation and the temporal distance for motion estimation at each level in Method 2


0 1 2 3


The ATD value is 1.53 both in Method 1 and in Method 2 while the ATD value is 2.13 in the original WSVC method. We argue that the ATD value 1.53 of the proposed two methods is the minimal ATD value among the possible ATD values with different directions for motion estimation when there are 16 frame pairs in a GOP. The ATD value represents the overall temporal distance for motion estimation. Since the overall temporal distance for motion estimation is minimum, the overall magnitude of motion vectors would be minimum in the proposed methods. This characteristic would give better coding efficiency compared to the conventional WSVC.

4

Simulation Results and Performance Evaluation

In this section, we present simulation results of the proposed methods and the conventional WSVC method. We also simulated another two more methods having different ATD values for comparison. We call these two methods as Method 3 and Method 4. The ATD values of Method 3 and Method 4 are 1.73 and 1.67, respectively.

27.5 27

Average PSNR (dB)

26.5 26 25.5 25 24.5 24 23.5 23 350

400

450

500

550 600 650 Bit rate (kbps)

700

750

800

850

Fig. 5. PSNR comparison between the conventional WSVC method and the proposed methods for the Canoa CIF sequence (diamond: conventional method, circle: Method 1-a, square: Method 2-a, *: Method 3, +: Method 4)


611

30 29.5

Average PSNR (dB)

29 28.5 28 27.5 27 26.5 26 25.5 25 350

400

450

500


700

750

800

850

Fig. 6. PSNR comparison between the conventional WSVC method and the proposed methods for the Bus CIF sequence (diamond: conventional method, circle: Method 1-a, square: Method 2-a, *: Method 3, +: Method 4)

42 41.5

Average PSNR (dB)

41 40.5 40 39.5 39 38.5 38 37.5 37 350

400

450

500


700

750

800

850

Fig. 7. PSNR comparison between the conventional WSVC method and the proposed methods for the Foreman CIF sequence (diamond: conventional method, circle: Method 1-a, square: Method 2-a, *: Method 3, +: Method 4)

Fig. 5 shows the peak signal-to-noise ratio (PSNR) comparison between the conventional WSVC method and the proposed methods for the Canoa CIF sequence. The performances of Method 1-a and Method 2-a are about the same, and give about 1 to 2 dB higher PSNR compared to the conventional WSVC method. Fig. 6 shows PSNR comparison between the conventional WSVC method and the proposed methods for the Bus CIF sequence. The Method 1-a and Method 2-a give about 1.0 dB and 1.5 dB PSNR improvements, respectively, compared to the conventional WSVC. Fig. 7 shows PSNR comparison between the conventional WSVC and the proposed methods for the Foreman QCIF sequence. The performances of Method

612

C. Yim et al.

1-a and Method 2-a are slightly better than those of Method 3 and Method 4. Method 1-a and Method 2-a give about 0.5 dB improvements compared to the conventional WSVC. These simulation results show that the performance of WSVC is closely related to the ATD for motion estimation. More specifically, smaller ATD value gives better performance (higher average PSNR). The Method 1 and Method 2 with minimum ATD give best performance. Also, all four methods (Method 1, Method 2, Method 3, and Method 4) give better performance compared to the conventional WSVC.

5

Conclusion

We proposed methods for minimizing the overall temporal distance for motion estimation to improve the coding efficiency of WSVC. To quantify the overall temporal distance for motion estimation, we defined average temporal distance (ATD) as the average value of temporal distance in 15 motion estimations. The ATD value is 1.53 both in the Method 1 and in Method 2, while the ATD value is 2.13 in the conventional WSVC. The proposed methods give minimal ATD when there are 16 frames in a GOP. We also simulated Method 3 and Method 4, which have larger ATD values compared to Method 1 and Method 2, for comparison. Also, all four methods (Method 1, Method 2, Method 3, and Method 4) give better performance compared to the conventional WSVC. Simulation results show that smaller ATD value gives better performance in WSVC. The proposed Method 1 and Method 2 with minimum ATD value give the best performance. One of the drawbacks of the proposed method is that it leads to nonequidistant temporal position of frames at higher levels of the wavelet tree, which results in more limited temporal scalability compared to the conventional WSVC. Hence there is a trade-off between the PSNR performance and the temporal scalability through the minimization of ATD. Since the WSVC has good compression capability and very flexible scalability for variable bandwidth environments, it can used efficiently for virtual reality applications over IP networks using scalable 3-D mesh compression.

References 1. Al-Regib, G., Altunbasak, Y., Mersereau, R.M.: Bit allocation for joint source and channel coding of progressively compressed 3-D models. IEEE Trans. Circuits Syst. Video. Technol. 15(2) (2005) 256–268 2. Taubin, G., Rossignac, G.: Geometric compression through topological surgery. ACM Trans. Graph. 17(2) (1998) 84–115 3. Compressed progressive meshes. IEEE Trans. Vis. Comput. Graphics, 6(1-3) (2000) 79–93 4. Sodagar, I., Lee, H.-J., Hatrack, P., Zhang, Y.-Q.: Scalable wavelet coding for synthetic/natural hybrid images. IEEE Trans. Circuits Syst. Video Technol. 9(2) (1999) 244–254 5. Applications and requirements for scalable video coding. ISO/IEC JTC1/SC29/ WG11 MPEG 65-th Meeting, N5889. (2003)


613

6. Ohm, J.-R.: Three Dimensional subband coding with motion compensation. IEEE Trans. Image Processing 3(5) (1994) 559–571 7. Choi, S.-J., Woods, J.W.: Motion-compensated 3D subband coding of video. Proc. IEEE Image Processing, 8(2) (1999) 8. Hsiang, S.-T., Woods, J.W.: Embedded video coding using invertible motion compensated 3-D subband/wavelet filter bank. Signal Processing: Image Comm., 16 (2001) 705-724 9. Hsiang, S.-T., Woods, J.W.: Highly scalable subband/wavelet image and video coding. Ph.D. Dissertation, Rensselaer Polytechnic Institute, (2002)

Single Camera Remote Eye Gaze Tracking Under Natural Head Movements Ruian Liu1,2, Shijiu Jin1, and Xiaorong Wu2 1

State Key Laboratory of Precision Measuring Technology and Instruments, Tianjin University, Tianjin 300072, China 2 College of Physics and Electronic Information Science, Tianjin Normal University, Tianjin 300074, China [email protected]

Abstract. Most available gaze tracking systems based on corneal reflection have low tolerance for head movements. In this paper, an eye-gaze tracking method, which can make this system be adaptive to external lighting changes and to the user’s head movements, is put forward to alleviate the restriction on the user’s head motion. We manage to improve the performance of the algorithm: The thresholds for extraction of the pupil and the corneal reflection are automated; the edge points of the pupil could be obtained by a 1D line search. To improve the resolution and accuracy of the system, the sub-pixel technique is used to locate the feature position accurately in the eye image. We apply a new randomized method of ellipse fitting to the edge points for good results. Our system can work under natural head movements by using only one camera to track eye-gaze. It can run in real time at video rate (25frames/second). A gaze accuracy of 1 degree is observed. Keywords: Gaze tracking; corneal reflection; threshold; ellipse fitting; calibration.

1 Introduction The eye is an important organ by which people acquire information from the outside world, and it is also a window, which reflects the activities in one’s mind. Eye trackers have been developed for many years and they are used in many fields such as medical research, psychiatry research and human machine interfaces. The eye-gaze interface gives us a more natural way to interact with devices than keyboard and mouse, and is convenient in situations where it is important to use the hands for other tasks. There are numerous methods which have been developed to track eye-gaze, however many of those methods need special hardware, require the user to wear a device on their head, or place limitations on the user’s head movements. Moreover, some eye trackers use more invasive methods, such as ultrasonic or electromagnetic devices to track the head position, which is undesirable for frequent usage. There are still many unresolved problems preventing the use of non-intrusive eye gaze systems in actual applications. The most prevalent of these are: accuracy, restricted head movement, robustness, and ease of calibration. Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 614 – 623, 2006. © Springer-Verlag Berlin Heidelberg 2006

Single Camera Remote Eye Gaze Tracking Under Natural Head Movements

615

In this paper, a novel corneal-pupil reflection scheme is proposed that using only one camera to track eye-gaze. Combing with the improved algorithms, the system is robust and good results are obtained. The paper is structured as following: Related works are introduced in Section 2. Eye structure and gaze model are described in Section 3. Our proposed eye-gaze estimation system is given in Section 4.The developed algorithms are explained in Section 5. The experiment result is shown in Section 6. Finally, the conclusion and future works are mentioned in Section 7.

2 Related Works The eye-gaze tracking technique, based on corneal reflection and image processing, has the advantage of unobtrusiveness and has made considerable headway in recent years. Several systems [1-9] have been built based on Pupil Center Corneal Reflection (PCCR) technique. Most of these systems show that high accuracy of the eye-gaze tracking results can be achieved if the user has the ability to keep his head fixed or via the help of a chin rest or bite bar to restrict the head motion. Therefore, the limited head movement is one of the worst issues in the current remote eye-gaze tracking systems. To solve the problem, Sugioka [1] measured the distance between the camera and the eye by using an ultrasonic sensor, however, the accuracy of the system is not good because the distance measured by an ultrasonic sensor is inaccurate. Ebisawa [6] developed an eye-gaze tracking system that used three cameras, and the two stereo cameras measured the 3D position of the pupil of one eye. This increased the complicity of the system and the performance of the system was affected by the accuracy of the stereo camera. To reduce the dependent on the environment, many systems [2-3]， [5-9] employed the bright-pupil technique and the image difference method. But this method increased the pupil-locating error. This paper also uses the bright-pupil and dark-pupil technique, however, different from the traditional method, we only use it to detect the pupil when eye tracking is failed. Currently many gaze tracking algorithms make use of the technique of thresholding and ellipse-fitting [4]， [5]， [7], [8], however, owing to the noises in eye images, it is difficult to determine the appropriate threshold values, also the ellipse-fitting algorithm is not robust. In this paper, the algorithms of threshold values extraction, feature detection and ellipse fitting have been improved; also, the sub-pixel technique is used to improve the resolution and accuracy of the system.

3 Eye Structure and Gaze Model The structure of the eye is shown in Fig.1 [4]. When the head is still, the gaze direction changes following the rotation of the eyeball. Experiments indicate that the rotation of the eyeball is a very complicated process, even only the eye moves horizontally; it is also moving on a trajectory in front and behind, not a simple spherical rotation. The gaze direction is defined as the direction from fovea to the center of the crystalline lens, however, the crystalline lens’s changes are very complicated and it is difficult to detect the position of the fovea, so the eye is commonly viewed as a spheroid, which rotates

616

R. Liu, S. Jin, and X. Wu

round a fixed point. The rotation center, which is the so-called eyeball center, is defined as the point that is 13.5mm behind the corneal on the gaze direction [10]. Gaze direction is defined as the direction from the eyeball center to the center of the pupil. For simple analyzing and calculating, the paper assumes that the estimated gaze direction can be derived from the center of the corneal curvature and the pupil center. The gaze model is shown in Fig. 2.

Fig. 1. The structure of the eye

4 System Specifications and Control Model The system is based on PCCR technique. Our proposed method needs three infrared LEDs and one CCD camera. LED2 and LED3 are used to project corneal reflections onto the eye; LED1 is located at the center of the camera lens to make the bright-pupil effect. A video camera is used to capture the eye movements. The camera is mounted on a pan-tilt unit to scan two-dimensionally, so the user’s eye is always in the FOV (field of view) of the camera. A motorized zoom lens is attached to the camera, and an infrared pass filter is used to reduce the amount of ambient visible light reaching the CCD. The system diagram is shown in Fig.2. Three glints, formed by the infrared light on the corneal of the eye, are shown in the captured image. The diagram of the captured eye image is shown in Fig.3. The eye images are sent to a computer through the imaging board, and are processed later by the computer to draw the information of the pupil and the glints. When the user holds the head still, the positions of the glints are not change, so the glints can be treated as the reference points. The pupil moves while the eye rotates for gazing at different points. The vector of op reflects gaze direction’s change owing to eye’s rotation. The gaze direction is determined by the relationship between the position of the pupil center and the corneal reflections. When the head moves in front and behind, the distance between glint2 and glint3 (d=d1+d2) changes; while the head moves horizontally, the ratio of d1 and d2 varies. The pan-tilt unit is controlled according to the central position between glint2 and glint3; the focus of the lens can be adjusted on the basis of the distance d, keeping the magnifying power of the optical system unchanged to overcome the influence of head movements in front and behind. The aperture of the lens and the LED’s luminous


617

power can be adjusted according to the intensity value of the glints; lens focusing is based on the gradient value of the pupil boundary points and the size of the glints. While the eye gets out of the FOV of the camera by head movements, making the camera has a large FOV, turn off LED2 and LED3, only turn on LED1. LED1 makes the diffusion of light on the retina to generate the bright-pupil effect; when LED2 and LED3 turn on, a dark pupil image is obtained. The difference image is used to detect the eye. Control signals realize the control through a parallel port of the computer.

Fig. 2. The diagram of the system and the gaze model

Fig. 3. The diagram of the captured eye image

5 Algorithm Description The pupil-glint vector is used to determine the gaze direction. This algorithm has the following phases:

618


5.1 Thresholds Extraction In order to detect the pupil and the glints accurately, it is crucial and difficult to find the appropriate threshold values owing to the movements of the head and the eye, external lighting changes and noises in the images. Some systems manually determined the thresholds. In this paper, finding the threshold values is automated. For the captured image a histogram has been calculated in which x-axis indicates intensity value of 0 through 255 and y-axis shows frequency for each intensity value. However, the histogram has small local noises in its envelope, which may result in wrong threshold value. Low pass filtering method is applied to remove this kind of local noises by averaging neighboring three points. The typical histogram of the eye image is shown in Fig.4.

Fig. 4. The typical histogram of the eye image

The histogram shows 3 peaks, which correspond to pupil, the remaining area and corneal reflection respectively. The valley between the first peak and the second peak is selected as the threshold value of the pupil. Glints, which correspond to the brightest regions in the eye image, can be obtained through thresholding. According to the positions of LEDs, camera and eye, we can estimate the approximate pixels of glints in the image, and then we search the histogram from right to the left. When the summarized pixels exceed the estimate pixels, the threshold value for corneal reflection is found. 5.2 Corneal Reflection Localization Given its small size and high brightness， the glints are approximately circles in the image. After binarizing the image, the positions of the glints are then given by the geometric center ( xc , yc ) of each reflection region in the image. Its size is initialized with

r = area pi .

We assume that the intensity profile of each glint follows a two-dimensional Gaussian distribution. If we find the radius r where the average decline in intensity is maximal and relate it to the radius with maximal decline for a Gaussian distribution, we can take the full extent of the reflection as 2.5r to capture 99% of the reflection profile. We find r through a gradient descent search that minimizes


∫ I (r + δ , x , y ,θ )dθ . ∫ I (r − δ , x , y ,θ )dθ c

c

c

c

619

(1)

Where δ = 1 , and I (r , x, y ,θ ) is the pixel intensity at angle θ on the contour of a circle defined by the parameters r, x and y . The position of the corneal reflection is computed with sub-pixel accuracy by computing an intensity-weighted centroid from this image. The experiments indicate that this method is robust and the accuracy is within 0.02 pixels for locating the glints even the threshold value changes in a big range. 5.3 Sub-pixel Edge Detection of the Pupil While other feature-based approaches apply edge detection to the entire eye image or to a region around the estimated pupil location, these approaches can be computationally wasteful as the pupil contour frequently occupies very little part of the image. In this paper, the edge points could be obtained by a 1D line search. This algorithm has the following phases: Estimation of the Pupil Center Based on the positions of the glints, a pupil-searching window is determined. All the pixels under the threshold value of the pupil are selected in the original image of the window and its mass center is computed, which is chosen as the estimated pupil center.

x pupil =

1 N 1 N xn , y pupil = ∑ yn . ∑ N n =1 N n =1

(2)

Searching of the Pupil Contour Then we detect the edge points along a limited number of rays that extend from this rough center of the pupil. Combining with the intensity value, gradient value of pixels and the smoothness of the boundary, we can find the pupil contour. This is shown in Fig.5.

Fig. 5. Edge detection of a pupil region by a 1D search

620


We search for edge points and find r through a search that maximizes

f (r ) = k0G (r ) − k1 ( I (r ) − Ip) − k 2 (r − rn−1 ) .

(3)

Where, I (r ) is the pixel intensity; Ip is the threshold value of the pupil; G (r ) is the gradient value of a searching point, it can be calculated through a 1D filter such as [-1 0 1]; rn −1 is the obtained radius of the edge point on the near search line,

k0 、 k1 、 k2 are the weighted parameters decided by experiments. Noises Elimination The pupil is always partially occulted by eyelid and eyelashes. When the head or eye moves, corneal reflection sometimes overlaps the pupil’s edge region. Affected by these noises in searching of pupil contour, some false edge points may be obtained; these false edge points can strongly influence the performance of the algorithm. Only the noises are eliminated, high accuracy results can be obtained. Ohno [4] used double-ellipse-fitting to eliminate noises， it is time consuming. We, instead, eliminate the overlapped noises from the pupil contour. For noises from eyelid or eyelashes, according to the gradient of the line linking two near points, we can determine if a point is a false edge point or not, those pixels far from the edge are disregarded. Sub-pixel Edge Detection Now, the boundary pixels of the pupil have been detected. To obtain the precise location of the pupil center, however, we need edge points at a sub-pixel level. Utilizing the local information of the boundary pixels on search line, the pupil contour can be located in sub-pixel accuracy. Calculate the gradient value of the adjacent pixels along the search line. Interpolate the gradients at sub-pixel positions with 1-dimentional second order polynomial interpolation. We use y to express the gradient value of a position that is close to the obtained edge point on the search line. The analytic relationship between y and the position’s coordinate x can be expressed as:

y = ax 2 + bx + c ,

(4)

Find the sub-pixel level edge point located at the sub-pixel position where y gets its maximum (minimum). x = −b

2a

.

(5)

5.4 A Randomized Method of Ellipse Fitting Given a set of candidate feature points, the next step of the algorithm is to find the best fitting ellipse. While other algorithms commonly used least squares fitting of an ellipse to all the feature points. However, experiments indicate that it is not robust and sensitive to big noises. A few feature points, which are not on the pupil contour, dramatically reduce the quality of the fit and affect the accuracy of the results. So, more robust


621

method needs to be developed. Like Theil-Sen、 Repeated Median[11],but these two methods are computationally wasteful and hard to realize. In this paper, a new method of ellipse fitting is proposed. It is robust and easy-to-implement． Given that discrete data always require 6 points to define an ellipse. So, for an edge point in the detected feature set, five samples are randomly chosen from this point’s supplement set. The six different points form a sample sub-set. Then we calculate it. If the result indicates it is not an ellipse or the parameters of an ellipse is not reasonable, we chose samples and calculate it again; if the result is rational, save the parameters. This algorithm makes same processing to every point. Finally, takes the median value of the saved parameters as the estimated ellipse parameters. 5.5 Map to Screen Coordinates The view vector needs to be mapped to screen coordinates. When a user gazes at a screen point ( Xs, Ys) , the pupil center’s coordinate ( x p , y p ) and the glint ’s coordinate

( xg , y g ) in the image are obtained. Then the view vector (shown as op in

Fig.3) is given as following:

dx = x p − xo , dy = y p − yo .

(6)

The transformation from the view vector to screen coordinate is taken to be a pair of simple second order polynomial as: Xs = a0 + a1(dx) + a2(dy) + a3(dx)(dy) + a4(dx) 2 + a5(dy) 2

;

(7)

Ys = b0 + b1( dx) + b 2( dy ) + b3( dx )(dy ) + b 4( dx) 2 + b5(dy ) 2

.

(8)

Where ai, bi (i = 0 ~ 5) need to be found. Given several known points on the screen, which serve as the calibration points. We calculate the view vector and draw the transformation coefficients. When the user moves horizontally, we can obtain the parameters in different positions. It can be used to calculate gaze directions in successive frame by interpolation.

6 Experimental Result Some experiments have been carried out to show the feasibility of the proposed method including the condition of natural head movements. The monitor has a 17’’screen; the resolution of the screen is 1280×1024 pixels. There are 6×5 points in the screen of the monitor. The user sat 50~80cm in front of the screen and looked at one of the target points. This system estimated the position where the user looked at, and then calculated the average error. The experimental result is shown in Table1. The result indicates that the average error is small when the head is still. We find that the average error is about 1-degree visual angle under natural head movements. Also, there is a higher horizontal accuracy in this method is observed.

622

R. Liu, S. Jin, and X. Wu Table 1. Result of the experiments

User 1 2 3 4 5

Average error (pixels) Hold the head still Natural head movement X Y X Y 6.67 11.34 12.05 23.75 6.53 10.58 16.26 24.64 7.12 15.26 19.97 34.47 6.96 17.75 16.64 32.96 9.89 16.11 15.52 28.38

Also the system has been tested for dynamic gaze tracking by experiments. In the experiments, there was a moving small ball on the screen, and the user’s eyes gazed at and followed the moving ball. In experiment 1, a ball moved from left to right, its y coordinates were measured. The measurement error was between 13.3 pixels and 43.2 pixels, the average was 31.7 pixels; in experiment 2, a ball moved from up to down, its x coordinates were measured. The measurement error was between 12.8 pixels and 35.7 pixels, the average was 26.1 pixels. Then the accuracy of gaze detection was calculated. Assuming that the distance between the user and the screen was 65 cm, this is equivalent to a view angle of 1.0 degree. We also found that the x-coordinate accuracy is better than y coordinate.

7 Conclusion In this paper, an eye-gaze tracking method, which can make this system be adaptive to external lighting changes and to the user’s head movements, is presented to improve the classical PCCR eye-gaze tracking technique. The system can run in real time at video rate (25frames/second) and a gaze accuracy of 1 degree is observed. While enjoying the advantages of high accuracy and robustness, our system uses only one camera to track eye-gaze and can work under natural head movements. Therefore, the user can move his head freely while using our eye tracking system. At the same time, there are still some works should be done to improve the performance of our system; such as the number of calibration procedures should be minimized. In future work we will mainly focus on making the calibration procedure simple. We believe, after further improvement, our system will find many applications, including human computer interaction and assistance of people with disability. Acknowledgments. The Scientific Research and Development Fund Program of Tianjian Higher Education, China, under Grant No. 20051017, support this work.

References 1. Akira Sugioka, Yoshinobu Ebisawa, Masao Ohtani: Noncontact Video-Based Eye-Gaze Detection Method Allowing Large Head Displacements. In: 18th Annual Internat. Conf.of the IEEE Engineering in Medicine and Biology Society, Amsterdam (1996) 526-528. 2. C.H. Morimoto, D. Koons, A. Amir, M. Flickner: Pupil Detection and Tracking Using Multiple Light Sources. Image and Vision Computing, 18 (2000) 331-335.


623

3. Wen Gang: Gaze Tracking Using One Fixed Camera. Seventh International Conference on Control, Automation, Robotics And Vision, Singapore(2002)1409-1414. 4. Takehiko Ohno, Naoki Mukawa, Atsushi Yoshikawa: FreeGaze: A Gaze Tracking System for Everyday Gaze Interaction. In Proc. of Eye Tracking Research and Applications Symposium (ETRA), New Orleans (2002) 125-132. 5. Lim Choon Kiat,Surendra Ranganath: One-Time Calibration Eye Gaze Detection System. International Conference on Image Processing (ICIP), Singapore (2004) 873-876. 6. Yoshinobu Ebisawa: Realtime 3D Position Detection of Human Pupil. VECIMS 2004IEEE International Conference on Virtual Environments, Human-Computer Interfaces, and Measurement Systems, Boston, MA, USA (2004) 8-12. 7. Zhiwei Zhu, Qiang Ji: Eye and Gaze Tracking for Interactive Graphic Display. Machine Vision and Applications 15(2004) 139-148. 8. Dong Hyun Yoo, Myung Jin Chung: A Novel Non-intrusive Eye Gaze Estimation Using Cross-ratio under Large Head Motion. Computer Vision and Image Understanding 98(2005) 25-51. 9. B. Noureddin, P.D. Lawrence, C.F. Man: A non-contact device for tracking gaze in a human computer interface. Computer Vision and Image Understanding 98(2005) 52-82. 10. Zhao Lejun, Wang Chaoying, Chen Huaichen: Research on Calibration and its Algorithm for Head-Mounted Eye Movement Measurement Recorder. Journal of Xidian University， 1998, 25(5): 606-610. 11. Paul L Rosin: Further Five-Point Fit Ellipse Fitting. Graphical Models and Image Processing, 1999, 61(5): 245-259.

High-Quality Shear-Warp Volume Rendering Using Efficient Supersampling and Pre-integration Technique Heewon Kye1 and Kyoungsu Oh2 1

School of Electrical Engineering and Computer Science, Seoul National University, San 56-1 Shinlim-dong Kwanak-gu, Seoul 151-742, Korea [email protected] 2 Dept. of Media, Soongsil University 156-743 Seoul, Korea [email protected]

Abstract. As shear-warp volume rendering is the fastest rendering method, image quality is not good as that of other high-quality rendering methods. In this paper, we propose two methods to improve the image quality of shear-warp volume rendering. First, the supersampling is performed in an intermediate image space. Then is proposed an efficient method to transform between the volume and the image coordinates at the arbitrary ratio. Second, the preintegrated rendering technique is utilized for shear-warp rendering. To do this, a new data structure called overlapped min-max block is used. Using this structure, the empty space leaping can be performed so the rendering speed is maintained even though when the pre-integrated rendering is applied. Consequently, shear-warp rendering can generate on high-quality images comparable to those generated by the ray-casting without degrading the speed. Keywords: Shear-warp rendering, high-quality volume rendering, preintegration, supersampling.

1 Introduction Volume rendering is a technique to visualize information from three-dimensional arrays of digital data, and is mainly used in medical and scientific applications [1]. Earlier volume rendering implementations require a high-performance workstations or special-purpose hardware. After many researches [2-8] to alleviate the weaknesses of rendering algorithms, general-purpose PCs are now capable of presenting quite satisfactory results. Shear-warp volume rendering [2] is a well-known rendering method. It is faster than others methods except for those which uses graphic hardware [3]. However, it has not been preferable in recent medical applications because it cannot create highquality image. We propose new methods that improve image quality of shear-warp rendering without noticeable performance loss. Generally, image-space supersampling and/or object-space supersampling can be used to improve volume rendering image quality. More rays are cast from a pixel in the image-space supersampling and more samples are taken along the ray at smaller Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 624 – 632, 2006. © Springer-Verlag Berlin Heidelberg 2006

High-Quality Shear-Warp Volume Rendering

625

intervals in the object-space supersampling [4]. However, it is hard to find efficient methods that have been presented for image- and object-space supersampling in shear-warp rendering. To apply image-space supersampling, Koo et al. generated multiple templates [4] for each ray. Kim et al. proposed one dimensional supersampling method for anisotropic volume data [5]. These methods limit the supersample ratio to positive integer for simplicity while our method is not limited. Moreover, the object-space supersampling makes the algorithm complicated caused by the tri-linear interpolation. Sweeney et al. tried to supersample in both image-space and object-space [6]. They stored density and the shading information instead of classified color for run-length encoded volume. However, their method duplicates the rendering time while an object-space supersampling is applied. Furthermore, it cannot change classification interactively though it uses postclassification algorithm. The special purpose of the hardware such as VolumePro board [7] enables the supersampling based on shear-warp factorization. In this study, by the revision of the shear-warp volume rendering, we generate high-quality images compared to using high-quality rendering method such as ray tracing [8]. Fast rendering speed of shear-warp rendering can be conserved. The method applied in this research is composed of two phases. The first is efficient supersampling method on the image space. In this research the volume data in the intermediate scanline order is rendered which could reduce the transform overhead between object and image coordinates. As a result, we can perform efficiently the supersampling with an arbitrary sampling ratio. The second is exploiting preintegration volume rendering [9] which has same the effect with object-space supersampling. Previously, the pre-integration rendering drops down the shear-warp rendering speed. To solve this problem, a new data structure called overlapped minmax block is proposed. By using this data structure the rendering can be accelerated owing to empty-space leaping. As the effects as image and object supersampling respectively of these two methods, high-quality images can be generated with the complementation of both methods.

2 Efficient Image-Space Supersampling for Shear-Warp Volume Rendering The acceleration techniques for shear-warp rendering are empty-space leaping and early-ray termination [2]. Empty-space leaping is performed in object space and earlyray termination is performed in intermediate image space. For simplicity, traditional shear-warp rendering assumes that the pixel size of the intermediate image is equal to the voxel size of the volume data, so these two acceleration techniques have been performed without transformation overhead between object- and intermediate imagecoordinate. While the image-space supersampling is applied, a fast transformation method is needed because the pixel size is not equal to the voxel size. In order to accelerate the transformation between coordinates, then a volume is rendered on a scanline by scanline basis in intermediate image coordinate. As an intermediate scanline is parallel to an object scanline, the weight value for y-direction becomes a constant for one scanline. In other words, is possible have in consideration only the x-coordinate. It is explained in Fig. 1.

626

H. Kye and K. Oh

Shear-warp rendering { for each slice zv in [0, number of volume slice-1] (y0v, y1v) = Get_Image_Scanline_Range (zv) for each scanline yi in Intermediate_Scanline_Range yv = Inverse Mapping(yi) (x0v, x1v) = Get_Voxel_Range (yv) Process_One_Intermediate_Scanline(x0v, x1v) } Fig. 1. Pseudo-code for the image-space supersampling algorithm of shear-warp rendering. For each slice, the range in intermediate image coordinate can be obtained. Then, a rendering scanline by scanline in intermediate image coordinate could be performed.

Each slice of volume data is transformed to intermediate the image coordinate, and is possible to obtain a bounding rectangle for each slice. Each scanline included in bounding rectangle is transformed to an object coordinate and is possible get a weight value for y-direction. After that, each scanline is rendered considering the xdirection of supersampling. The processing of one scanline is illustrated in the Fig. 2. For the simplicity, interpolation between scanlines is ignored without loss of generality.

Fig. 2. An example for rendering one scanline. The algorithm is mainly processed in image space except for skipping transparent voxels (step 5), so that transformation between volumeand imagecoordinate is only occurred at boundary of transparent voxels.

High-Quality Shear-Warp Volume Rendering

627

One scanline is rendered as follows: the first voxel is projected to image scanline (1) and find first transparent pixel is founded (2). To fill the pixel, is sampled to the object coordinate (3) but the opaque pixels are skipped (4). If a sample point is between adjacent transparent voxels, is possible skip transparent voxels (5). The last transparent voxel is projected to image scanline (6). Now is able to finish one scanline at the position that the last voxel of the volume scanline which is projected to (7). To Summarize, pseudo-code is presented in Fig. 3. Process_One_Intermediate_Scanline(x0v, x1v) { pixel_start = RoundingUp(Mapping(x0v)) pixel_end = RoundingOff(Mapping(x1v)) for (xi = pixel_start; xi 0 r

(1)

Saturation-weighted hue statistic is used as the statistical descriptors. Let (θi2 , si ), i = 1, . . . , n be n pairs of observations sampled from a population of hue and associated saturation values. Vector components from hue in Cartesian coordinates is given by h = (cos θiH , sin θiH )T . Then saturation-weighted resultant of hue components is defined as Cs =

n

si cos θiH , Ss =

i=1

n

si sin θiH

(2)

i=1

The mean of resultant chrominance vector over n samples is ¯ cn = (Cs /n, Ss /n)T

(3)

The variation of observed chrominance vector c = sh from its mean cn is defined as a Euclidean distance in a chromatic plane D = (¯ cn − so ho )T (¯ cn − so ho ) (4) where so and ho are the observed saturation and hue vector respectively. A simple background subtraction algorithm following [15] has been devised in [2]. A background pixel is modeled as the luminance mean μy and its standard deviation σy , with mean chrominance vector cn and mean Euclidean distance σD between cn and the observed chrominance vector. Foreground decision is taken if the following conditions are satisfied: |(y − μy )| > ασy ∧ c¯n − c > ασD

(5)

where α is the foreground decision threshold set between 2 and 3.5. 2.2

Gaussian Mixture Model

Gaussian Mixture Model is pixel-wise segmentation model. Each pixel is modeled as independent random variable from its neighborhood. The value of a pixel over

GMM in Improved HLS Color Space for Human Silhouette Extraction

735

a period of time can be modeled as a random variable X. X is an n-dimensional space vector (usually 3 for RGB color). This pixel value process X is assumed to be a mixture of K Gaussian densities (K usually assumed constant between 3 and 7) with parameter sets θk for each state density k, such that: fX|k (X|k, θk ) =

1

1

(2π)n/2 |Σ

k

|1/2

e− 2 (X−μk )

T

Σ−1 k (X−μk )

(6)

where μk and Σk are the mean and the covariance matrix of the kth density respectively. Elements of X are assumed independent so that Σk may be represented by n-dimensions diagonal matrix of variance σk2 . The density parameter is defined as θk = {μk , σk } for a given k and total parameters for existing K densities become Φ = ω1 , . . . , ωK , θ1 , . . . , θK . The parameter ωKk represents a priory probability of density k to appear in pixel process and k=1 ωk = 1. Distribution of X can be modeled as sum of Gaussian mixture fX (X|Φ) =

K

P (k)fX|k (X|k, θk )

(7)

k=1

where P (k) = ωk . All the parameters in Φ are needed to be estimated from observation of X in parallel with the estimation of hidden state k. Finding maximum likelihood from incomplete data in mixture model could be solved using expectation-minimization (EM) algorithm [13][3][2]. The GMM algorithms that we used for foreground segmentation can be outlined as follows. We will assume single dimension of X as every channel of color in RGB is assumed to be independent. Initially, for all K Gaussians, variances are set to high values and weights to small values. Current pixel value Xt is compared to K Gaussian distributions. Match is defined if current value is similar to mean value of a distribution within distant threshold λσk , where λ usually between 2 and 3. This can be expressed as 2 (Xt − μk,t )2 < λ2 σk,t

(8)

If a match distribution is not found, a new distribution is created with high variance and low weight. Distributions with lowest confidence rank (ωk , t/σk , t sorting criterion) is replaced by a newly created distribution. For a matched Gaussian, its mean and variance are updated with update algorithms from [9]. For initialization where we expected sufficient statistics, update equations are: ω ˆ k,t = ωk,t + μ ˆk,t = μk,t + 2 σ ˆk,t

=

2 σk,t

1 N +1 (P (k|Xt , Φ) P (k|Xt ,Φ) (Xt N +1 P (k|Xi ,Φ)

i=1

+

− ωk,t ) − μk,t )

P (k|X ,Φ) ((Xt − μˆk,t) P (k|Xt ,Φ)

T

N +1 i=1

(9) (Xt − μ ˆk,t ) −

2 σk,t )

i

where P (k|Xt , Φ) =

1 match 0 otherwise

(10)

736

N.A. Setiawan et al.

K and k=1 P (k|Xi , Φ) is the sum of weight K Gaussian distributions. After the initial phase, update equations switches to L recent window update equations ω ˆ k,t = ωk,t + μ ˆk,t = μk,t + 2 2 σ ˆk,t = σk,t +

1 L (P

(k|Xt , Φ) − ωk,t ) P (k|Xt ,Φ)Xt 1 − μk,t L

ω ˆ k,t P (k|Xt ,Φ)(Xt −ˆ μk,t )2 1 − L ω ˆ k,t

2 σk,t

(11)

The K distributions are ordered by criterion ωk , t/σk , t and the first B distributions are used as a model of the background b

B = arg min ωk > T (12) b

k=1

with T as the threshold for minimum fraction of the background model.

3

Gaussian Mixture Model in IHLS Color Space

Using IHLS statistics in mixture model is straightforward with statistical descriptors luminance and saturated-weighted hue as explained in Sect. 2.1. In this section we explain our background model, shadow detection and background maintenance. 3.1

Implementing IHLS in Mixture Model

Luminance y is one-dimensional GMM with density parameters θy = {μy , σy } which have been explained in Sect. 2.2. While chrominance vector, using circular statistics, has density parameters θc = {¯ cn , σD }. Match criteria for chrominance can be expressed in Euclidean distance as in (4). 2 ¯ cn − c ξ , the point pk is preserved for further process. Otherwise, we will regard it as outlier and remove it from point set Ps (see Fig. 1).

Fig. 1. Density function filtering. Left is the bosom of the raw Stanford bunny. Right is the result of density function filter the noisy point data. The outliers have been removed well.

2.3 Curvature Penalty

Weight is frequently used to give the different feature point a special influence. In this part, we will consider assigning an optimized weight to each noisy point data. Firstly we assess the normals through the correlation matrix Mk of neighboring points. Mk =

∑

pi ∈SPT ( qk )

( pi − qk )( pi − qk )t .

(3)

Here qk is the center of the neighborhood pi. Its three eigenvalues are {λ1, λ2, λ3}, λ1 0, and then p is inside the solid. If f (p) = 0, then p is on the boundary of the solid. And if f (p) < 0, then p is outside the solid. Let f1 = 0 and f2 = 0 to define two implicit surface, f3 is the result implicit surface by Boolean operation of f1 and f2 . Wyvill [17] presented a definition of implicit surface Boolean operation by means of limit form of functions fusion. 1

f3 = (f1n + f2n ) n .

(1)

Pasko proposed a general form of Boolean operations by functions fusion [18]. 1

1

f3 = (f1 + f2 + α(f12 + f22 ) 2 )(f12 + f22 ) 2 .

(2)

In Eq. (2), α ∈ [−1, 1], when α = −1, the equation indicates intersection of f1 and f2 , and when α = 1, indicates the union of f1 and f2 . Although the Eq. (2) presents a uniform mode of Boolean operations and functions fusion for implicit surfaces, but it is inconvenient to implement. In order to decrease the computation cost and convenient to implement, we present following Boolean operations for implicit surfaces. Union: f3 = f1 ∪ f2 = max(f1 , f2 ) . (3) Intersection: f3 = f1 ∩ f2 = min(f1 , f2 ) .

(4)

842

X. Qin, X. Yang, and H. Zheng

Subtraction: f3 = f1 − f2 = min(f1 , −f2 ) .

(5)

Fig. 1 shows Boolean operation results from two spheres defined by implicit function.

Fig. 1. Boolean operation results from two spheres defined by implicit function. (a)Union; (b)Intersection; and (c)Subtraction.

4

Interactively Cutting and Alignment for Mesh Models

In order to implement the cut-and-paste operation of meshes, part of the meshes of source model should be cut down and then be pasted to the target model. So a convenient and quick interactive method is necessary. Since the mesh model will be converted to point model, on which Boolean operations will be performed, the cutting does not need to be accurate. What’s more, the cutting edge does not need to be plain. Here, we use simple polyhedron such as plains, cubes and balls to cut the mesh model. During the cutting process, inside-and-outside test is done for vertexes and triangle facets of the mesh model, vertexes and facets inside the cutting cube are saved, thus complicated, time-consuming of facet-tofacet and facet-to-line segment intersection testing are unnecessary. For example, in Fig. 2, balls are used to cut the rabbit model interactively, Fig. 2(b) is part of the meshes cut down from Fig. 2(a). The part of meshes cut from source model should be pasted on target mesh. We designed interactive tools such as moving, rotation, zooming and so on, with manual interaction, it is convenient to aligning the mesh. Fig. 2(c) is the result of alignment meshes.

Fig. 2. Cutting and aligning for mesh models. (a)Interactively cutting for mesh model; (b)Result model after cutting; (c)aligning the two models.

Implicit Surface Boolean Operations Based Cut-and-Paste Algorithm

5 5.1

843

RBF Implicit Surface Converting and Boolean Operation for Mesh Models Converting Mesh Model to Point Model

In order to convert mesh model into point model, we can just use vertexes of mesh model as sample points of point model. The accuracy of point model is related to the density of sample points. For dense mesh model, it can be converted to point model directly, for sparse mesh model, we can improve its density by subdividing the mesh, then converted the subdivided mesh into dot model. Hence, we use Loop method [19] to subdivide the mesh. Fig. 3 is an example of subdivision of sparse mesh and its convention into point model. Fig. 3(a) is a source model. Fig. 3(b) is the result of mesh after twice Loop subdivision. Fig. 3(c) is the corresponding point model of Fig. 3(b).

Fig. 3. Subdividing spare mesh model and converting it to point model. (a)Original model; (b) subdivided twice with Loop subdivision method; (c) convert into point model.

5.2

Implicit Surface Transforming and Boolean Operations on Point-Sampled Models

Transforming point model to implicit function representation, obviously it is a multi-variant scatter data interpolation problem. We use variational interpolation to obtain the implicit surface of the point model, the form is f (p) = 0. Considering that the interpolated surface is smooth, in CAGD, smooth property of surface can evaluate by thin-plate energy function [20]. 2 2 2 E= [fxx (x, y) + fxy (x, y) + fyy (x, y)]dxdy . (6) Ω

Where fxx , fxy , fyy are second partial derivatives of f , Ω is region of interpolate points. Because the energy E essentially is a square curvature evaluate of function f in region Ω. When f is rumpled, the energy E is increase. So, minimize the energy can make the surface f smooth . The above energy function can easily to extend to high dimension case. The equation’s general solution form is linear combination of radial basis functions φ(p) and a linear term Q(p). Its general expression is:

844


f (p) =

n

λj φ(|p − pj |) + Q(p) .

(7)

j=1

where Q(p) is a polynomial of degree at most a specified number, λj is a realvalued weight, φ(•) is a radial basis, and | • | is a Euclid norm. In order to solve the weights , we can convert to solve the following linear system: ⎛ ⎞⎛ ⎞ ⎛ ⎞ φ11 φ12 . . . φ1n x1 y1 z1 1 λ1 h1 ⎜ φ21 φ22 . . . φ2n x2 y2 z2 1 ⎟ ⎜ λ2 ⎟ ⎜ h2 ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟ ⎜ .. .. . . .. .. .. .. .. ⎟ ⎜ .. ⎟ ⎜ .. ⎟ ⎜ . ⎟⎜ . ⎟ ⎜ . ⎟ . . . . . . . ⎜ ⎟⎜ ⎟ ⎜ ⎟ ⎜ φn1 φn2 . . . φnn xn yn zn 1 ⎟ ⎜ λn ⎟ ⎜ hn ⎟ (8) ⎜ ⎟⎜ ⎟ = ⎜ ⎟ . ⎜ x1 x2 . . . xn 0 0 0 0 ⎟ ⎜ c0 ⎟ ⎜ 0 ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟ ⎜ y 1 y 2 . . . y n 0 0 0 0 ⎟ ⎜ c1 ⎟ ⎜ 0 ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟ ⎝ z 1 z 2 . . . z n 0 0 0 0 ⎠ ⎝ c2 ⎠ ⎝ 0 ⎠ 1 1 ... 1 0 0 0 0 c3 0 where denoting hj = f (pj ), φij = φ(|pi − pj )|, Q(p) = c0 x + c1 y + c2 z + c3 . In Eq. (8), the coefficient matrix is obviously real symmetric, and with proper selection of basis functions it can be made positive-definite. Thus, a solution always exists to the linear system [21]. Use the variational interpolation described above, we can construct a scalar distance field f (p) = r from point-sampled model. This is a typical implicit surface form. When r = 0, f (p) = r is the interpolated surface of the point model. After obtaining the implicit surface of the point model, we can realize Boolean operations for point models by using the implicit surface Boolean operation method described in section 3. Those points in two point models that satisfy the equation f3 (p) = 0 are the points of Boolean operations result model. Fig. 2 indicates the Boolean operation of two point models. 5.3

Polygonization for Boolean Operation

Because after the point model is converted into implicit function, the result of Boolean operation is an implicit surface, we can get the mesh after Boolean operation by polygonizing the implicit surface. For the polygonization of implicit

Fig. 4. Boolean operation on point models. (a) Venus head model;(b) Bunny model; (c) Union of (a) and (b); (d) Subtraction of (a) and (b); (e) Polygonization result of (c); (f) Polygonization result of (d).


845

surface, Marching cubes [22] and Bloomenthal [23] are two well known and classical algorithms. In this paper, we choose the latter. Fig. 4(c) and 4(d) is the result of Boolean operation on point models, Fig. 4(e) and 4(f) is the result of polygonization.

6

Experiment Results

We develop a prototype system to test our algorithm on PC with CPU of 2.4 GHZ, RAM of 512M, and using the platform of Microsoft Visual C++ 6.0. Fig. 5 shows that the wings of felines model are cut down and pasted on horse model. Fig. 5(b) is the model after trimming. Fig. 5(c) is the mesh model after aligned. Fig. 5(d) is the point models convert from the two mesh models. Fig. 5(e) is the point model after union operation. Fig. 5(f) is the result of polygonization after union.

Fig. 5. Cut-and-paste operation on felines and horse models. (a) felines model; (b) model after cutting;(c) mesh model alignment; (d) Convert into point model; (e) point model after Boolean operation; (f) Polygonization result.

Fig. 6. Cut-and-paste operation on mesh models. (a) Venus head model; (b) Face model; (c) Result of Boolean operation; (d) Surface model after polygonization.

Fig. 6, Fig. 7 and Fig. 8 are results of Boolean operations on mesh models. Fig. 6(a) and 6(b) are original models. Fig. 6(c) is the point model after union operation. Figure 6(d) is the polygonizing surface model from Fig. 6(c). In this way, we can cut part of mesh from one model and paste it on another model. Fig. 7 shows an ox head fusion with horse head. Fig. 8 shows fusing Venus head and body. Table 1 shows the running time of Boolean operations on point models in Fig. 4 to Fig. 8, including RBF distance fields constructing time and polygonizing time for Boolean operation result represented by implicit surface. From Table 1 we can easily see that the time efficiency of the algorithm can acceptable.

846


Fig. 7. Fusing the ox head with horse body. (a) ox model; (b) horse model; and (c) result model of fusion.

Fig. 8. Fusing the Venus head and body. (a) Venus head model; (b) Venus body model; and (c) result model of fusion. Table 1. Running time of Boolean operation on point models Figure Points of Points of RBF RBF Polygonizing model A model B constructing constructing time time(model A) time(model B) Fig. 4 35947 8266 6.141s 1.203s 12.172s Fig. 5 7251 7689 0.813s 1.125s 3.031s Fig. 6 8268 6461 1.219s 0.841s 3.422s Fig. 7 6668 6195 0.843s 0.817s 2.859s Fig. 8 8268 4509 0.819s 0.704s 3.121s

7

Conclusions

A novel cut-and-paste algorithm for mesh models is presented. The algorithm use Boolean operation for implicit surface to fusion two mesh models. Using the approach, the fusion region is smooth, complex harmonic map and local parameterization for fusion region is not necessary. The algorithm can used in the case of genus is not zero of the models. Because the Boolean operation is not directly with the mesh models, so mass of time consuming intersection test for facet-to-facet, and facet-to-line segment is avoid. The experiments show that good results can be obtained with the algorithm. This approach belongs to the part of novel method of geometry modeling by examples with cut-and-paste editing. The method constructs a new mesh model from two or more existed mesh models by cutting some useful parts from source


847

models and pasting to target model. The new approach can be used to create lots of special exaggerated modeling effects for computer animation. Acknowledgments. This research is supported by Zhejiang provincial Nature Science Foundation of China (Grant No. Y104341 and Y105303), and Opening Research Fund of State Key Laboratory of CAD&CG, Zhejiang University.

References 1. Barghiel H., Bartels R., Forsey D.: Pasting spline surfaces. In: Mathematical Methods for Curves and Surfaces, Vanderbilt Univ. Press,(1994)31-40 2. Biermann H., Martin I., Bernardini F., et al.: Cutand-paste editing of multiresolution surfaces. In: Proc.of ACM SIGGRAPH’02,(2002)312-321 3. Conrad B., Mann S.: Better pasting via quasiinterpolation. Curve and Surface Design: Saint-Malo 1999, Vanderbilt Univ. Press, (2000)27-36 4. Kanai T., Suzuki H., Mitani J., et al.: Interactive mesh fusion based on local 3D metamorphosis. In: Proc. of Graphics Interface’99, June, (1999)148-156 5. Furukawa Y., Masuda H., Miura KT., et al.: Cut-and-Paste Editing Based on Constrained B-spline Volume Fitting. In: Proc. of Computer Graphics International, (2003)222-225 6. Bartels R. and Forsey D.: Spline overlay surfaces. In: Technical Report CS-92-08, University of Waterloo, waterloo, Ontario, Canada N2L 3G1, (1991) 7. Chan L., Mann S., and Bartels R.: World space surface pasting. In: Proc. of Graphics Interface, May ,(1997)146-154 8. Ohtake Y., Belyaev A., Alexa M., et al.: Multi-level partition of unity implicits. ACM Trans. Graph. Vol.22(3):(2003)463-470. 9. Mann S. and Yeung T.:Cylindrical surface pasting. In: Geometric Modeling’99, (1999)233-248 10. Museth K., Breen D. E., Whitaker R.T., et al.: Level set surface editing operators. ACM Transactions on Graphics, Vol.21(3):(2002)330-338 11. Ranta M., Inui M., Kimura F., M¨ antyl¨ a M.: Cut and paste based modeling with boundary features. In: Rossignac J., Turner J., Allen G.,(eds.): Proc. of the 2nd ACM Solid Modeling. ACM Press, (1993)303-312 12. Singh K., Parent R.: Joining polyhedral objects using implicitly defined surfaces. The Visual Computer, Vol.17(7):(2001)415-428 13. Pedersen HK.: Decorating implicit surfaces. Computer Graphics, Vol.29(3):(1995)291-300 14. Kanai T., Suzuki H., Kimura F.: Three-Dimensional geometric metamorphosis based on harmonic maps. The Visual Computer, Vol.14(4):(1998)166-176 15. Kanai T., Suzuki H., Kimura F.: Metamorphosis of arbitrary triangular meshes. IEEE Computer Graphics and Applications, Vol.20(2):(2000)62-75 16. Liu G., Jin XG., Feng JQ.,et al.: Montage mesh musion. Journal of Software, Vol.14(8):(2003)1425-1432 17. Wyvill B., Gallin E., and Guy A.: Extending the CSG tree. warping,blending and Boolean operations in an implicit surface modeling system. Computer Graphics Forum, Vol.18(2): (1999)149-158 18. Pasko A., Adzhiev V., Sourin A.,et al.: Function Representation in Geometric Modeling: Concepts, Implementation and Applications . The Visual Computer, Vol.11(8): (1995)429-446.

848


19. Loop C.: Smooth subdivision surfaces based on triangles. USA: Department of Mathematics, University of Utah, (1987) 20. Zhu XX.: Free form Curve/Surface Modeling. Beijing: Science Press, (2001) 21. Morse BS., Yoo TS., Rheingans P., et al.: Interpolating Implicit Surfaces From Scattered Surface Data Using Compactly Supported Radial Basis Functions. In: Proc. of Shape Modeling Conference, Genova, Italy, May, (2001) 89-98 22. Lorensen WE., Cline HE.: Marching Cubes:A high resolution 3D surface construction algorithm. Conputer Graphics, Vol.21(4):(1987)163-169 23. Bloomenthal J.: An implicit surface polygonizer. In: Graphics Gems IV, New York: Academic Press, (1994) 324-350

Meshes Simplification Based on Reverse Subdivision Jian-ping Ma, Xiao-nan Luo, Shi-jia Ma, and Guifeng Zheng Institute of Computer Application, Sun Yat-sen University, Guangzhou 510275, China [email protected]

Abstract. In Virtual Reality, 3D graphics is generally illustrated by meshes of 3D geometric model. This paper provides a novel algorithm to construct progressive meshes based on the reverse subdivision. The dense mesh is simplified into coarse one by decimating the redundant information level by level. Loop subdivision scheme, which is an approximating scheme, is adopted as interpolatory one in the simplification process. The implementation of our algorithm consists of three key steps, splitting, predicting and updating. The progressive mesh can be reconstructed lossless. The experiments have shown that the new algorithm is efficient and more faster than previous related techniques.

1

Introduction

3D Graphics is a very important technology in Virtual Reality, wherein the 3D graphics is usually illustrated by meshes of 3D geometric models[1], and often it needs real-time rendering with Level Of Detail(LOD)[2]. Two methods are often used to construct the geometric meshes. One is through 3D laser scanner, in which the meshes of the model are dense and arbitrary. Another is from subdivision scheme, in which the mesh is a Remesh[3]. A Remesh is of subdivision connectivity and provides a user-friendly environment for mesh editing and splitting. Subdivision scheme is an efficient method, which generates geometry meshes from initial coarse ones. According to a regular rule, a coarse mesh is split into the dense mesh and a smooth surface with subdivision connectivity is obtained after a limited process. The meshes from 3D scanner are highly detailed and expensive to represent, store, transmit and manipulate. So it is mandatory to simplify the dense meshes. The key problem in Virtual Reality application is what kind of meshes can represent graphics and can they be reconstructed efficiently in short time. Many researchers have considered the problem of the geometry surface model simplification[4][5]. In the mesh simplification, frequently used algorithms are vertex decimation[6], iterative edge contraction[7], and triangle decimation[8]. The aim of most approaches is to simplify arbitrary triangle meshes into the Remesh[4][9]. Among these methods, Mongkolnam[5] has used the Quadric Error Metrics (QEM)[10] to compress the arbitrary meshes. Then the meshes are fitted to the original mesh by adjusting to the subdivision limit positions[11]. Finally a remesh is obtained by performing a displacement operation[9]. Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 849–860, 2006. c Springer-Verlag Berlin Heidelberg 2006

850

J.-p. Ma et al.

The concept of progressive mesh was first introduced by Hoppe[12]. Khodakovsky proposed a progressive geometry compression methods based on smooth semi-regular meshes and wavelet transforms[13]. Luo applied the progressive meshes on mobile device based on reverse butterfly subdivision scheme[14]. This has partly solved the rendering speed in graphics. But the vertex affine combination of butterfly scheme is relatively complex. Thereby, it reduces the speed of mesh simplification and reconstruction. This paper presents an algorithm for constructing progressive meshes based on the reverse subdivision. Loop subdivision scheme[15], which is approximating scheme, is adopted as interpolatory one in the simplification process, without adjusting the vertex points position. Three key steps: splitting, predicting and updating are implemented in our simplification algorithm. The algorithm which is based on reverse subdivision can be used in many stationary subdivision schemes, such as butterfly and Loop. Practically, the algorithm using Loop scheme is faster and more efficient than using butterfly scheme. The rest of this paper is organized as follows. Section 2 introduces Loop subdivision scheme and fundamental idea of new algorithm which is acquired from this scheme. Section 3 reviews the principle and algorithm to generate the progressive meshes based on the reverse subdivision. Section 4 describes the progressive meshes reconstruction process. In the last two sections, some examples are provided followed by further analysis and conclusion.

2

Loop Subdivision Scheme and Basic Idea

Subdivision surface produces a finer mesh from a given mesh according to a refinement rules. Most known refinement schemes are based on triangle or square tilings. The well-known triangle refinement schemes are butterfly subdivision[16], √ 3 subdivision[17] and Loop subdivision[15]. Loop subdivision is an approximating scheme. A vertex is inserted at each edge, the original vertices are allocated to new positions accordingly. Then the new vertices form a new topology, as a consequence, each triangle face is split into four faces. In a subdivision level, see Fig.1, the triangle with light lines is a mesh, v j is vertex set of level j. ej+1 denotes edge point and v j+1 represents vertex point. The sets of both edge points ej+1 and vertex points v j+1 are the new vertices generated by affine combination[18] of v j . As a result, one triangle face is split into four faces, denoted with the darker lines. By repeating the above steps, a smooth surface modeling could be achieved. Both the coordinates of edge point ej+1 and vertex point v j+1 can be calculated from subdivision mask[15]. It can be expressed by following formula. ve =

3 1 (v1 + v2) + (v3 + v4) . 8 8

vv = (1 − nβn )v + βn

n−1 i=0

vi

(n ≥ 3) .

(1)

(2)

Meshes Simplification Based on Reverse Subdivision

851

Fig. 1. Loop subdivision

where v1 and v2 are the positions of the vertices incident to the split edge, v3 and v4 are other two vertices of the triangle sharing this edge, βn = n1 ( 58 − ( 38 + 1 2π 2 4 cos n ) ), where n is valence of the vertex. The vertex position of Loop subdivision can be expressed in the form of matrix equation: j+1 V = M(n+m)×n V j n×1 . (3) E j+1 (n+m)×1 where V is the set of vertex points, E represents the set of edge points, j denotes the subdivision level, and M stands for the subdivision matrix. The new set of edge points is obtained from one lower level vertices. The set of edge points E j+1 does not contribute any new information in solving the Eq.(3), so it can be ignored. Thereby Eq.(3) is rewritten as: j+1 V = Mn×n V j n×1 . (4) n×1 Subdivision surface converts the coarse mesh into the dense one, whereas a coarse mesh can be obtained by reversing this process. That is, in the matrix Eq.(4), the V j+1 is given, thereby V j can be determined. To solve the Eq.(4) is complex and difficult, so alternate algorithm is opted in this paper. By carefully analyzing the generation of edge and vertex points in Loop subdivision scheme gives rise to the following ideas which are used in the proposed mesh simplification. • Subdivision scheme can generate dense meshes from coarse ones. Certainly, the coarse meshes can be obtained by reversing this process. We denote this process as reverse subdivision. • The edge points generated by subdivision are only affine combination from the vertices of fore-level, and these vertices are vertex points in this level. That is, in the reverse Loop subdivision, the edge points do not contribute any new information of next-level vertices. They are redundant and can be deleted.

852

J.-p. Ma et al.

• The new vertex points obtained by Loop subdivision are derived from the vertices of previous level with adjacent vertices affine combination. The aim of adjusting vertex points is to generate a smooth surface. Whereas in the reverse process, the surface is supposed to be smooth, so there is no need to adjust the vertex points.

3 3.1

Methodology of Mesh Simplification The Principle of Mesh Simplification

We assume that input triangle mesh has 2-manifold topology and no boundaries. We further assume that the input mesh is a Remesh. Any arbitrary mesh should be preprocessed to a Remesh. A mesh that every vertex has valance six is a regular mesh and most of vertices have valance six is a semi-regular mesh. For a regular case, see Fig.2, according to the Loop subdivision, circle vertices are the edge points and the others are vertex points. By reversing this subdivision process, the mesh can be simplified. During each level of simplification, the edge points are decimated and the vertex points are preserved for the next level.

Fig. 2. Regular mesh simplification,the dark triangles are decimated. (a)Simplification time one, (b)Simplification times two.

From the close examination of Fig. 2(a), the edge points and vertex points are dotted regularly in the triangle mesh and each vertex point is encircled by six edge points. For a semi-regular case, see Fig.3, since the extraordinary vertices generally express the sharp features such as creases and corners of the models, they should be preserved for the next level. So the extraordinary vertex is assigned as the vertex point. The vertices around this vertex point are edge points. Fig.3 shows the irregular mesh simplification. The analysis results from regular and irregular meshes are that each vertex point is encircled by the edge points. The number of edge points around a vertex point is the valence of that vertex point. 3.2

The Algorithm for Mesh Simplification

Firstly, we introduce some notations which are adapted in [19][20]. We denote a triangle mesh as a pair (P, K),where P is a set of n point positions pi = (xi , yj , zi )


853

Fig. 3. Irregular mesh simplification, the dark triangles are decimated. (a) Simplification time one, (b) Simplification times two.

with 1 ≤ i ≤ n, and K is an abstract simplicial complex which contains all the topologies. The significant part of our algorithm is to construct a mesh hierarchy. The original mesh M n =(P n , k n ) is simplified into a series of homeomorphic meshes M j =(P j , k j ) with 0 ≤ j ≤ n, where M 0 =(P 0 , k 0 ) is a base mesh. Consider M n is the initial mesh. M n−1 is obtained after first simplification level. For a level M j simplifying process, it contains the following three steps. (1) Splitting: Split all the vertices P j of the mesh M j into two sets: EV EN j and ODDj , which contain vertex points and edge points, respectively. The set of ODDj is the candidates to decimate. For a regular mesh, firstly any vertex

Fig. 4. Vertices relationship. v is an extraordinary vertex, va is adjacent to v, vs is in the symmetrical position of v and va.

is assigned as vertex point. Then the six adjacent vertices are assigned as edge points. The vertices which are in the symmetrical position of these six edge points are assigned as vertex points. By repeating this process, the mesh can be split. For a semi-regular case(see Fig.4), firstly, an extraordinary vertex v should be determined and assigned as vertex point. Then the adjacent vertices va are

854

J.-p. Ma et al.

assigned as edge points. The vertices vs which are in the symmetrical position of these edge points are assigned as vertex points. Generally, all surface models have at least one extraordinary vertex, so the algorithm of assigning even points could be described as following. Procedure Set Even V ertex(v) begin if v EV EN , return true; if v ODD , return fault; Set Even Vertex(v); For each adjacent va of v begin if va is an extraordinary vertex or va EV EN , return fault; Set Odd Vertex(va); Find vs; Set Even Vertex(vs); end return true; end The Set Even V ertex(v) is recursively called until all the vertices are split. The split mesh must be a Remesh which is of subdivision connectivity. That is the Eq.(5) must be satisfied. Distance(Pi , Pj ) = 2k ,

k = 1, 2, 3... .

(5)

where Pi and Pj are any extraordinary points in the dense mesh, Distance(Pi ,Pj ) denotes the number of edges between Pi and Pj . (2)Predicting: In order to reconstruct the mesh, for a level j, displacement value dji of each edge point oddji must be calculated before they are decimated. The vertex points are utilized to predict the position of edge points oddi by using Eq.(1), thereby the displacement value dji can be obtained by Eq.(6). dji = oddji − oddi

(i = 1, 2, . . . .n) .

(6)

Fig.5(b) illustrates the predicting process, O is an edge point and O’ is the predicted edge point, displacement values are the distances from the edge points to the corresponding predicted edge points. Now, the displacement values dji are obtained and persevered for reconstruction. (3)Updating: After decimating the redundant edge points, the remaining vertex points form a new point set P n−1 . They are re-triangled and a new topological structure K n−1 is updated. Thereby the new level mesh M j−1 =(P n−1 ,K n−1 ) is obtained. By repeating the above three steps, the dense mesh M n can be simplified into a base mesh M 0 with a series of displacement values (M 0 d0 d1 ...dn−1 ). During each level of simplification, there is no need to adjust the coordinates of vertex points and each of the edge points has its displacement value. These groups of data could be packed by the several packets and form a set of progressive meshes. M 0 → (M 0 d0 ) → (M 0 d0 d1 ) → ... → (M 0 d0 d1 ...dn−1 )


855

Fig. 5. Predict and update process. (a) Before simplification. (b) Predicted edge points. (c) After simplification. Table 1. Progressive Meshes with a base mesh and a series of displacement values Progressive Mesh Displacement Value

Mn

M n−1 ... M j

...

M1

M0

dn−1

...

d1

d0

...

dj

That is, M 1 is determined by (M 0 d0 ), M 2 is determined by (M 0 d0 d1 ) , ... , and M n , (M 0 d0 d1 ...dn−1 ). It is shown in Table 1 .

4

Progressive Meshes Reconstruction

Progressive meshes which is a base mesh and a series of displacement values have been obtained. The progressive meshes are easy to manipulate and store. It can be packed into several small packets and transmit through wire/wireless network. The key aim of constructed progressive mesh is to supply a good mesh structure to represent the 3D graphics and the mesh could be reconstructed in a short time. A progressive mesh reconstruction method is given as follows. The progressive mesh reconstruction is a reverse process of the mesh simplification. To reconstruction the mesh from M j =(P j ,K j ) to M j+1 =(P j+1 ,K j+1 ), the following step should be done. (1) set P j as the vertex points EV EN j+1 of level j+1 (2) use P j to predict the edge points ODD at level j+1; (3) use dj and ODD to calculate the predicted position of edge points by use Eq. (6), and the set of ODDj+1 is obtained. oddj+1 = oddi + dji i

(i = 1, 2, . . . .n) .

(7)

(4) EV EN j+1 , ODDj+1 constitute the vertices P j+1 . Re-triangling the meshe form a new topology K j+1 , thereby the mesh M j+1 is achieved. That is, when the base mesh M 0 is achieved, by adding predicted positions of edge points oddi to d0i , the real position of edge points odd1i are achieved.

856

J.-p. Ma et al.

By updating the topology, M 1 can be obtained. By repeating this process, the mesh M n could be reconstructed. At this situation, the mesh is the same as the original one. So the mesh reconstruction from the progressive meshes is lossless. In application, a mesh is reconstructed on-demand according to resolution of device. If the resolution of the user device is low, it is not necessary to reconstruct the whole data, just a base mesh and few levels of displacement values are sufficient to represent, that is M i (i=0,1,2,...,n-1) will satisfy the resolution request of user. Because M i is the middle level comparing with the original mesh M n . It is noted that the new mesh is very close to M n but not the same. At this situation, progressive meshes reconstruction is not seamless. From above analysis, using any reverse subdivision either the Interpolatory or approximating schemes such as butterfly subdivision or Loop subdivision, the meshes can be simplified efficiently. According to their subdivision mask, the vertex affine combination of Loop is four[15] and Butterfly is eight[16]. Obviously, the algorithm utilized Loop subdivision could accelerate the simplification and reconstruction speed.

5

Results and Discussion

This algorithm is implemented by using VC++. The models such as head, esphere and venus are tested. The progressive meshes can be seen in Fig.6-8. Fig.7-8. show the rendering effect of the e-Sphere and venus model.

Fig. 6. Four levels of head model (Vertices/faces). (a) Original Mesh(45442/90880) (b)After simplification times one(11362/22720). (c) After simplification times two(2824/5680) (d) Base mesh(712vertices/1420faces).

Both reverse Loop approximating and reverse Butterfly interpolatory subdivision schemes are utilized in the new algorithm. The base meshes obtained from two schemes are the same, but their displacement values are different. Table 2. shows the runtime statistics of both schemes. Times are in seconds, measured on a PC(AMD 1800Mhz,256MB RAM). The experiments have indicted the speed


857

Fig. 7. e-Sphere simplification preserving vertices/faces from left to right (a) 2562/5120 (b) 642/1280 (c) 162/320 (d) 42/80 (e) 12/20 respectively

Fig. 8. Venus simplification(Vertices/faces). (a) Original(9603/19200). (b) Base mesh(153/300).

of simplification and reconstruction by the new algorithm are more faster than by butterfly scheme. The meshes can be progressively represented. It is lossless if the entire meshes are reconstructed. Otherwise, the reconstruction is not seamless. To measure the distortion between the reconstructed mesh and the original mesh, we use the Metro error measurement software [21](Metro v4.06) to compute the Maximal error and Mean error.

858

J.-p. Ma et al. Table 2. Runtime Statistics for simplification and reconstruction Data Vertices no./ Vertices no./ Simplified Faces no.

Faces no.

(In)

(Out)

times

712/1420

3

101.1 152.232 17.744 23.314

12/20

4

3.164

4.005

0.512

0.646

153/300

3

14.617 21.124

2.406

3.152

Head 45442/90880 e-Sphere 2562/5120 Venus

Runtimes

9603/19200

simplification

reconstruction

Loop Butterfly Loop Butterfly

Table 3. Error Statistics for e-Sphere Model Progressive Vertex Face M eshes no. no.

Box Diagonal

Maximal Error

Mean Error

RMS

Hausdorff Distance

M4

2562

5120

20.963922

0

0

0

0

M3

642

1280

20.987868

0.197355

0.020310

0.032450

0.225822

M2

162

320

21.227575

0.949826

0.078561

0.111246

0.949826

42

80

19.838592

0.932492

0.152490

0.195484

1.626944

12

20

15.714804

1.506523

0.374888

0.449767

3.836133

M M

1 0

See table 3, where M 4 is the original mesh, M 0 are base mesh, others are middle level of the progressive meshes. All the errors and hausdorff distance are compared with the original mesh M 4 . Actually, natural and efficient 3D interactions play one of key role in the development of Virtual Reality applications. It is always a tradeoff between the runtime and details of graphics. In some cases, such as on mobile device, there is no need to reconstruct the entire mesh. Only a base mesh M 0 and parts of displacement values(d0 , d1 , ..., di ) are sufficient to represent the 3D graphics for visibility. In other cases, M i ( for all i) can be allocated as the base mesh according to screen resolution, processor speed and memory capacity of the user device. Adopting fewer levels of the progressive meshes can increase the speed of the mesh reconstruction, and choosing denser mesh as a base mesh can achieved the good detail of graphics in shorter time but need good performance of computing devices.

6


Our goal is successfully fulfilled by simplifying the dense mesh into progressive meshes which is a base mesh with a set of displacement values. Three key steps of our simplification process are illustrated based on the reverse subdivision. The original mesh can be reconstructed lossless. It can be reconstructed with loss by using the middle level of progressive meshes when it is used in the low resolution


859

systems. Errors analysis is given using Metro software. The main contributions of this work are: • Loop subdivision scheme, which is a approximation scheme, is adopted as interpolatory one in simplification process, without adjusting the coordinate of vertex points. It greatly improves both the simplification and reconstruction speed. • A novel algorithm to simplify the dense mesh by splitting, predicting and updating steps is presented, which is easily represented and reconstructed in application, without solving the Eq.(4) or calculating the complex vertex affine combination of butterfly scheme as in literature[14]. The experiments have shown that the new algorithm is efficient and more faster than previous related techniques. The progressive meshes obtained by our method can represent 3D graphics efficiently and the original can be reconstructed quickly. It certainly can represent object movement through a virtual 3D workspace rapidly. On the other hand, our work still has limitation that the input mesh must be a Remesh. So more work need be done on surface fitting. In addition, improvement on the data structure could greatly accelerate the computational speed of our algorithm. Acknowledgements. The work described in this article is supported by the National Science Fund for Distinguished Young Scholars(N0.60525213) and the Key project(No.60533030) of the National Natural Science Foundation of China, and 973 Program of China (No.206CB303106).

References 1. Sequeira V., K.N., Wolfart E.,Goncalves J.G.M.,Hogg D., Automated reconstruction of 3D models from real environments. Journal of Photogrammetry and Remote Sensing, 54(1999)1-22. 2. Ridha Hambli, A.C., Hedi BEl Hadj Salah, Real-time deformation of structure using finite element and neural networks in virtual reality applications. Finite Elements in Analysis and Design,42(2006)985-991. 3. Eck M, DeRose T, Duchamp T, Hoppe H, Lounsbery M, Stuetzle W. Multiresolution Analysis of Arbitrary Meshes. ACM SIGGRAPH 1995 Conference Proceedings (1995)173-182. 4. Weiyin Ma, Xiaohu Ma, Shiu-kit Tso and Zhigeng Pan, A direct approach for subdivision surface fitting from a dense triangle mesh,Computer-Aided Design, 36(2004)525-536. 5. Mongkolnam P.,Raidan A. and Farin G., Loop Progressive Geometry Compression for Triangle Meshes, Computer Aided Geometric Design, 16(1999)837-840. 6. William J. Schroeder, Jonathan A. Zarge and William E. Lorensen, Decimation of triangle meshes, In:Computer Graphics(SIGGRAPH’92 Proc.),26(1992)65-70. 7. Gueziec A. Surface simplification inside a tolerance volume. Technical Report RC 20440,IBM T.J. Watson Research Center,1996. 8. Hamann B., A data reduction scheme for triangulated surfaces, Computer Aided Geometric Design,11(1994)197-214.

860

J.-p. Ma et al.

9. Lee.A, Moreton.H and Hoppe.H, Displaced Subdivision Surfaces, in:Proceedings of the 27th annual conference on computer graphics(SIGGRAPH 2000),(2000)85-94. 10. Garland M. and Heckbert P.S., Surface Simplification Using Quadric Error Metrics, In:Proceedings of the 24th annual conference on Computer graphics and interactive techniques(SIGGRAPH 1997),(1997)209-216. 11. Suzuki H.,Takeuchi S. and Kanai T., Subdivision Surface Fitting to a Range of Points, in:The 7th Pacific Conference on Computer Graphics and Application,IEEE Proceedings,(1999)158-167. 12. Hoppe H., Progressive meshes. In:Proceedings of the 23rd annual conference on Computer graphics and interactive techniques(SIGGRAPH 1997),(1996)99-108. 13. Khodakovsky A.,Schroder P. and Sweldens W., Progressive Geometry Compression, in:Proceedings of the 27th Annual Conference on Computer Graphics and interactive Techniques(SIGGRAPH 2000),(2000)271-278. 14. Xiaonan Luo and Guifeng Zheng, Progressive Meshes Transmission over a Wiredto-wireless Network, ACM Journal of Wireless Networks(WINET),2005, (In press). 15. Loop C., Smooth subdivision surface based on triangles,[Master’s thesis], University of Utah, Department of Mathematics,1987. 16. Dyn N.,Levin D. and Gregory J.A., A butterfly subdivision scheme for surface interpolatory with tension control, ACM transactions on Graphics(TOG), 9(1990)160169. √ 17. Labsik U. and Greiner G., Interpolatory 3 subdivision, in:Computer Graphics Forum (EUROGRAPHICS 2000), (2000)131-138. 18. Farin G.E. Curves and Surfaces for Computer Aided Geometric Design: A Practical Guide. 5th Edition. Morgan Kaufmann,2001. 19. Aaron WF Lee, Wim Sweldens, Peter Schroder, Lawrence Cowsar, and David Dobkin. MAPS: Multiresolution Adaptive Parameterization of Surfaces. In:Proceedings of the 25th annual conference on Computer graphics and interactive techniques(SIGGRAPH 98),(1998)95-104. 20. Hoppe H., Efficient Implementation of Progressive Meshes. Technical Report MSRTR-98-02, Microsoft Research, 1998. 21. Cignoni P., Rocchini C. and Scopigno R., Metro:Measuring Error on Simplified Surfaces. Computer Graphics Form,17(1998)167-174.

Kd-Tree Based OLS in Implicit Surface Reconstruction with Radial Basis Function Peizhi Wen1,4 , Xiaojun Wu2 , Tao Gao3 , and Chengke Wu4 1

Guilin University of Electronic Technology, Guilin, 541004, China 2 HIT Shenzhen Graduate School, Shenzhen, 518055, China 3 Xi’an Jiaotong University, Xi’an, 710071, China 4 Xidian University, Xi’an, 710071, China

Abstract. In this paper, we propose a new method for surface reconstruction from scattered point set based on least square radial basis function (LSRBF) and orthogonal least square forward selection procedure. Firstly, the traditional RBF formulation is rewritten into least square formula. A implicit surface can be represented with fewer centers. Then, the orthogonal least square procedure is utilized to select significant centers from original point data set. The RBF coefficients can be solved from the triangular matrix from OLS selection through backward substitution method. So, this scheme can offer a fast surface reconstruction tool and can overcome the numerical ill-conditioning of coefficient matrix and over-fitting problem. Some examples are presented to show the effectiveness of our algorithm in 2D and 3D cases.

1

Introduction

The problem of surface reconstruction from scattered cloud points has been studied extensively in computer graphics and engineering, in particular, the use of a range scanner or laser scanners produces large amount of unorganized point sets in industry, entertainment and archeology, etc. It is desirable to quickly and robustly reconstruct a continuous surface with attributes from the unorganized points. Researchers in computer graphics and computational geometry offer some useful tools and method to tackle this problem, for example, Hoppe’s [7]signed distance function based method, Amenta’s [1,2] Voronoi and crust based method, Bernardini’s [3] α shape based algorithm, Dey’s [4] concone based approach, Level-set based method of Zhao [5], Moving least square(MLS) of Shen [10] and MPU of Ohtake [9], etc. Radial Basis Function (RBF) has been used in several fields because of its accurate and stable interpolation properties. It is proved that any continuous function can be modeled up to a certain precision by RBF [6]. In this paper, we use the least square radial basis functions to solve the problem of surface reconstruction with fewer interpolation centers. Implicit surface models are popular since they can describe complex shapes with capabilities for surface and volume modeling and complex editing operations are easy to perform on such Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 861–870, 2006. c Springer-Verlag Berlin Heidelberg 2006

862

P. Wen et al.

models. Radial basis functions attract more attentions recently in data interpolation in multi-dimensions [8,11,12,13,14]. Though traditional RBF can offer efficient reconstruction algorithms, some significant problems still remains unsolved. Since the collocation method uses the scattered points as both data and centers, numerical ill-conditioning often occurs due to too small distance between some centers, which will cause linear dependency of coefficients matrix. Moreover, when the number of centers used in interpolation is too large, it will cause the so-called over-fitting problem, which means that the interpolant is too flexible so that it not only fits the surface but also fits the noise that was introduced into the data. Lastly, in order to further reduce the computational cost, greedy algorithm or so-called ”thinning” algorithm was introduced into RBF reconstruction methods by using only a small portion of given points. Least square RBF, fortunately, lends us powerful tools to solve all the problems mentioned above. In this paper we describe an algorithm to the problem of surface reconstruction from large unorganized point sets. We adapt the methodology of least square RBF to surface reconstruction and deduce a set of corresponding formulations. Then the orthogonal least square method is used to select significant centers from the large data set, from which a upper triangular matrix is produced and the RBF coefficients can be solved via backward substitution directly rather than using LU, SVD or preconditioned conjugate gradient method (PCG) [18]. Paper is organized as follows. The radial basis functions networks are introduced in section 2, and in section 3 the LSRBF scheme is described. The OLS selection procedure is presented in section 4. Some reconstructed examples are presented in section 5. Part 6 is the conclusion section.

2

Radial Basis Function

RBF has traditionally been considered as single-layer networks with symmetrical N radial functions. Samples {xi }N i=1 are inputs of this network and {αi }i=1 are the coefficients. The problem of scattered data interpolation can be stated as given a set of fixed points x1 , x2 , ..., xN ∈ Rd on a surfaces S in Rd (in three dimensions d = 3) and a set of function values f1 , f2 , ..., fN ∈ R, find an interpolant φ : R3 → R such that φ(xi ) = fi , i = 1, 2, ..., N (1) The traditional RBF has the following form φ(x) =

N

αj gj (x − xj ) + p(x)

(2)

j=1

where p(x) is a polynomial, αj are coefficients corresponding to each basis and · is the Euclidean norm on R3 . The basis function g is a real valued function on [0, ∞),usually unbounded and of a global support. There are several popular basis functions used in literature [8], therein. The polynomial p(x) is appended

Kd-Tree Based OLS in Implicit Surface Reconstruction with RBF

863

for achieving polynomial precision according to the basis functions used. Additional so-called natural constraints are needed. For example, if p(x) is a linear polynomial, the coefficients α must satisfy the following side conditions. N j=1

αj =

N

αj xj =

j=1

N

αj y j =

j=1

N

αj z j = 0

(3)

j=1

Combining equation (1) and (2) and using collocation technique, we can give rise to compute the RBF coefficients. φ(xi ) =

N

αj gj (xi − xj ) + p(xi )

(4)

j=1

The coefficients can be found by solving the linear system 4. The solutions of the system compose of the weighting coefficients and the polynomial coefficients for the interpolation function φ(x). However, it is unpractical to solve the linear system (4) directly for large scale point dataset and with complexity of O(N 3 ) interpolant computing and O(N ) evaluation. The concept of partition of unity (POU) is rooted in applied mathematics [15,16]. The main idea of the partition of unity method is to divide the global domain of interest into smaller overlapping subdomains where the problem can be solved locally on a small scale. The local solutions are combined together by using blending functions to obtain the global solution. The smoothness of the global solution in the overlap regions of two subdomians can be guaranteed by a polynomial blending function. The POU method for RBF based surface reconstruction has been applied by Tobor et al. [17] where all the points in each subdomain are offset along the normal to avoid the RBF trivial solutions.

3

Least Square Surfaces

Only a portion of points are used to interpolate the implicit surface. We rewrite the RBF formulation (2) into φ(x) =

M

αj g(x − xj ) = g T α

(5)

j=1

where g = [g 1 , g 2 , · · · , g M ]T , α = [α1 , α2 , · · · , αM ]T , M is the number of points used to estimate the coefficients (M N ), and T is the transpose operator. ˆ indicating the funciton value with noisy Function values on the data h and h T T T T data. Let G = [g 1 , g 2 , · · · , g N ]N ×M , we have φ(x) = Gα

(6)

We write equation 6 into matrix form [Gx,¯ x ][αx ¯ ] = [f x ]

(7)

864

P. Wen et al.

where x = x1 , x2 , · · · , xN stands for the the input points in a processing unit, ¯ j ), i = and x ¯ = x ¯1 , x ¯2 , · · · , x ¯n is the selected centers. (G)i,j = φ(xi − x 1, 2, · · · , N, j = 1, 2, · · · , M . α = [α1 , α2 , · · · , αM ]T . f = [f1 , f2 , · · · , fN ]. In these equations the low degree of polynomial is not considered. It can be appended in accordance with specific basis functions. When coefficients α are solved, the implicit surface can be reconstructed with fewer centers, M , much less than the total of whole set of samples N . Some significant centers can be used to compute the RBF coefficients from which the implicit surfaces can be achieved. As a illustration, a 2D example is displayed in Fig.1. In this example, the interpolation centers are selected uniformly instead of using significant centers. From Fig.1, we can find when the interpolation centers are not fewer enough, the reconstructed curve has almost no visual distinctions.

#192

#192

#96

#64

#48

#24

Fig. 1. Curve interpolation using fewer RBF centers

4

Orthogonal Least Square Procedure

The orthogonal least square method often used to select a suitable set of centers from large scale point sets and a fixed RBF center corresponds to a given regressor in a linear regression model [19]. We utilize the Gram-Schmidt orthonolization process [18] to ensure the new added column of RBF matrix is orthogonal to all previous columns, and then the corresponding approximation error is reduced. Other non-orthogonal column vectors can be regarded as the linear combination of the orthogonal basis, which can eliminate the problem of ill-conditioning. In our another method, we use Chen’s OLS method as a forward regression procedure [20]. But in that scheme, we just used OLS method to select significant centers and reconstruct implicit surfaces from these selected set using conventional RBF interpolation technique. We also employed a octree structure to organize the large point set, and we took each octree cell as a unit to carry out the selection procedure individually. Though the point in each octree cell can be resampled according to OLS selection criterion, the points in the area of

Fig. 2. The drawback of octree based OLS method


865

the two neighbor cells can not be handled correctly, as illustrated in Fig. 2. We call this a neighboring cell problem. The left figure in Fig.2 is octree cells and the corresponding point data. The afterward result is shown in the right figure from which we can find that the points near the wall of two neighbor octree cells remain the same. 4.1

OLS Selection Criterion

For completeness, we write out the pseudo code for OLS selection in a subdomain with the input point set C with the amount of points N and approximation tolerance ρ, 0 < ρ < 1, which is a user defined tolerence controlling the amount of crucial centers. – The first selected center for 1 i N (i)

w1 = pi ; (i)

(i)

(i)

(i)

q1 = (w 1 )T h/((w1 )T w1 ) (i)

(i)

(i)

(i)

[err]1 = (q1 )2 (w 1 )T w1 /(hT h) end for (i ) (i) Find [err]1 1 = max [err]1 , 1 i N (i )

And select w1 = w1 1 = g i1 – At the kth step, where k 2, for 1 i N, (i = i1 , · · · , i = ik−1 ) for 1 j < k (i)

αjk = wTj pi /(wTj w j ) k−1 (i) (i) w k = pi − j=1 αjk wj (i)

(i)

(i)

(i)

qk = (wk )T h/((wk )T w k ) end for (i) (i) (i) (i) [err]k = ((w k )T h)2 /(((wk )T w k )(hT h)) (i)

Find [err]ikk = max [err]k , 1 i N, i = i1 , · · · , i = ik−1 k−1 ik ik Select wk = wk = pik − j−1 αjk wj – The procedure is stopped at the M th step when the error ration satisfies 1− M j=1 [err]j < ρ In above procedures, pi , i = 1, 2, · · · , N is the column vector of Matrix G. In the OLS selection, we can get a upper triangular matrix Q and a vector q from which the coefficients can be solved by backward substitution unlike the traditional RBF reconstruction method by using LU decomposition or SVD. So this method can avoid the ill-conditioning problem. [Q][α] = [q]

(8)

866

4.2

P. Wen et al.

Kd-Tree Based OLS

In the above subsection, we have obtained a criterion for significant center selection. We will describe the new scheme specifically. Firstly, the point data is divided via a octree into some local subdomains. Here, and we use the idea of partition of unity [15,9] to blend the local surface into global solution. Readers can refer the technique of partition of unity in [15,9,11]. Then, we substitute the processing unit from octree cell to a ball centered at the the octree cell and using a Kd-tree to collect points in this spherical subdomian that can avoid the neighboring cell problem, shown in Fig. 3.

Fig. 3. New processing units are used for OLS selection

By comparing the Fig. 2 and Fig. 3, we can find that our new scheme can get more optimal data set than the previous method. Thus, the final reconstructed surface will achieve good quality.

5

Experimental Results

All results revealed in this section were performed on the hardware configurations of Intel PIV 3GHz with 513M RAM running Windows XP. Table 1 shows the results of three examples reconstructed with conventional RBF method, where LU decomposition is employed to solve the linear system and the triharmonic basis function is used. And Table 2 gives the reconstruction time and OLS parameters. In table 1, N points stands for the the number of points, and T rec is the time of reconstruction including octree setup and linear system solving. T isosurf means the iso-surface extraction cost, and T total is the total time consumption of surface reconstruction. In table 2, P ration refers to the ration of selected point set and the whole point set. T rec includes the time of Table 1. Traditional RBF surface reconstruction Model Sculpture Bunny Igea Sk hand

N points 25,382 34,834 72,545 327,323

T rec 12.41 18.05 51.77 365.64

T isosurf 197.65 218.25 291.72 1261.04

T total 210.06 236.3 343.43 1,626.68


867

Table 2. Surface reconstruction with Orthogonal least square Model ρ Sculpture 0.01 0.05 0.1 Bunny 0.01 0.05 0.1 Igea 0.01 0.05 0.1 Sk hand 0.1

(a)

P ration 63% 21% 20% 59% 20% 19% 67% 21% 16% 15%

T rec 20.81 4.08 3.56 32.92 5.64 5.07 53.34 9.65 8.76 156.23

(b)

T isosurf 123.86 118.96 115.83 168.63 149.48 145.95 243.86 167.31 112.47 827.66

T totl 144.67 123.04 119.39 201.55 155.12 151.02 297.2 176.96 121.23 983.89

Max error 0.0942 0.1029 0.1731 0.1079 0.0848 0.1335 0.1614 0.1674 0.2821 0.0699

(c)

Mean error RMS 0.0084 0.0127 0.0098 0.0133 0.0096 0.0129 0.0060 0.01 0.0079 0.0108 0.008 0.0116 0.0078 0.0117 0.0112 0.0161 0.0174 0.0287 0.0049 0.0061

(d)

Fig. 4. Meshes from different point sets, (a) is resulted from the original point set, (b) is reconstructed from 64% point set, (c) and (d) are from 21% and 20% point data sets

(a)

(b)

(c)

(d)

Fig. 5. Meshes from different point sets, (a) is reconstructed from the whole data set, (b) is from 59%, (c) is got from 20%, and (d) is from 19%

octree setup, OLS selection and backward substitution. Max error, Mean error and RMS represent maximal error and mean error between the results from OLS and the the surface reconstructed from original points, and root-mean-square,

868

P. Wen et al.

(a)

(b)

(c)

(d)

Fig. 6. Igea model reconstructed from different point sets, (a) is from 100%, (b) is reconstructed with 67%, (c) is from point set of 21%, (d) is obtained from 16%

(a)

(b)

(c)

Fig. 7. Skeleton hand reconstructed from original point set(a) and 15% point set (b), (c) is error map of mesh (a) and (b)

respectively. Fig.4, Fig.5, Fig.6 and Fig.7 are reconstructed surface models. There are almost no visual artifacts on these series of experimental results. By comparing the two tables, we can find that the computation time is reduced greatly but the quality of the surface does not affected severely. We use the a free available tool–Metro, to measure the error differences between two meshes[21]. As indicated by the last three columns of Table2, the mean-error and RMS are relatively small. Taking Stanford bunny as an instance, the mean errors and the RMS are 0.006, 0.0079, 0.008 and 0.01, 0.0108, 0.0129 in case of ρ=0.01, 0.05 and 0.1, respectively. However, when ρ=0.1, the Max error is obviously standing out, for example, maximal error equals to 0.1731 of sculpture model and igea is 0.2821. Fig.7 (c) also shows a error color map of surface (a) and (b). As a matter of fact, most of the computational time is consumed by the iso-surface extraction procedure, which also can be found from Table 1 and Table 2. Actually, the time cost of solving RBF linear system in the example of skeleton hand is only 13.3 second. Most of the time spends on the preprocessing and iso-surface extraction. So a fast implicit surface polygonization algorithm is necessary in our approach.

6

Conclusions

In this paper, we presented a RBF implicit surface reconstruction method on the basis of modified orthogonal least square subset selection through which


869

the interpolation centers can be reduced and the reconstruction speed can be improved significantly but not severely affect the quality of the surface. In this scheme, unlike the traditional RBF method, the LU, SVD or PCG method will needed to solve the linear system, instead we use the triangular matrix obtained in the OLS selection and backward substitution to resolve the coefficient. The experiments show that our method are very efficient to deal with the large point set and can also be utilized in many fields, such as virtual reality, archeology, video game development, etc.

References 1. Amenta, N., Bern, M. and Kamvysselis, M.: A new Voronoi-based surface reconstruction algorithm. In Proceedings of SIGGRAPH’98, (1998), 415-421. 2. Amenta, N., Choi, S. and Kolluri, R.: The power crust. In Proceedings of 6th ACM Symposium on Solid Modeling, (2001), 249-260. 3. Bernardini, F. and Bajaj, C.L.: Sampling and reconstructing manifolds using alphashapes. In proceedings of 9th Canada Conference on Computational Geometry. (1997), 193-198. 4. Dey, T. K. and Goswami, S.: Tight cocone: A watertight surface reconstructor. In Proc. 8th ACM Sympos. Solid Modeling Applications, (2003), 127-134. 5. Zhao, H., Osher, S. and Fedkiw, R.: Fast surface reconstruction using the Level Set method, 1st IEEE Workshop on Variational and Level Set Methods, in conjunction with the 8th International Conference on Computer Vision (ICCV), Vancouver, Canada, (2001), 194-202. 6. Park, J. and Sandberg, J.W. Universal approximation using radial basis functions network. Neural Computation, (1991), Vol.3, 246-257. 7. Hoppe, H, DeRose, T., Duchamp, T., McDonald, J. and Stuetzle, W. Surface ronstruction from unorganized points. In Proceedings of ACM SIGGRAPH 1992, (1992), 71-78. 8. Carr, J. C., Beatson, R. K., Cherrie, J. B., Mitchell, T. J., Fright, W. R., McCallum, B. C. and Evans, T. R. Reconstruction and representation of 3D objects with radial basis functions. In Proceedings of ACM SIGGRAPH 2001, (2001), 67-76. 9. Ohtake, Y., Belyaev, A., Alexa, M., Turk, G. and Seidel, H. P. Multi-level partition of unity implicits. ACM Transactions on Graphics, Proceedings of SIGGRAPH 2003. (2003), Vol. 22, No. 3, 463-470. 10. Shen, C., O’Brien, J.F., and Shewchuk, J.R.: Interpolating and approximating implicit surfaces from polygon soup. In Proceedings of ACM SIGGRAPH (2004), 896-904. 11. Tobor, P., Reuter, and Schilck, C. Efficient reconstruction of large scattered geometric datasets using the partition of unity and radial basis functions. Journal of WSCG 2004, (2004), Vol. 12, 467-474. 12. Turk, G. and O’Brien, J.: Variational implicit surfaces. Technical Report GITGVU-99-15, Georgia Institute of Technology, (1998). 13. Turk, G. and O’brien, J., Modelling with implicit surfaces that interpolate. ACM Transactions on Graphics, (2002), 21(4), 855-873. 14. Morse, B., Yoo, T.S. Rheingans, P. et al. Interpolating implicit surfaces from scattered surfaces data using compactly supported radial basis functions. In Proceedings of Shape Modeling International (SMI2001), (2001), 89-98.

870

P. Wen et al.

15. Babuˇska, I. and Melenk, J.M.: The partition of unity method. International Journal of Numerical Methods in Engineering, (1997) 727-758. 16. Wu J. and Kobbelt, L. P., A stream algorithm for the decimation of massive meshes. In Graphics Interface 2003 Proceedings, Halifax, Canada, 2003, 185-192. 17. Tobor, I., Reuter, P. and Schlick, C. Multiresolution reconstruction of implicit surfaces with attributes from large unorganized point sets. In Proceedings of Shape Modeling International (SMI 2004), (2004), 19-30. 18. Horn, A.E., Johnson, C.R.: Matrix analysis. Cambridge University Press, Cambridge, UK (1985). 19. Chen, S., Billings, S.A., and Luo, W.: Orthogonal least square methods and their application to nonlinear system identification. International Journal of Control, (1989), 50(5), 1873-1896. 20. Chen, S., Cowan, C.F.N., and Grant, P.M.: Orthogonal least square learning algorithm for radial basis function networks. IEEE Transactions on Neural Network, (1991), 2(2), 302-309. 21. Cignoni, P., Rocchini, C. and Scopigno, R.: Metro: measuring error on simplified surfaces. Computer Graphics Forum, (1998), 17(2), 283-390.

Retrieving 3D CAD Model with Regional Entropy Distributions Liu Wei and He Yuanjun Department of Computer Science and Engineering, Shanghai Jiaotong University, Shanghai, 200240, P.R. China

Abstract. In this paper, we present a novel method to match and retrieve 3D CAD models. Firstly the CAD model is voxelized. Then the voxelized model is divided into several regions, and the entropy of each region is calculated. All the entropies constitute the feature vector representing the models when compared with other models using Euclidean distance. A system based on this idea is implemented and its feasibility is validated. Keywords: Voxelization, 3D model retrieval, entropy, feature extraction.

1 Introduction Now 3D models play an important role in many applications and become the forth multimedia type after sound, image and video. Recently, the development of 3D modeling and digitizing technologies has made the model generating process much easier. Also, through the Internet, users can download a large number of free 3D models from all over the world. In the field of mechanical engineering, various kinds of 3D CAD design tools such as Pro/Engineering, CATIA, SolidWorks, etc, have replaced the status of the 2D ones. As a result many manufacture enterprises commonly have a 3D parts library which contains tens of thousands of CAD models. These models are used to communicate the exact shape and dimensions of components to both customers and sub-contracting manufacturers. Consequently these models are of great value and importance to the companies. Since even the simplest components have significant production costs, designers should make use of existing parts whenever practicable, which gives rise to a following problem of rapidly finding the parts needed in large parts library. However, obtaining and surveying catalogues of existing parts, even within a single enterprise, is still a costly, time-consuming task. Traditionally CAD models are indexed by alpha-numeric part numbers with syntax specific to each company and in the area of group-technology, various “part coding” schemes have been proposed[1]. But in despite of their wide use, such manual shape classification schemes are subjective and limited to standard or general mechanical parts, so they can’t wok automatically only with computers. To enable the savings of time and money that are associated with design reuse, which will greatly reduce the cost of part design in manufacture enterprises, it is necessary to develop a 3D CAD automatic retrieval system based on content with its goal being to find the similar Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 871 – 879, 2006. © Springer-Verlag Berlin Heidelberg 2006

872

W. Liu and Y. He

parts for a given query mainly according to their shape. One of the main challenges in this context is the mapping of 3D model into compact canonical representations referred to as descriptor or feature vector, which serve as search keys during the retrieval process. The descriptor decisively influences the performance of the search engine in terms of computational efficiency and relevance of the results. In this paper, we put forward to a regional distributions based method with entropy which proves to be effective in CAD model retrieval. The motivation behind our work is to enhance the performance of 3D mechanical models retrieval and to develop a fast, simple, and robust system acted on CAD model library. The outline of the rest of this paper is as following: in the next section we review the previous work. In Section 3 we describe our method for descriptor extraction based on voxelization and entropy. In Section 4 experiments and results are presented in detail. Finally we give the conclusions and future work in Section 5.

2 Previous Work Last several years have seen increasing academic research into 3D shape retrieval methods for various kinds of applications including mechanical components. Paquet[2-3] made the initial research in 1990s and got remarkable achievements. From then on, a lot of researches have been carried on. Now many approaches have been proposed. In general, they can be divided into four types: shape-based retrieval, topology-based retrieval, image-based retrieval and surface-attributes-based retrieval. Anderst[4] directly syncopated 3D model with some mode, then calculated the proportions of points number of each unit in that of the whole model, thus a shape histograms was formed to compare different models. Suzuki[5] put forward to a different method to syncopate 3D models which was called point density. This method did not form feature vector simply by the syncopated units but classified the syncopated units, so the dimension of feature vector and computational quantity decreased greatly. Osada[6] investigated a 3D model retrieval system based on shape distributions. The main idea was to calculate and get large numbers of statistical dates which could be served as shape distributions to describe the features of models. The key step was to define the functions which could describe the models. He defined five simple and easy-to-compute shape functions: A3, D1, D2, D3 and D4. Because this method was based on large number of statistical dates, it was robust to noise, resampling and predigestion. Vranic’s[7] method firstly voxelized the 3D model, then applied 3D Fourier Transform on these voxelizations to decompose the model into different frequencies, finally chose certain amount of coefficients of frequency as this model’s feature vector. Another approach[8-9] investigated by Vranic and Saupe was spherical harmonics analysis, which was also called 2D Fourier Transform on unitary sphere. This method needed sampling and harmonics transform, so the process of feature extraction was slow. Chen[10] had developed a web-based 3D model retrieval system in which a topology method using Reeb graph[11] was introduced. Reeb graph could be constructed by different precisions, thus in this way multi-resolution retrieval was available. There are also many researches on image-based retrieval[12-14] of which Chen at Taiwan University made outstanding productions. Surfaceattributes-based retrieval is not mature as the above three types and only few researches were carried on[15].

Retrieving 3D CAD Model with Regional Entropy Distributions

873

Different from common 3D models, structures or semantic information is often used to differentiate mechanical ones. Elinson[16] described a domain-specific approach to assessing similarity of mechanical parts. He developed design signatures for mechanical parts which was a graph structure containing nodes that represented various attributes of the design, and edges that were labeled to represent relationships among those attributes. He used graph isomorphism theoryto to compare objects and subgraphs. Sun[17] reported a system for assessing the shape similarity of polyhedra by matching subgraphs extracted from their boundary models. However the system required input of two types of solid models. A prototype implementation successfully identified similarities between various mechanical parts with intersecting features. McWherter[18] described a system for partitioning a collection of models into clusters of similar shapes. Models were characterized by a MSG(Model Signature Graph) generated from the model’s faceedge graphs. Various semantic and structural properties of the MSG were used to place each in a multidimensional vector space. The approach had been used to cluster the contents of the Design Repository and the results appeared to group approximately similar parts into the same clusters.

3 Our Method In this paper, we don’t intent to use any structural or semantic information to extract the feature of 3D models as described in Section 2 to retrieve mechanical models. Our method also mainly depends on the shape character of 3D CAD models. Comparing with other medium such as image, sound and video, the structure of 3D model is correspondingly complicated. It is well known that 3D models are in an arbitrarily curving space which can’t be continuously parameterized, and generally they are composed of various spatial polygons and haven’t been sampled, so they can be viewed as irregularly signal. As a result, many classical analysis methods such as moment invariant, Fourier and wavelet transform can’t be applied on 3D models directly, which not only makes the process of feature extraction difficult but also depresses the distinguishing power of the features extracted. A common way to convert irregularly signal to regularly signal is sampling. But for common 3D model, sampling will lost many details unless using enormous sampling points because of the complexity of 3D model. Instead, we notice that the 3D CAD models have particular characteristic which are obviously different from other types of models. Generally speaking, CAD models are comparatively simpler than other ones, especially models of animals, plants, natural scenes, etc, and have stronger trend of symmetry. At the same time, they are usually composed of several finite planes or curves and the normal of their surfaces vary drastically at certain places thus often they have distinct contours. So we can suppose that CAD models can be sampled by a set of small cubes which are called voxels in the later depiction without biggish distortion. Then we can get an approximation of the original 3D part by means of voxelization which proves to approach the original model with considerable precision. The following entropybased arithmetic for feature extraction is entirely based on this supposition. In this

874

W. Liu and Y. He

paper, we assume that the CAD models are not solid ones and their surfaces are composed of polygons. 3.1 Voxelization of 3D CAD Models Let the continuous 3D space voxelized space

( R × R × R ) be marked as R 3 , while the discrete 3D

( Z × Z × Z ) , which is a 3D array of grid points, shall be referred to

Z 3 . We term a voxel or the region contained by a 3D discrete point ( x, y, z ) , as the continuous region (u , v, w) such that:

as

x − 05 p u ≤ x + 0.5 , y − 0 .5 p v ≤ y + 0 .5 , z − 0 .5 p w ≤ z + 0 . 5 .

(1)

This assumes that the voxel occupies a unit cube centered at the grid 3

point ( x, y , z ) , and the array of voxels tessellates Z . The aggregate of all voxels is a tessellation of 3D Euclidean space. A voxel’s value is mapped into the set {0, 1}: voxels assigned the value “1'' are called “black'' or “non-empty'' voxels, while those assigned the value “0'' are called “white'' or “empty'' voxels. A straightforward algorithm is to compute the intersection of the surface of 3D model and the voxel grids. But this method leads to contain more voxels than necessary. Various improved voxelization algorithms for spatial polygons have been proposed[19-20]. Kaufman proposed a 3D scan-conversion for polyhedra in Reference [19] which is a variation of 2D scan-line filling algorithm and uses only simple operations like additions and testy inside the inner loops. This method is a most effective one. Fig.1 shows the voxelization of a hammer using this algorithm and from this figure we can observe that the original model and the voxelized one are very similar, so we can use the feature of the latter to replace that of the former. Through this step, we convert an irregular polygonal model to a regular voxelized model, which bring great convenience for analysis.

Fig. 1. An example of voxelization

3.2 Feature Extraction Shannon’s Theory of Information defines information as a transcendentally bushtailed factor and information’s loader is called “message”[21]. Suppose that there is a limited message set A = {a1 , a2 , L , an } in which the emergence of each message is unattached to each other and the probability is respectively:


875

n

P = { p1 , p 2 , L, p n } , ( ∑ p i = 1 , pi ≥ 0 ).

(2)

i =1

Then the entropy of this message set can be defined as: n

H ( x) = −∑ p i log 2 p i .

(3)

i =1

The value of this entropy embodies the average bushtailed factor of emergences of the messages in the set A . As for the voxelized 3D models, since different voxels with different probability distribution in different spatial regions, they exhibit multifarious shape features. There are only two kinds of voxels: “non-empty'' voxels(Valued 1) and “empty” voxels(Valued 0), then its entropy is

H 2 = − p1 log 2 p1 − p 2 log 2 p 2 , p1 + p 2 = 1 .

(4)

p1 and p 2 are the proportions of the two kinds of voxels in the total L × L × L space( L is the resolution for voxelization). Since the entropy is only And

related to the numbers but not the locations of the two kinds of voxels, two completely different models may have the same value of entropy, as illustrated in Fig.2 using 2D case with its resolution being 2 × 2 . They have obvious difference in vision while having the same entropy.

Fig. 2. Different objects have the same entropy

So using global entropy as feature of the 3D model is inapposite. Since partial entropy also reflects the distributing of the region it covers, so if we divide the voxelized model into several regions and compute their entropies, the union of them will represent the feature of the model in a more elaborate manner. We divide the total L × L × L voxels containing the voxelized original model into K × K × K ( K ∈ N ) regions, and to ensure that the voxel be the minimal unit, we

d = L K ∈ N . Let V be the L × L × L voxels and Vi be the i-th region. If the entropy of Vi is f i , then the feature vector of the model can be

presume that

constructed as:

f = { f1 , f 2 ,L, f d 3 } .

(5)

876

W. Liu and Y. He

Through this method, we solve the problem that different models have the same value of entropy. The feature vector not only describes the global characteristic but also reflects the spatial distributing of the 3D model. Fig.3 shows a model and its feature vector in one situation.

Fig. 3. A model and its feature vector in one situation

3.3 Models Comparison

f A and f B be the feature vectors of two 3D CAD models, then the difference between them can be defined using a certain Distance Metric such as L1 , L2 and

Let

Mahalanobis distance, etc. Throughout our experiments the common Euclidean distance( L2 ) is used, that is to say:

D( A, B) = L2 ( f A , f B ) =

d3

∑( f

Ai

− f Bi ) 2 .

(6)

i =1

But direct computation doesn’t work since 3D CAD models have arbitrary placements when they are created. 3D CAD models have different position, orientation and scaling in 3D space and an entropy component in the feature vector represents the distribution of a certain region at the 3D model, so an important work we must do firstly is to find the correspondence between two models and make the comparison valid. Since a majority of CAD models are created in a comparatively standardized place as mechanical parts generally have obvious orientations and usually have at least a symmetry axis, we can easily find the correspondences. We needn’t search for all possible placements, and later experiments showed that finite postures will be enough to get a good correspondence. There are 6 orientations from the center of the cube to the center of its faces, called as main orientations superposed by the primary symmetry axis of the CAD model. For each main orientation, we rotate the model around it per 90 degrees and get 4 postures. And if we take reflection into account, we can get total 6 × 4 × 2 = 48 postures. As each orientation’s symmetrical situation is also within the 48 cases, when matching two 3D CAD models, we can assume that one remains fixed while the other adjusts its pose to the 48 cases, if D1 , D2 , …, D48 be the matching distances, then the actual difference between the two 3D models can thus be denoted as:

D = min( D1 , D2 , L, D48 ) .

(7)


877

4 Experiments Result According to the method described in Section 3, we have developed a 3D CAD models retrieval system using Visual C++ on a PC with Pentium IV 1.8G CPU and Windows 2000 Server Operation System. The system consists of two parts: offline process module and online retrieval process module, as showed in Fig.4. In the offline process, features of 3D models are extracted and stored in database. To efficiently search a large collection online, an indexing data structure and searching algorithm is available with B+ Tree. In the online process, users select a 3D model as the query mode, then the system engine compares the distances between the query and the models in model library, finally several models most similar to the input one are returned.

Fig. 4. Architecture of our system

In this method, we need choose two appropriate parameters:

L

K

L and K where

∈ Z . As for L , the larger L is, the better the voxelized model approaches the

original one, but the computation is accordingly increasing. The same discussion to K , the larger K is, the subtlier the entropy information is expressed, but the pertinence among voxels decreases. In our experiments, we tested several combinations of L and K , in which we find that when L = 100 and K = 5 the arithmetic obtains optimal performance(perhaps related to model library), and in this case the dimension of the feature vector is K = 125. Fig.5 shows a series of 3D CAD models of plane and the distributions of their feature vectors. 3

878

W. Liu and Y. He

Fig. 5. 3D CAD models and corresponding feature vectors

5 Conclusions and Feature Work Retrieval and reuse of 3D CAD models in part library is an effective way that can significantly reduce development costs and time. For this reason, the paper has presented a novel method for retrieval of 3D models, extraordinarily for CAD models. We use voxelizaion to representation the original 3D model, then calculate the entropies of all sunspaces as feature vector, and experiments preliminarily validate our method. As for the further work, we plan to orderly organize and delaminate the models library according to the PLIB standard[22] to make the similar models stored in the same categories, so the system can quickly eliminates the obvious dissimilar models. The future aim of our system is that each input model can automatically find its category and be stored in corresponding position to quicken and optimize the retrieval performance in online retrieval. Acknowledgements. This research has been funded by The National Natural Science Foundation of China (60573146).

References 1. Burbridge, J. Group Technology in the Engineering Industry. London: Mechanical Engineering Publications, 1979. 2. Paquet E, Rioux M. A query by content software for three dimensions database management. Proceeding of International Conference on Recent Advances in 3D Digital Imaging and Modeling, Ottawa, Canada, 1997, 345-352 3. Paquet E, Rioux M. A content-based search for VRML database. Proceeding of the 1998 Computer Society Conference on Computer Vision and Pattern Recognition, Santa Barbara, CA, 1998, 541-546 4. Ankerst Mihael, Kastenmller Gabi, Kriegel Hans. 3D Shape Histograms for Similarity Search and Classification in Spatial database. Proceeding of the 6th International Symposium on Large Spatial Database, Hong Kong, 1999, 207-226 5. Suzuki T, Kato Toshikazu, Otsu Nobuyuki. A Similarity Retrieval of 3D Polygonal Model Using Rotation Invariant Shape Descriptors. Proceeding of IEEE International on Systems, Man, and Cybernetics, Nashville, Tennessee, 2000, 2946-2952


879

6. Osada Robert, Funkhouser Thomas, Chazelle Bernard. Shape Distributions. ACM Transactions on Graphics, 2002, 21(4): 807-832 7. Vranic D V, Saupe D. 3D Shape Descriptor Based on 3D Fourier Transform. Proceeding of the EURASIP Conference on Digital Signal Processing of Multimedia Communications and Services, Budapest, Hungary, 2001, 271-274 8. Vranic D V, Saupe D. Description of 3D Shape Using a Complex Function on Sphere. Proceeding of the IEEE International Conference on Multimedia and Expo, Lausanne, Switzerland, 2002, 177-180 9. Vranic D V , Saupe D. Tools for 3D object retrieval: Karhunen-Loeve Transform and spherical harmonics [A]. In : IEEE Workshop on Multimedia Signal Processing , Cannes, 2001.1 293-298 10. Chen Dingyun, Ouhyoung Ming. A 3D Object Retrieval System Based on MultiResolution Reeb Graph. Proceedings of Computer Graphics Workshop, Taiwan, China, 2002, 16-20 11. Hilaga Masaki, Shinagawa Yoshihisa, Kohmura Taku. Topology matching for fully automatic similarity estimation of 3D shapes [A]. In : Computer Graphics Proceedings , Annual Conference Series, ACM SIGGRAPH, Los Angeles, CA ,2001.1, 203-212 12. Min Patrick, Chen Joyce, Funkhouser Thomas. A 2D Sketch Interface for a 3D Model Search Engine. Computer Graphics Proceedings, Annual Conference Series, ACM SIGGRAPH 2002 Technical Sketch, San Autonio, Texas, 2002 13. Chen Dingyun, Tian Xiaopei, Shen Yute. On Visual Similarity Based 3D Model Retrieval. Computer Graphics Forum, 2003, 22 (3): 223-232 14. Jobst Loffler. Content-Based Retrieval of 3D Models in Distributed Web Databases by Visual Shape Information. IEEE International Conference on Information Visualization, London ,UK, 2000.1, 82-87 15. Suzuki M. A web-based retrieval system for 3D polygonal models. In : Proceedings of Joint 9th IFSA World Congress and 20th NAFIPS International Conference , Vancouver , Canada , 2001 , 2271-2276 16. Elinson A., Nau D. S, Regli, W. C. Feature-based Similarity Assessment of Solid Models. Proc ACM Solid Modelling Conference, 1997, May 17. Sun T. L, Su C. J, Mayer R. J. Shape similarity assessment of mechanical parts based on solid models. Proc of the 1995, ASME DETC, Vol.83, No.2 Pt 2, pp. 953-962, Boston, MA, USA, Sep 17-20 18. McWherter D, Peabody M, Shokoufandeh, A. Database techniques for archival of solid models. Proc Solid Modeling 01, pp. 78-87, Ann Arbor, Michigan, ACM Press 19. Kaufman A, Shimony, E. 3D Scan-Conversion Algorithms for Voxel-Based Graphics'. Proc. ACM Workshop on Interactive 3D Graphics, Chapel Hill, NC, October 1986, 45-76 20. Jian Huang, Roni Yagel, Vassily Filippov. An accurate method for voxelizing polygon meshes. IEEE Symposium on Volume Visualization, 119-126 21. Qingsheng Meng. Theory of Information. Press of Beijing University of Technology, 1991 22. Cao Yunfu, Li Feng, Han Yongsheng. Information Model of Parts Library and Its Representation Based on PLIB. Journal of Computer-Aided Design and Computer Graphics, 2002, 14(6): 598-602

Virtual Tang-Style Timber-Frame Building Complex* Deren Li1, Yixuan Zhu2, Zhiqiang Du1,** , and Tao Hong1 1

State Key Lab of Information Engineering in Surveying Mapping and Remote Sensing, Wuhan University, 129 LuoYu Road, Wuhan, Hubei, 430079, China [email protected], [email protected], [email protected] 2 School of Remote Sensing Information Engineering, Wuhan University, 129 LuoYu Road, Wuhan, Hubei, 430079, China [email protected]

Abstract. Timber-frame building is one of gems of Chinese ancient buildings. Protecting and researching historic buildings by using computer technologies has become a hot topic and urgent task in the field of digital cultural heritage, with increasingly disappearing the timber-frame buildings and their architectural arts and crafts. This paper introduces constructing an information system based on virtual reality technology of timber-frame buildings, and addresses important technologies involved in reconstructing ancient buildings, such as data collection, 3D reconstruction, data organization and management, virtual assembly, and virtual roaming. An integrated solution is proposed and provides a brand-new method for 3D reconstruction, documentation, repair, research and protection of timber-frame buildings. Keywords: digital cultural heritage, timber-frame building, 3D construction, virtual assembly, 3D roaming.

1 Introduction Digital Cultural Heritage (DCH) is to research and protect the world history, cultures, and natural heritages by utilizing computer technologies. DCH emphasize that it should link people in the virtual environment with human culture tradition, who will realize function and orientation of current culture in the human history, and make a experience from furthest foretime, recently to future in the historical corridor. Multi-dimensional information is expressed by DCH, which realizes the truth of history, has the cultural art, and possesses the fidelity of real environment. The process of the multi-dimensional information not only is supported by highperformance hardware, but also involved in the developed computer technologies, which include Virtual Reality (VR) as one of the key technologies. It is an important measure to apply and synthesize photogrammetry, architecture and archeology for collecting the multi-information. *

This research is funded by the project titled the Digital Architectural Information System of Tang-Style Temple Complex for Chi Lin Nunnery. ** Corresponding author. Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 880 – 888, 2006. © Springer-Verlag Berlin Heidelberg 2006

Virtual Tang-Style Timber-Frame Building Complex

881

As a new technology, VR is extensively concerned in science and engineering in the last decades. It produces one kind of simulation environment using the computer, and realizes the interactive action between the users and the virtual environment in which the user immersing through many kinds of sensory equipment. Using 3D digital reconstruction of ancient buildings or places within VR, it is possible to simulate, discuss, and evaluate diverse solutions to archaeological, cultural, or restoration-related problems without touching the original. A novel research platform and an enjoying approach are provided to kinds of users by applying several techniques, such as virtual preservation, information demonstration and 3D visualization.

2 Chinese Ancient Timber-Frame Building Sui and Tang Dynasty are the mature period during the development course of Chinese ancient architectural system and scale the peak of ancient architecture in the Chinese feudal age. The ruins of an ancient city, palaces, tombs, grottoes, pagodas, and bridges which leave over from Sui and Tang Dynasty have very high level of architectural art and technology no matter distribution and shaping, especially the sculptures and frescoes. The distribution of Chinese ancient palaces and temples employs the manner symmetrical. The main buildings are centering and have longitudinal axis; a courtyard is composed through covered corridors connecting some buildings which are constituted by rooms; and the courtyards as unit elements spread along the axis to form a depth arrangement. The feature of Tang timber-frame architecture takes the timber frame as structure system, in which the columns, TouKung and beams as the essential elements compose the weight-bearing system and mortise and tenon in the timber components are fit each other to grouping a frame full of elasticity [1]. The architecture, especially timber-frame building, is characterized by magnificent, unpretentious and religious architectural style. It has broad and curved roof, unadorned and practicable doors and windows, and concise and bright color.

Fig. 1. (a) The timber-frame building complex. (b) A hall in the complex. (c) A set of Toukung.

The wood, as the basic material of buildings in Tang Dynasty, is easier to be perishable and the construction of timber structure can be easily razed by wars or religious struggles; many crafts have seldom survived to this day due to the manual skills and crafts of timber-frame building are orally passed; and a few literatures

882

D. Li et al.

record the arts and crafts about them hardly exist, so that the research and protection about timber-frame buildings in Tang style become a critical task, which includes how to represent the vanished or destroyed historic buildings and sites and how to preserve cultural information about the ruins of ancient buildings. This also is the main object of historic building research using computer technologies in DCH.

3 Virtual Tang-Style Timber-Frame Buildings Chi Lin Nunnery, which is a famous Tang-style timber building complex, is on a grand scale; is great and momentous; and is magnificent but not stiff. Virtual architectural information system of Tang-style timber-frame building complex for Chi Lin Nunnery can bring forth a new technology for documentation, research and protection of Chinese ancient buildings. 3.1 Data Collection Collecting real 3D data is the essential prerequisite for reconstructing 3D model of virtual building complex. It is important that different modeling method is corresponding different data resource, for example, the modeling method based on remote-sensing image and LIght Detection And Ranging (LIDAR) is suitable for modeling large-range 3D city; vehicle-carried photogrammetry can apply to modeling 3D landscape beside of street; a single building or buildings are usually modeled by engineering survey, close-range photogrammetry, or laser scan on ground [2]. 3D block models based image modeling methods can not particularly describe the all details or features of buildings although they can be used in rapid modeling urban scene. Computer Aided Design (CAD) technology has a unique advantage in graphic process and real 3D modeling. A 3D CAD model is composed of one or more polyhedral models and can represent detailed geometric characteristics of a building. These models demonstrate not only the detailed appearance but also inner complicated structure of objects comparing with the models established by imagedbased modeling method. But the modeling process is involved in sophisticated manmachine interaction, overload manual work, and higher cost. In the practical modeling for virtual Tang-style building complex, several data collecting methods are utilized: engineering survey, close-range photogrammetry, ground laser scan, and architectural design blueprint. For the regular timber components, the traditional survey can satisfy the requisition of data precision and obtain the effective featured point. But the data received by this method usually have the problem that the 2 adjacent surfaces have not a shared line. It is required to think carefully about sample frequency of featured point [3]. For the irregular timber components, like Chiwen on the roof, laser scan is the best method for data collection although the degree of the automatic processing point cloud is not in high level yet. The textures are mapped onto the 3D model is another effective method for obtaining photorealistic effect. But the textures collected by digital camera are usually processed by color or geometric correction. This method is usually suitable for collecting data of the architectural decoration [2].


883

The architectural design blueprint has become another important data resource. 2D CAD data provided in blueprints can help survey sampling and quality checking, and moreover, it can be converted to 3D model through giving special reference data.

Fig. 2. (a) The workflow of supervision for a timber-frame building. (b) Model Normalization. (c) Surveying. (d) Data sampling. (e) Texture mapping. (f) Blueprint.

3.2 3D Modeling 3D modeling based on CAD is the best choice to reproduce the structural characteristics of timber-frame building and make possible virtual assembling, dynamic browsing, interior and exterior data management and component information inquire. Volume-based model in CAD can build complicate forms by composing simple volumes, and architectural component can be digitalized quickly on the base of second developing [4]. Timber-frame buildings are composed of thousands or even millions wooden components, so 3D models are highly complicate and have a huge data volume. Even high performance graphic computer can't render scene with fluent frame rate. Modeling reductionism basing on CAD 3D data is the important method in the process of 3D reconstruction. The usual solution is to reduce the amount of triangles and vertexes for simplifying the scene. But LOD is the method to describe less important visual subjects with simple geometry forms, which includes discrete and continuous LOD. Continuous LOD model's generation algorithm is applicable to regular modern architecture and macro-scale urban model. Discrete LOD can improve the efficiency of rendering greatly by taking measures to reduce visual mutation phenomenon. At the same time it can find a balance between rendering efficiency and representing effect to meet the demands of representing historic Timber-frame architecture.

884

D. Li et al.

Fig. 3. 3D models of a hall with different levels of details

The method based on LOD and CAD incarnates fully the idea from complexity to simplicity during 3D modeling, filling the requirements of acquiring the possible best quality of rendering images and achieving virtual roam at smooth frame rates [5]. Furthermore, it is important to determine how many levels of discrete LOD models to establish. The limited system resource would be used in over-frequently locating and loading huge volumes of data from storage so that real-time rendering is not smooth enough considering too many levels of a model are plotted out, obviously it conflicts with the principle of LOD. But the “popping effect” caused by models switching would be notable if plotting out over-less levels, especially when outline of model needs to be kept consistent. In fact, 4-5 levels discrete 3D models with different detailed degrees would be a feasible scheme for real-time rendering of virtual tang-style timber-frame building complex [5].


885

3.3 Data Organization and Management For virtual Tang-style building complex, more fine degree of 3D CAD model is, much more data of model will be. It is important to establish an appropriate spatial index to accelerate calling and invoking 3D vector model data [6]. LOD-R-trees, which is perfectly suited to organize and manage large amounts of 3D data, as shown as Figure 5, the majority of models in the scene are subdivided into several parts according the different detailed representation, and they can become integrated effectively by R-trees index. The disassembled models can also serve for 3D space partitioning. Moreover, the 3D models based on LOD-R-trees are contributed in smoothly real-time rendering and meet the goal of multi-scale spatial query and spatial analysis by integrated applications of database management techniques [7]. In fact, an independent-developed 3D model database management engine (DBME) that is based on the integration of Oracle 8i/9i database and file management system, allows for efficient management and manipulation of 3D data and other data within the visualization environment.

Fig. 4. Data organization based on LOD-R-tree [5]

3.4 Virtual Assembly Virtual assembly not only can be utilized in engineer quality check, but also is helpful for structure and craft research of ancient timber-frame building [8]. It is possible to face with dismantlement and reworks for constructing or repairing an ancient building because there are errors in the process of design or machining and the timber-frame building is made up of components through mortise and tenon accurately. So the mistakes and errors will be found in time by virtual assembly, which provides a solid foundation for design and construction. The process of virtual assembly can entirely show the whole structure of building, can analyze structural relationship among components in important nodes, and also can make the finite element analysis combining with the theory of architectural mechanics.

886

D. Li et al.

Planning the process of the virtual assembly is the first step, which includes recording the order of assembly and path information and considering the physical attribution of models. And then the 3D component model is imported into the virtual assembly system from CAD platform. The original position of every component is valued by practical survey data. The viewpoint and watching direction can be adjusted freely and the operation including pan, rotation, scale, can be finished conveniently. The system can achieve real-time collision detection and constraint discernment and generate the assembly animation according to the movement path of components. Collision detection, a key technology for virtual assembly and dynamic roaming, can avoid the penetration between the viewpoint and objects in order to promote the fidelity in the navigation. The combination of collision detection algorithm based on object space and based on image space can enhance the capability of detection, that is, firstly to analyze intersection of scene objects based on 2D projection and depth information in reference frame, then to compute intersection of points, lines and faces of the “intersecting” objects.

Fig. 5. A sample of assembling process from image (a) to (f)

3.5 Virtual Roaming Verisimilitude of display and real-time response are two important guidelines to evaluate visual function [9]. While roaming in the dummy wood-frame buildings scene, image quality is the same important as response time. Data needed to render always exceed them though the capacities of graphic hardware have been advanced, so it is necessary to use software techniques to resolve the problem. Some acceleration techniques, such as occlusion culling, LOD, and programmable rendering, are adopted for balancing between the highest rendering quality and smooth animation. Occlusion culling is designed to cull the invisible objects during rendering. When a dummy visitor is walking through the scene, because of being occluded by other


887

buildings closed the viewpoint, many buildings and decorations are not visible although they are within the view frustum. The most culling algorithms are based on partitioning the view position space or finding out the set of potential visible objects. The real-time culling algorithm used in the system is sufficient to utilize the spatial and temporal consistency. The algorithm is sufficient to cull the majority of invisible objects if the user is walking through the scene, though it is not efficient for the height of viewpoint above the roofs. Fortunately, the navigation of users is mainly used in simulating human activities in the virtual scene, and the system will automatically call LOD4 models for a complement of display speed depressed by occlusion culling invalidation when the height of viewpoint above the roofs. It is difficult to achieve the satisfied display speed although the multi-levels data management and occlusion culling technique are adopted in rendering process. Programmable rendering technique is one of the most effective approaches for obtaining a balance between image realism and smooth display, whenever flying over or walking through the scene, it can optimize the refreshing frequency and graphic quality of rendering by batch program of vertex and illumination computing of segment render, and improve evidently verisimilitude of virtual roaming.

Fig. 6. The image on the left is a snap of 3D virtual scene and the image on the right is a photo of Chi Lin Nunnery

4 Conclusion and Future Works An architectural information system based on virtual reality technology of Tang-style timber-frame building complex is introduced particularly. The experiment suggests that survey, 3D laser scan and close-range photogrammetry are the effective way for obtaining multi-resource spatial information, especially the 3D laser scan promote the efficiency and quality of collecting data. The modeling method integrated LOD with CAD is suitable for 3D reconstruction of timber-frame building. This method not only really represents the detailed appearance features, but also makes possible for smooth roaming. LOD-R-tree, a spatial index, provides a solid foundation for multiple levels query and dynamic visualization combining database management engine. Virtual assembly supplies a new method for craft research of timber-frame buildings. Finally, interactive virtual roaming shows the architectural art of Tang-style timber-frame building complex sufficiently.

888

D. Li et al.

It is an important for utilizing the computer technologies to protect and research historic building in digital heritage. The technologies, such as fast collecting data, automatic 3D reconstruction, multi-resolution rendering will become the important development for 3D modeling and dynamic visualization.

References 1. Liang, S.: A Pictorial History of Chinese Architecture. Baihua Literature & Art Publishing House, Tianjing (2001) 2. Li, D., Yang, J., Zhu, Y.: Application of Computer Technique in the Reconstruction of Chinese Ancient Buildings. Proceedings of the ISPRS commission V symposium, Corfu, Greece, Intern Archives of P&RS Volume XXXIV, Part 5 Comm. V. (2002) 404-406 3. Li, D., Hong, T., Zhu, Y., Yang, J.: 3D Reconstruction and Simulating Assembly of Ancient Chinese Timber-frame Building. In the proceedings of CIPA 2005 XX International Sympsium, Torino, Italy (2005) 906-911 4. Xu, M., Zhu, Y.: Computer-Based Simulation and Quality Control for Building. Working Group V/5, ISPRS, (1998) 252-258 5. Du, Z., Li, D., Zhu, Y., Zhu, Q.: 3DGIS-Based Reconstruction and Visualization of Timberframe Building Cluster. Xitong Fangzhen Xuebao/Journal of System Simulation. Vol.18, No.7 (2006) 1184-1189 6. Zhu, Q., Lin, H.: Digital City Geographic Information System. Wuhan University Press, Wuhan (2004) 7. Berndt, E., Carlos, J.: Cultural Heritage in the Mature Era of Computer Graphics. Computer Graphics and Applications, IEEE, Vol. 20, No.1 (2000) 36-37 8. Zisserman, A., Fitzgibbon, A., Baillard, C., Cross, G.: From Images to Virtual and Augmented Reality. In: Confluence of Computer Vision and Computer Graphics, NATO Science Series. Kluwer Academic Publishers (2000) 1–23 9. Gruber, M.: Managing Large 3D Urban Databases. 47th Photogrammetric Week (Dieter Fritsch and Rudi Spiller, editor). Wichmann, Germany (1999) 341-349

An Accelerating Rendering Method of Hybrid Point and Polygon for Complex Three-Dimensional Models Aimin Hao, Guifen Tian, Qinping Zhao, and Zhide Li Ministry of Education Key Laboratory of Virtual Reality Technology, Beihang University, Beijing, China [email protected]

Abstract. This paper presents an accelerating hybrid rendering method for complex three-dimensional models using both points and polygons. In circumstance of current PC hardware, our rendering method that integrates advantages of both GBMR (Graphics -Based Modeling and Rendering) and PBMR (Point-Based Modeling and Rendering) works well with complex geometric models which consist of millions of small triangles. In the pre-processing phase, model faces are segmented into regions, triangles and vertex point clouds of each region are stored, and all vertices are sorted in an ascending order according to their importance degree and serialized into a linear structure. In the real-time rendering phase, view-dependent frustum culling and backface clipping are performed; different regions are rendered using triangles or points according to their distance from the viewpoint. These two phases fully utilize the parallelism nature of GPU (Graphic Process Unit) and efficiently implement a continuous multi-resolution rendering for HPPO (Hybrid Point and Polygon Object). Keywords: Virtual reality, Modeling and rendering, Real-time property, Point clouds, Continuous multi-resolution models.

1

Introduction

Recently, the number of complex geometric models get by 3D scanning technique has increased a lot. Usually these models contain high surface details and dense vertices. As a result, these models contain huge data information and hence the real-time processing is tough. In order to resolve such problem, researchers have tried various techniques, such as LOD (Levels of Detail)[1], view-frustum culling [2], image transformation [3], and Light Field Rendering [4]. These techniques all help to improve performance, but pity that limitations go with them at the same time. Point-Based Rendering (PBR) is a new accelerating rendering technique recently developed. It takes the geometric point as its rendering primitive and

Supported by Opening Foundation of Beijing Key Laboratory of Multimedia and Intelligent Software in Beijing University of Technology.


890

A. Hao et al.

works well with the rendering of large-scale complex models with rich, organic shapes or high surface details. In contrast to the polygon-based technique, this method does not need to store topologic information between vertices, which does a lot of good for efficiency. However, it has some disadvantages as follows: It requires a lot of projection, clipping and transmission of large numbers of geometric points and has a high requirement in hardware pixel filling. Another disadvantage is that holes and alias usually appear during this rendering technique. To avoid these holes and alias, many Point-Based Rendering methods (such as Far Voxels [5] in SIGGRAPH 2005) request that point sampling on object surfaces should meet sufficient sampling conditions and that the viewpoint should not be too close to the target object. Obviously, the latter request makes arbitrary interactivity impossible. Hybrid rendering can solve partial problems mentioned above and currently it has become the prevailing method of modeling and rendering. Hybrid of point and polygon [6,7], one of such hybrid rendering methods, uses triangles, points and images together to present the same model. To go into detail, this method uses points with textures to render objects far away from the viewpoint and uses texture-mapping triangles to render objects close to the viewpoint. It has been proved that this hybrid method can successfully avoid holes, which may appear when purely applying point-based method to render close or middle-positional objects. At the same time, rendering speed is notably improved. Here, referring to the nomenclature for IBO [8](Image-based object) , BVO [9] (Volume-based object), V Object[10] (Object in virtual environment) defined by former researchers, our paper defines HPPO (Hybrid Point and Polygon Object). HPPO is especially applicable to present two-dimensional continuous manifold models in virtual environment, which may contain dense and complex geometric information, and this presentation itself is continuous and multi-resolution. Further more, our paper presents a new accelerating technique of hybrid of points and polygons, which is competent for the rendering of large-scale, dense geometric models. By choosing polygons or points as rendering primitives, our method gets a good balance between rendering speed and image quality.

2

Related Work

There are two disadvantages [6] inherent to pure point representation. First, once objects are represented in point representation, their resolution is fixed. When the resolution of the destination image is larger than the maximal resolution that can be presented by points, interpolation between adjacent points must be done to ensure smooth shading. Second, for large, flat surfaces, point rendering becomes less efficient than polygon rendering. To resolve this problem, Chen et al. put forward an algorithm called POP [6], which in nature is an extension of pure point rendering systems such as Qsplat [10]. For POP algorithm starts from triangles as leaf nodes in its hierarchical tree structure: when the screen contribution of the node is larger than one pixel, the node is presented by a

An Accelerating Rendering Method of Hybrid Point and Polygon

891

polygon rather pure points In this way, POP can accelerate the rendering of large models, compromising little in image quality. Improvement made by POP lies in that QSplat only works well with models consisting of triangles of approximately equal sizes, while POP works well for any triangular model. Wimmer [11] substitutes point-based impostors for distant geometric models.This approach relies on object-space sampled representation similar to point clouds or layered depth images; therefore the appearance of holes is successfully prevented at sampling phase. For large, complex models, as sample points are not enough to make a complete surface of an object, Wimmer computes the contribution of these points to the image using Monte Carlo integration. The resulting representation is compact, and by using existing graphic hardware, renders can produce a high quality image with little aliasing, and no missing or disappearing objects as the viewpoint moves. Another rendering system PMR presented by Dey et al. [12] builds an octree of points and triangles similar to POP. However, unlike POP, the hierarchy in PMR is built on input point clouds, and then the Voronoi diagram is used to estimate local features of the point clouds and simplify them in the object space. Continuously, the simplified result is transformed into multi-resolution triangles. Similar to other mentioned rendering methods, PMR chooses points to render distant objects and triangles to render close objects. A simple userdefined parameter can be used to control the balance of points and triangles, thus achieving the speed/quality tradeoff. However, simplification of point clouds is consumptive and inefficient. Similar work researched by Liviu Coconu [13] is Hardware-Oriented hybrid Rendering with the layered depth octree as its basic data structure. As regard to domestic work on hybrid point-polygon rendering, P-Quadtrees algorithm presented by Junfeng Ji [15], could be a representative. Another researcher, Yanci Zhang, also makes some contributions.

3 3.1

Hybrid of Point and Polygon Rendering Basic Definitions

Point cloud: A set of dense discrete points. Mathematically, it can be represented asP = {Pi }i∈index(P ) , Pi ∈ IR3 . IR3 represents Euclid space, index(P ) represents the index set of sample points, usually in [1, n]. Here, suppose n = |P | represents the number of points in a point cloud. Surfels: A sort of rendering and presentation primitive, defined by the local geometric attributes of a small surface that the sample point is on. Typically, sample points are acquired by 3D scanning. Dense geometric model: A sort of triangle mesh. If more than 30% of the triangles’ screen width is less or equal to 1 pixel, we can call it a dense triangle mesh or a dense geometric model. Typically, high dense meshes acquired by 3D scanning belong to this sort.

892

A. Hao et al.

Fig. 1. Procedure of the accelerating hybrid rendering

3.2

Operating Principle of Our Hybrid Rendering Method

The accelerating rendering method of hybrid point and polygon introduced in our paper takes the complex, dense manifold triangle model as its processing object. There are two main steps: pre-processing and real-time rendering as shown in Fig 1. Pre-processing sees to data re-organization of the input geometric model. Concretely, it performs region segmentation on model mesh faces, so the input model is segmented into sub-models. Meanwhile, the importance degree of each vertex is estimated. According to their importance degree, vertices are re-sorted in an ascending order and stored in a linear structure. After this, we get a new model and we call it HPPO (Hybrid Point and Polygon Object) in our paper. Through the pre-processing phase, the number of triangles of the input model remains the same, but the number of vertices increases a little bit because vertices on the verge are stored more than once. Then comes to the real-time rendering. During this phase, we can fully utilize the parallelism nature of GPU (Graphic Process Unit) and its vertex shader. In detail, triangles or point clouds of sub-models are chosen as the rendering primitive according to the distance from the viewpoint to the center of each sub-model, that is, triangles are chosen to render close targets and point clouds to render distant targets. As the distance becomes larger, the number of points needed becomes continuously smaller, and thus we can achieve a continuous multi-resolution point-based rendering. 3.3

Region Segmentation Algorithm for the Geometric Model Mesh

In the pre-processing phase, our paper presents a region segmentation algorithm directed by the frame rate feedback. And it is based on dynamic normal clustering. Its aim is to segment the input model into normal related sub-models, so


893

that in the rendering phase we can accelerate backface clipping and continuous point cloud rendering for each sub-model. In our algorithm, segmentation depends on the divergence of each triangle’s normal vector. Concretely, normals in the same sub-model or group should be as adjacent as possible, while those in different sub-models or groups should be as disjunct as possible. With segmentation completed, the original model is separated into sub-models, and each sub-model has a normal cone. In addition, region connectivity is considered to ensure that sub-models are topologically continuous. Pay attention that the number of triangles produced by segmentation is limited within an appropriate range, neither larger than the maximal value allowed nor smaller than the minimal value controlled. In this way, sub-models are properly controlled in scale. The following steps show how our segmentation algorithm works. 1. Initialize the controlling parameter, and use the neighbor extension method to implement neighbor clustering on the input model. 2. Re-compute the average normal of all clusters. Take the results of step 1 as the initial segmentation. Conduct the second clustering using ISODATA algorithm. 3. Unite the small clusters that may appear in some special cases. 4. Test the current rendering frame rate according to clustering results. 5. Adjust the controlling parameter according to the frame rate tested in step 4. Then go back to step 1 to iterate the controlling parameter until the maximal frame rate is reached. Then end clustering. Specifying the three parameters by hand is too arbitrary. Besides, the result of clustering is also uncertain. Therefore, step 4 tests the current frame rate and uses it as the feedback to iterate AngleofNormal. As a peak point does exist on the curve of the frame rate and the clustering result, iteration is convergent and thus we can get an optimal value for AngleofNormal. The round-looking test function is adopted in our algorithm to test the frame rate, that is, to suppose that the viewpoint locates on a circle with the target object as its center. As the viewpoint changes its position but remains on the circle, we compute the average frame rate and take it as the rendering frame rate of mesh segmentation. After region segmentation on the model surface, the input model is separated into a group of sub-models with the maximal rendering frame rate. 3.4

Importance Degree Sorting Based Continuous Multi-resolution Point Cloud Presentation

Taking vertices of each sub-model acquired in 3.3 as a point cloud, this section will talk about a new continuous multi-resolution presentation of point clouds. By this kind of presentation, we can utilize GPU to accelerate rendering. See Fig. 2 to get a clear understanding. The rectangle above represents a linear vertex data pool. Pay attention to the color of the rectangle: the heavier it becomes, the more important the vertex is. Variable t, which varies from 0 to 1, represents the importance degree of vertices.

894

A. Hao et al.

Fig. 2. Control the production of continuous LOD according to importance degree of vertices

Obviously, when t becomes closer to 1, fewer vertices are needed to form a point cloud and the current point cloud is in a low-resolution state; whereas, when t becomes closer to 0, the point cloud is in a high-resolution state. Remarkably, in a low-resolution state, vertices hold a high importance degree and our rendering algorithm can preserve the appearance of models. How to define the importance degree of each vertex is a key issue. Through comparison with PM [14](Progressive Meshes), we try to give a new definition. For all triangles adjacent to the current vertex, we compute dihedrals between every two triangles that share the same edge, and then average them; for all edges adjacent to the current vertex, we compute their average length. Then we use these two averages to define the importance degree of the current vertex. ¯ to represent the average length of all edges. Thus Import L nb 1 ¯ L= Length(Li ), nb i=1

(1)

nb shows the number of edges that are adjacent to the current vertex, namely the degree of the current vertex. Function Length() computes the length of each edge. L1 , L2 ......Lnb represent all adjacent edges. Import Ψ¯ to represent the average dihedral. By mathematic deduction, we have following results: (1) if current vertex is an inner vertex, then nt 1 Ψ¯ = Angle(Tj , Tmod((j+1),t) ), nt j=1

(2)

(2) if current vertex is an border vertex and nt = 1, then Ψ¯ =

(nt −1) 1 Angle(Tj , T(j+1) ), (nt − 1) j=1

(3)

(3) if there is only one adjacent vertex for current vertex and nt = 1, then Ψ¯ = π. nt shows the number of adjacent triangles of the current vertex. Function Angle() computes the dihedrals between every two triangles. Tj , Tj+1 , j = 1, 2.... represents triangles which are adjacent to the current vertex and share the same edge. Pay attention that Tnt and T1 share the same edge. We can see that vertices are divided into three types: the inner vertex, the border vertex and the isolated vertex (with only one triangle adjacent to it). For each type, we can get the average dihedral from the above mathematic expression easily.


895

Now, using the linear combination of the average dihedral and the average length, we can define the importance degree of a given vertex: ¯ + (1 − α)Ψ¯ , Important(p) = αL

(4)

denotes normalization of numeric values. Here, Ψ¯i =

Ψ¯i , max(Ψ¯ )

(5)

L¯i =

L¯i ¯ , max(L)

(6)

i varies from 1 to the number of vertices. Function max() computes the maxΨ¯ ≤ 1, imal average dihedral or average length. By normalization, we ensure L ¯ ≤ 1. α is the coefficient of linear combination. By appointing different values of α , ¯ and Ψ¯ to the importance we can control the contribution of the two parameters L degree of the current vertex. In conclusion, our presentation of the importance degree of vertices has several advantages: (1) Using the average dihedral and the average length together to define the importance degree of vertices can avoid many problems that may appear in quadric error measuring (QEM). And it also can improve the presentation of high curvature positions. (2) Compared with quadric error measuring (QEM), importance degree computing in our algorithm requires shorter time and less space. (3) By appointing different α , we can adjust the contribution of ¯ and Ψ¯ to the importance degree of the target vertex. L 3.5

Real-Time Rendering of HPPO

The real-time rendering phase implements normal cone based backface clipping of sub-models first, and then it carries out a continuous multi-resolution rendering using point clouds. In this rendering, view-dependent LOD selection is also performed. Combine viewpoint parameters and point clouds that are stored as a linear structure in section 3.4. When given the viewpoint, we can compute levels of details of each sub-model in HPPO. Main data structures in LOD selecting algorithm include the object-level attribute definition of HPPO and the sub-modellevel attribute definition of HPPO. Four parameters related to LOD selection are listed below: Object-level parameter defined in LOD selection consist of the following three: (1)m f N earClip, represents the near control point distance of point clouds; (2)m f F arClip, represents the distant control point distance of point clouds; (3)m f P ercent, the percentage of points used for point clouds. Sub-model-level parameter defined is ”center”,which is the center coordinate of sub-models.

896

A. Hao et al.

LOD selecting algorithm for HPPO is described below in detail. Step 1: for each sub-model, compute the distance from its center to the current viewpoint. Step2: If distance < m f N earClip, select triangles to render the current sub-model. And it represents as the highest level of details. If m f N earClip ≤ distance ≤ m f F arClip, select the point-based rendering method to render the current sub-model. Levels of details of the sub-model linearly change with distance. At this point, the sub-model represents as a serial of mid-level continuous LOD. The percentage of surfels involved in rendering is computed as below: percent =

(1 − m f P ercent)(m f F arClip − distance) + m f P ercent, (7) (m f F arClip − m f N earClip)

If distance > m f F arClip, we also choose point-based rendering. However, let percent = m f P ercent. The number of surfels involved in rendering no longer decreases so as to preserve visual attributes of HPPO. And at this time, the sub-model represents as the lowest level of details. Step3: For each sub-model of HPPO, implement LOD selecting rendering in turn. As a sort of GROUP node, HPPO can be easily added to synthetic scenes. The instance of HPPO, which also can be viewed as a GROUP node, provides a container for the appointed object to display and update. So an object in a synthetic scene like Buddha can be represented in many nodes, namely, it can be instantiated as objects in many scenes. HPPO nodes contain specific threedimensional coordinate information. Besides, they define implicative and subordinate relationships between objects. HPPO in our algorithm doesn’t contain other rendering information, and the rendering is completed by the transformation matrix and the objects subordinate to low layer nodes. Notice that the matrix is acted on the current HPPO node. The object layer of the HPPO node provides the detailed rendering method, including: (1)DWORD IPointCloudObject::OnCreate ObjectInit * pParams),this function initializes the point cloud object and perform the matrix transformation (including scaling, rotation, translation and etc.) with parameters provided by the controlling node. (2)DWORD IPointCloudObject: Release (void),this function echoes with the function ”OnCreate” defined abovewhich releases all applications and memorythen deletes the current point cloud object. (3)void IPointCloudObject::Update(void) (4)void IPointCloudObject::Render(DWORD dwRenderFlags) These two functions are the key procedure in the emulation loop. ”Update” is a must in every loop but ”Render” is not, for the display may need to be optimized. (5)PROCESSRESULT IPointCloudObject::OnInputMessage (DWORD dwMSG, WPARAM wParam, LPARAM lParam ), this function is responsible for message processing and parameter updating. It enables the current object to react to specific controls and thus achieve interactive effects in time.


4

897

Experiments and Results

Our hybrid of point and polygon algorithm is implemented and verified by using C++ and DirectX 9.0 SDK, on an ordinary PC with the PIV3.0G CPU, the 256M memory, and the Nvidia6200 128M display card. Tested models come from the standard model base of Stanford University, including Happy, Dragon, Bunny and Armadillo. Related data are described in Table 1. Table 1. Tested model data model name number of triangles number of vertices Happy 1087474 553898 Dragon 871306 447600 Bunny 69451 34834 Armadillo 345944 174840

4.1

storage 41621K 33039K 2963K 6757K

Verification of Region Segmentation for the Geometric Model Mesh

Fig. 3 shows the relational curve of the normal threshold parameter and the rendering frame rate. It reflects two important facts. First, region segmentation on the model mesh surface is significant for efficiency. Without segmentation, the rendering frame rate of Happy is 17 fps or so. However, using our rendering algorithm with AngleofNumber set to 18 (4518 clusters), the frame rate increases to 49.44 fps, nearly triples the former. Second, it is not always the case that more sub-models lead to higher rendering efficiency. On the contrary, when the number of sub-models comes to a point that the cost of employing rendering API surpasses the processing ability of the hardware, the rendering efficiency will be greatly damaged. From Fig.3 we can also see that the curve is single-peaked. so our segmentation algorithm can be fast converged. 4.2

Verification of Importance Degree Definition ¯ and Ψ¯ (defined in section Given different α , we verify the contribution of L 3.4) in the importance degree formula (4). When α = 1, the importance degree ¯ When α = 0, the importance degree only depends on Ψ¯ . only depends on L. Fig 4. shows the rendered image when α = 1 and α = 0 respectively. In order to observe the distributing effect of point clouds, our verification chooses Bunny rendering object, for its point cloud density is comparatively low. And we make some special limitations to LOD controlling parameters and the point size. Consequently, when viewpoint is very close to the object, we use only 20 percent of points to render Bunny. Pay attention to the top edge of Bunny’s right ear. Obviously, we can see that using the same percentage of points, Ψ¯ gives better representation for edge characteristics and parts with big changes in curvature. Through repeated verifications, we find that when α = 0.2, we get a better visual effect.

898

A. Hao et al.

Fig. 3. Relationship between the normal threshold and the rendering frame rate

Fig. 4. (a) α = 1, vertex distribution of 20% points rendered (b) α = 0, vertex distribution of 20% points rendered

4.3

Rendering Effect of HPPO

Fig.5 (a)-(d) shows the continuous LOD rendering effect of Happy, which consists of 1080,000 triangles. In our experiment, Happy is put into a synthetic scene and viewed from different positions, so the rendering is at different levels of details. Fig.5(d) shows the real-time rendering effect that 10 Happy are added to a synthetic scene and located in the view field. On a PC with the PIV3.0G CPUthe 256M memory and the Nvidia7800 display cardour real-time rendering frame rate is 20 fps. In all, from Fig.5 we can see that when HPPO gets farther, fewer points are needed for rendering. The image quality is acceptable and rendering speed is apparently increased.


899

Fig. 5. (a)Happy is rendered using triangles and the current frame rate is 29.94 fps (b) Happy is rendered using 50.86% of points and the frame rate is 43.83 fps (c)Happy is rendered with 20% of points and the frame rate is 62.88 fps (d)Rendering of multiHPPO with tens of millions of triangles and the frame rate is 21.3 fps

5

Conclusion

In order to render large-scale, dense geometric models effectively, this paper introduces a hybrid point and polygon rendering method that integrates the advantages of both point-based rendering and polygon-based rendering. Region segmentation on the model surface and vertex data linearization are performed at the pre-processing phase. And the real-time rendering is implemented on GPU to improve efficiency. For example, when rendering a model consisted of 30,0000 point clouds on a PC with PIV2.4G CPU, 1GRAM, NV5900 graphics card, sequential point tree method put forward by Miguelsainz [15] in 2004 , achievces a frame rate of about 30 fps. At the same time , when rendering a model consisted of 50,0000 point clouds on a PC with PIV3.0G CPU, 256MRAM, NV6200 graphics card, our HPPO method achieves a frame rate of 51.6 fps. So our rendering speed nearly doubles that of the sequential point tree. There are two reasons: firstly, data organization of HPPO is more effective; secondly, the implementation of HPPO method on GPU only uses Vertex Shader. In all, HPPO uses both points and triangles as rendering primitives, with the pointbased rendering used to render middle or distant objects. In this way, there’s no need to use Pixel Shader to fill holes that may appear in pure point-based rendering. Thus, our algorithm gets good efficiency.

900

A. Hao et al.

Even though the point-based rendering in our algorithm has some damages in image quality, it can satisfy general simulation applications. In future work, we will explore ways to improve image quality and further enhance realistic features.

References 1. Scopigno R. Puppo E.: Simplification LOD and Multiresolution - Principles and Applications. In Eurographics’97 Tutorial Notes PS97 TN4, (1997)31-42 2. Levoy M. and Hanrahan P.:Light Field Rendering. In Computer Graphics, Proceedings of SIGGRAPH 1996,(1996)31-42 3. Gobbetti Enrico, Fabio Marton.Far Voxels: A Multiresolution Framework for Interactive Rendering of Huge Complex 3D Models on Commodity Graphics Platforms ACM Trans. Graph. 2005,24(3): 878-885 4. Chen B.and Nguyen M.X. POP: A Hybrid Point and Polygon Rendering System for Large Data. In Proceedings of IEEE Visualization, (2001)45-52 5. Cohen J.D., Aliaga D.G., Zhang W.: Hybrid Simplification: Combining Multiresolution Polygon and Point Rendering. Proceedings of IEEE Visualization 2001.(2001)37-44 6. Oliveira M., Bishop G.: Image-Based Objects. Proceedings of 1999 ACM Symposium on Interactive 3D Graphics. Atlanta, Ga, April 6-28, (1999)191-198 7. Zhirkov A.: Binary Volumetric Octree Representation for Image Based Rendering. Proc.of GRAPHICON’01,(2001) 8. Rusinkiewicz S. and Levoy M. Qsplat: A Multiresolution Point Rendering System for Large Meshes. In Computer Graphics, Proceedings of SIGGRAPH 2000,(2000)343-352 9. Wimmer Michael, Wonka Peter, Sillion Francois X.: Point-Based Impostors for Real-Time Visualization. Proceedings of the 12th Eurographics Workshop on Rendering Techniques. (2001)163-176 10. Coconu L., Hege H.C.: Hardware-Oriented Point-Based Rendering of Complex Scenes. In Proceedings Eurographics Workshop on Rendering. (2002) 43-52 11. Guthe M., Borodin P., et al.: Real-Time Appearance Preserving Out-of-Core Rendering with Shadows. In Proceedings of Eurographics Symposium on Rendering (2004)69-79 12. Junfeng Ji, Sheng Li, Xuehui Liu, Enhua Wu.: Parameter Space Based Multiresolution Structure for Hybrid Rendering Software Jurnal.(2004) 15(10): 1515-1521 13. Yanci ZhangEnhua Wu.: Research On Image Based Real-time Wandering .Doctor paper of Institute of Software, The Chinese Academy of Sciences,2003.6 14. Hoppe H.: Progressive Meshes.Proc.SIGGRAPH 1996,Perth,Australia.(1996)99108 15. Sainz M., Pajarola R. and Lario R.: Points Reloaded: Point-Based Rendering Revisited. In Proceedings of the EG Symposium on Point-Based Graphics, pp. (2004)121-128

User Mediated Hypermedia Presentation Generation on the Semantic Web Framework Jayan C Kurian, Payam M. Barnaghi, and Michael Ian Hartley School of Computer Science and Information Technology, University of Nottingham (Malaysia Campus), Jalan Broga, 43500 Semenyih, Selangor Darul Ehsan, Malaysia {eyx5jkc, payam.barnaghi, michael.hartley}@nottingham.edu.my

Abstract. The art of authoring digital multimedia involves collecting and organizing different sorts of media items and transforming them into a coherent presentation. Existing authoring tools for multimedia presentations provide functional support for the authoring process that requires domain knowledge or presentation skills. The authoring process can be enhanced if the authors are supported with decision making, material collection, selection, and presentation composition in generating web presentations. We apply the semantic web technology to generate hypermedia presentations based on media resources retrieved from the web. A discourse model represents the discussion of a subject with a theme supported by a discourse structure that represents the arrangement of contents in a discourse. In this paper we define a discourse model (i.e. Neural Network Architecture) that specifies the knowledge of composing various discourse entities (e.g. Feed-Forward Neural Networks) which enables the building of discourse structures for various themes (e.g. Lecture Notes). Keywords: Multimedia Applications, Hypermedia Presentation Generation, Semantic Web.

1 Introduction Multimedia authoring is the process of creating multimedia applications. The main phases in the authoring process are the data selection phase (i.e. searching the binary media objects), the presentation structuring phase (i.e. organizing the collected data in sequence), and the presentation generation phase (i.e. constructing the presentation based on the organized data). For existing authoring systems, authors must have knowledge about what contents they want to author and how they want to author these contents. Three general classes of support systems exists: systems that support strict manual presentation authoring (e.g. Microsoft PowerPoint) [1], systems that support mainly automatic presentation generation with limited user influence (e.g. Topia) [2], and semi-automatic authoring environments (e.g. SampLe) [3] that provide semantic-based support during at least one of the authoring phases. With respect to content collection and meaningful integration of content into a discourse structure the manually crafted authoring tools Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 901 – 907, 2006. © Springer-Verlag Berlin Heidelberg 2006

902

J.C. Kurian, P.M. Barnaghi, and M.I. Hartley

do not offer any support. In automatic authoring approach, the author involvement is only at the beginning of the authoring process. The approach towards semi-automatic authoring will support the different types of authors in getting through the presentation generation process starting from the initial exploration of the domain to the final presentation. The semantic web, which is an extension to the current web, enables the access and retrieval of machine-processable web contents and can support information interoperability at various phases in an authoring process. In a semantic environment, Resource Description Framework (RDF) [4] models the knowledge representation structure. Relations between conceptual structures are explicitly defined by an ontology that facilitates knowledge sharing and reuse in a discourse domain. To make the content of the multimedia objects accessible, a data model with rich metadata structure that can annotate the diversity of the media, the subject matter, and the context are required. In this paper we investigate semantic based semi-automatic authoring methods to support the authors during the presentation generation process. The remainder of this ongoing research paper is structured as follows. The next section describes the related work. Section 3 describes the knowledge representation and the data model. Section 4 describes the system architecture and section 5 concludes the paper.

2 Related Work In this section we investigate and describe a number of authoring systems that use semantic web technology as means for generating presentation contents. Topia [2] uses RDF multimedia repository of the Rijksmuseum collection [5] and creates hypermedia presentations as a result of a query. In Topia, clustering is done on pre-selected concepts in the domain semantics and the users can indicate their preferences of the topic using a web interface. This supports the users with useful information while exploring a repository. Topia uses a domain specific clustering mechanism for generating the presentations. The SemInf [6] system semi-automatically generates multimedia presentations by combining semantic inferencing with multimedia presentation tools. The system applies an iterative sequence of searches across the Dublin Core (DC) [7] metadata to infer semantic relationships between mixed media objects which are retrieved from a repository. These semantic relationships are mapped to spatial and temporal relationships between the objects using predefined mapping rules and are expressed within SMIL [8] format to generate multimedia presentations. The Artequakt [9] project applies semantic associations between different entities which are representing an artist’s personal and professional life to generate biographies. The domain ontology is used to provide a common understanding on concepts in the discourse domain. The web resources are associated to these ontology concepts. The inferring process refers to these concepts and attempts to realize implicit relationships between these resources. Interpreting the relationships enables

User Mediated Hypermedia Presentation

903

the system to create a coherent presentation structure that demonstrates an artist’s biography using objects retrieved from heterogeneous resources. The domain ontology and the annotated multimedia repository of the Rijksmuseum are used by DISC [10] to create multimedia presentations. The aim of the system is to build a multimedia presentation about a certain topic by traversing a semantic graph. A semantic graph consists of domain ontology of classes, instances and relations between them together with the media material related to the instances. The semantic graph is transformed into structured progression and this data structure supported by the domain, discourse, and design knowledge is used for generating the multimedia presentation. A hypermedia presentation generation system in a multi-facet environment is described in SampLe [3]. The system employs semantic web technologies and supports authors during a hypermedia presentation generation process. The process is divided in to four phases: topic identification, discourse structure building, media material collection, and production of the final-form presentation. SampLe supports authors during every phase of the process independent of the particular workflow using ontology-based and context oriented information as well as semantic interrelationships between different types of meta-data.

3 Knowledge Representation and Data Model For knowledge representation, we employ an ontology that defines a common vocabulary for machine-interpretable definitions of common concepts in the domain and relations among them. The ontology gives information related to media objects (e.g. media type), content of the image (e.g. subject) and semantic relationship between media objects (e.g. relation identity). Protégé [11], an ontology editor tool is used to develop the knowledge-base for the domain ontology. This is represented in RDF(S) [12]. A simplified version of the domain ontology for a Neural Network instance domain is illustrated in Fig. 1. A data model represents the basic guidelines for annotating the media items for data selection. The data model adopted from [13] has the components content schema, semantic schema, and media schema that describes the multimedia objects. The content schema is represented by Dublin Core attributes (e.g. title, identifier), the semantic schema is represented by Learning Objective Metadata (LOM) [14] attributes (e.g. language, level), and the media schema is represented by MPEG-7 [15] attributes (e.g. media type, media URI). The ontology and the data model make the knowledge representation structure independent from the data representation structure. MPEG-7 Multimedia Description Schemes can effectively describe the multimedia entities. MPEG-7 Visual Description Tools describes the visual features (e.g. color, motion). MPEG-7 Audio provides the standard for describing the audio contents (e.g. sound recognition). MPEG-7 standard is chosen since Dublin Core is not specifically designed for representing spatial and temporal aspects of multimedia data. The above data model has been adopted since the schema components can effectively represent the media objects used in generating the presentation.

904


Fig. 1. A simplified version of Neural Network domain ontology

4 The System Architecture In the proposed system the domain ontology is represented in RDF/XML and the media objects are stored in a relational database or are coming from heterogeneous resources. The Jena [16] application API will be used to query the RDF/XML knowledge-base. Using the author interaction interface the author can browse the domain ontology and select a particular concept. The methodology involves Selection, Assembling, Transforming and Presenting. In the selection phase, the author selects the theme of the presentation (e.g. Lecture Notes). Next the title (e.g. Neural Network Architecture) of the presentation is selected and the contents are assembled in the presentation. In the media addition phase, the relevant media items are added to the presentation. The presentation style (e.g. style sheets) is selected in the next phase. In the transformation phase of this ongoing research, the hypermedia presentation will be generated in the form of SMIL. The system architecture is illustrated in Fig. 2. The user interface of the proposed system shows the summarized information of all the selections the author has made (author activities) in creating the presentation.


905

Fig. 2. Logical diagram of the system architecture

This includes the duration or size of the presentation as well as which phase the user is currently active and with which task within that phase. The presentation will be generated in the form of a time-based mode (slide-show) or static modes (interactive and thumbnail views) based on the user choice. Successful hypermedia presentations will be stored in the system repository based on the concept searched so that these built-in presentations can be suggested when a new user browses for a related concept. These suggestions can be done by using the a priori algorithm [17] for generating association rules. On a generated presentation, the user can click on a particular media object of his/her interest to trigger a new presentation of related concepts. This process is called the tangent [2] and will be done by dynamically grouping media objects using agglomerative clustering algorithm [18]. The algorithm provides groups depending on the level in the hierarchy and provides structures for simple interactive selection and rapid non-linear expansion of a selection. In this way, new presentations, focusing on a related concept can be generated dynamically.

5 Conclusion This paper describes an ongoing research to develop a semi-automatic interaction of author within the different stages of hypermedia presentation generation. A context aware guidance for authors is provided in the authoring process by defining a

906


discourse model that specifies the knowledge of composing various discourse entities which enables the building of discourse structures for various themes. The domain ontology for Neural Network is designed for representing the domain concepts and relations. Dublin Core, MPEG – 7 and IEEE LOM are used for representing the metadata structure in a data model. Protégé is used to develop the knowledge base and shared web resources are described using RDF. Jena, a Java API is used to query the RDF/XML knowledge-base. Future work will concentrate on the automatic classification of new annotated media items to different user levels (e.g. novice, beginner, and expert) based on the Bayesian model [19] and how to make inferences for calculating the system's determination on a user's knowledge for generating customized presentations.

References 1. Microsoft PowerPoint, the Microsoft Office presentation program, available at: 2. Rutledge, L., Alberink, M., Brussee, R., Pokraev, S., Van Dieten, W., Veenstra, M.: Finding the Story —Broader Applicability of Semantics and Discourse for Hypermedia Generation, In Proceedings of the 14th ACM Conference on Hypertext and Hypermedia, (2003) 67–76 3. Falkovych, K., Bocconi, S.: Creating a Semantic-based Discourse Model for Hypermedia Presentations: (Un)discovered Problems, In Workshop on Narrative, Musical, Cinematic and Gaming Hyperstructure, 2005 4. The Resource Description Framework (RDF), available at: 5. Rijksmuseum Amsterdam, a museum of Dutch art and history, available at: 6. Little, S., Geurts, J., Hunter, J.: Dynamic Generation of Intelligent Multimedia Presentations through Semantic Inferencing, In 6th European Conference on Research and Advanced Technology for Digital Libraries, (2002) 158–189 7. Dublin Core Metadata Element Set Version1.1 (DC), Reference Description, Dublin Core Metadata Initiative (1999), available at: 8. Synchronized Multimedia Integration Language (SMIL), available at: 9. Kim, S., Alani, H., Hall, W., Lewis, P.H., Millard, D.E., Shadbolt, N., Weal, M.J.: Artequakt: Generating Tailored Biographies with Automatically Annotated Fragments from the Web, Workshop on Semantic Authoring, Annotation & Knowledge Markup, 15 European Conf. on Artificial Intelligence (ECAI), (2002) 1-6 10. Geurts, J., Bocconi, S., van Ossenbruggen, J., Hardman, L.: Towards Ontology-driven Discourse: From Semantic Graphs to Multimedia Presentations, In Second International Semantic Web Conference, (2003) 597–612 11. Protégé, an ontology editor and knowledge-base framework, available at: 12. Resource Description Framework (RDF) Schema, available at: 13. Barnaghi, P.M., Kareem, S.A.: Ontology-Based Multimedia Presentation Generation, IEEE TENCON Conference, (2005)


907

14. Learning Object Metadata (LOM), available at: 15. Manjunath, B.S., Salembier, P., Sikora, T.: Introduction to MPEG-7: Multimedia Content Description Interface. John Wiley (2002) 16. Jena, a semantic web framework for Java, available at: 17. Larose, T.D.: Discovering Knowledge in Data: An Introduction to Data Mining. WileyInterscience (2004) 18. Chiu, P., Wilcox, L.: A Dynamic Grouping Technique for Ink and Audio Notes, In the Proc. ACM UIST (1998) 195-202 19. Han, J., Kamber, M.: Data Mining: Concepts and Techniques. 1st edn. Morgan Kaufmann (2000)

Progressive Transmission of Vector Map Data Based on Polygonal Chain Simplification Haisheng Zhan and Guangxin Li School of Network Education, Xidian University, Xi’an 710071, China {Zhan_haisheng, Li_guangxin}@vip.163.com

Abstract. In distributed Geographic Information Systems (GIS) and Web-GIS, remote access to geospatial database is recognized as a time-consuming task. Progressive transmission, a technology dealing with massive amounts of data and slow communication links, can be used to solve this kind of problem because it allows users to start work with a partially delivered dataset. In this paper, we particularly discussed this problem and propose a method, by which we can progressively transmit the sequence of levels of detail of map representation while preserving topological consistency between different levels. We employ the algorithm of minimum number of safe sets to simplify the polyline. In addition, we propose a structure based on Simplicial MultiComplex and Reactive-tree to avoid data redundancy as well as reduce the space complexity of geospatial database. We show that the volume of coarsest level data can generally be reduced to one tenth of that of the finest level. Keywords: progressive transmission, vector map, polyline simplification, Simplicial Multi-Complex, geospatial database.

1 Introduction Following communication technology and Internet development, distributed GIS and Web-GIS arise. In these systems, large geospatial datasets reside in some computers called server. Users can make remote access to these datasets with a computer called client through communication links. It also generates new problems such as the availability of vast amounts of data stored in huge unstructured that need to be downloaded. Therefore, users need better tools to search and download the datasets they required, be it through improved spatial metadata or spatial summaries. Metadata in the form of textual summaries is only used to convey the idea of what is contained in the database but to manipulate the datasets. For example, during remote access, overview of the fully detailed dataset may be used to perform some preliminary manipulations or analysis. For this purpose, spatial summaries can be generated by means of subsetting techniques in which a meaningful sample of the data is provided instead of the whole dataset in [1]. Progressive transmission of raster images over the WWW has been successfully applied to provide the user with temporary versions of the data before downloading a complete image. While implementations have focused on progressive transmission of Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 908 – 917, 2006. © Springer-Verlag Berlin Heidelberg 2006

Progressive Transmission of Vector Map Data

909

raster data through the Web, the transmission of vector data, with the exception of the particular case of data in form of triangular meshes, is generally done by means of a one-step long process. The user attempting to download a vector file needs to wait for the complete version without having the possibility to start working with a coarser version of the data in a smaller file in [2], [3].

2 Background and Requirement of Progressive Transmission In this section, some essential preliminaries, include vector map representation, polygonal chain simplification, and SMC, are declared first. 2.1 Polygonal Chain Simplification Let S be a subdivision that models a map. A subdivision is a geometric structure that represents an embedded planar graph. We adopt the terminology standard in computational geometry and say that the subdivision S consists of vertices, edges and faces. The degree of a vertex is the number of edges incident to it. A vertex of degree one is a leaf, a vertex of degree two is an interior vertex, and a vertex of degree at least three is a junction. Generally the number of leafs and junctions is small compared to the number of interior vertices. Any sequence of vertices and edges starting and ending at a leaf or junction, and with only interior vertices in between, is called a polygonal chain, or simply a chain. For convenience we also consider a cycle of interior vertices (which occur for islands) as a chain, where we choose one of the vertices as the junction. It is the start and the end vertex of the chain.

Fig. 1. Incompatible topological relationships accruing in polygonal chain simplification

Let P be a set of points that model special positions inside the regions of the map. Subdivision simplification can now be performed as follows. Keep the positions of all leaf and junctions fixed, and also the positions of the points in P. Replace every chain between a start and end vertex by a new chain with the same start and end vertex but with fewer interior vertices. For a polygonal chain C={v1, v2, …, vm} of size size(C)=m, we require of its simplification C’ = {v1= vj1, vj2, … , vjm = vm }, which is a chain whose vertices are a subsequence of the vertices of C: (1) No point on the chain C has distance more than a specified error tolerance to its simplification C’. (2) The simplification C’ is a chain with no self-intersections. See Fig.1(a).

910

H. Zhan and G. Li

(3) The simplification C’ may not intersect other chains of the subdivision. See Fig.1(b). (4) Each point of P lies to the same side of C’ as of C. See Fig.1(c). In the simplification C’, a line segment vjkvjk+1 is a shortcut for the subchain {vi, vi+1, … , vj} (i = jk, j = jk+1) in chain C, The error of a shortcut vjkvjk+1 is the maximum distance from vjkvjk+1 to a point vk, where i ≤ k ≤ j . A shortcut is allowed if and only if the error it induces is at most some prespecified positive real value ε . We wish to replace C by a chain consisting of allowed shortcuts. This chain should have as few shortcuts as possible. In this paper we don't consider simplifications that use vertices other than those of the input chain. 2.2 Simplicial Multi-complex The concept of simplicial complex derives from algebraic topology. An abstract simplicial complex K consists of a set of vertices {1, …, m} together with a set of non-empty subsets of the vertices, called the simplices of K, such that any set consisting of exactly one vertex is a simplex in K, and every non-empty subset of a simplex in K is also a simplex in K. In geometry, an abstract simplicial complex K is a combinational structure specifying the adjacency of vertices, edges, triangles, and tetrahedron. Definition 1. A k-dimensional cell (or k-cell, for brevity) in IEd is a subset of IEd homeomorphism to a closed k-dimensional ball, k≤d. Let Γ be a connected finite set of n-cells of heterogeneous dimension embedded in the Euclidean space IEd, where n is the maximum of the dimensions of the cells of Γ, and the boundary of each cell in Γ is a collection of cells of lower dimensions, called facets, belonging to Γ. Then, Γ is an n-dimensional mesh if and only if it satisfies the following properties: (1) The interiors of any pair of n-dimensional cells of Γ are disjoint; (2) Any k-dimensional cell of Γ, with k < n, bounds at least one n-dimensional cell of Γ. 

Fig. 2. A two-dimensional mesh (left) and two sets of cells that are not meshes: (central) violates condition 1 and (right) violates condition 2 in the definition of a mesh

Fig.2 shows an example of a two-dimensional mesh, and two connected sets of cells in IE2 that are not meshes in [4]. Definition 2. A simplicial multi-complex (or k-simplex, for brevity) in IEd is the locus of the points in IEd that can be expressed as the convex combination of k+1 affinely independent points. 


911

Definition 3. A mesh is called conforming if and only if, for each pair of ndimensional cells Υ1, Υ2, the intersection of the boundaries of Υ1 and Υ2 is either empty, or it consists of a k-facet belonging to the boundary of both Υ1 and Υ2, for some k < n.  Definition 4. A modification is the basic operation for changing a mesh locally. For simplicity, we define modifications in the case of meshes without boundary. We define an n-dimensional modification as a pair of n-dimensional meshes M = (Γ1, Γ2), such that the boundaries of Γ1 and Γ2 are coincident. The intuitive idea is that modification M can be applied to a mesh Γ, such that Γ1 ⊆ Γ , and its application modifies Γ by replacing Γ1 with Γ2. The condition on the boundaries ensures that Γ2 fits the hole left in Γ by the removal of Γ1.  Definition 5. A modification M = (Γ1, Γ2) is conforming if meshes Γ1 and Γ2 are conforming meshes, and Γ1 and Γ2 have the same combinatorial boundary. Examples of a conforming and of a non-conforming modification are shown in Fig. 3. The result of applying a conforming modification to a conforming mesh (if it gives a mesh) is a conforming mesh.  M1

M2

Fig. 3. Modification M1 is not conforming, while M2 is conforming. (left) shows a meshΓ, (central) the results of applying M1 , and (right) the results of applying M2.

Definition 6. A modification M = (Γ1, Γ2) is said to be non-redundant if no n-cell belongs to both Γ1 and Γ2, i.e., M does not delete and re-create the same cell. From now on, we will implicitly assume that all modifications we consider are non-redundant.  A modification can be reversed. The inverse of a modification M = (Γ1, Γ2) is M-1 = (Γ2, Γ1). We say that a modification M = (Γ1, Γ2) is a refinement modification if #Γ2 > #Γ1. M is a coarsening modification if #Γ2 < #Γ1. M is a neutral modification if #Γ2 = #Γ1. In what follows, we will always assume non-neutral modification, and we will use the alternative notation M- and M+ to denote the two meshes forming a modification M, according to the intuition that M- has fewer n-cells than M+. As described in subsection 2.2, a polygonal chain modeling the boundaries of subdivisions is the most important geometric entity in planar map. A chain is a collection of line segments which are 1-simplex while a line segment is bounded by

912

H. Zhan and G. Li

0-simplex (vertex). So a planar map can be treated as a very big collection of 1-simplexes and 0-simplexes. To manipulate the collection and modification applied to it, we design a correspondent data structure and algorithms in Section 5.

3 Related Work We briefly survey previous work in the domain of progressive transmission of vector map as well as of map simplification and multi-display of geometric objects related to it. A straightforward way to transferring vector map over WWW progressively is storing multiple map representation from coarse level to fine level in the server. Because some entities are preserved throughout the levels they should be stored once in the sequence instead of redundantly encoding them at each level. Bertolotto and Egenhofer [2], [4] proposed a strategy for progressive vector transmission in which a model generating multiple map representation [5], [6] is used for progressive transmission. The model is built by applying a set of generalization operators that perform topological changes on vector maps. In their method polyline simplification is not employed because it is time-consuming and may cause topological error as we describe later. But polyline simplification can reduce the number of vertices of map object greatly while preserving visual similarity (see Fig. 4). We introduce polygonal chain simplification to reduce the volume of the geospatial database in our method. To encoding levels of detail efficiently, Reactive-tree [7] and SMC [8] based data structure are both used.

Fig. 4. Polylines with 378 vertices (left), 186 vertices (central) and with 95 vertices (right)

There are many algorithms for polyline simplification in the literature of GIS, Computer Vision, and Computational Geometry. The practical literature tends toward heuristic that run in linear or near linear time, while the Computational Geometry literature tends toward optimal simplification under various criteria (e.g. minimum error for a given number of segments, or minimum number of segments for a given error) using algorithms that run in quadratic to cubic time. Most algorithms consider each linear feature in isolation, and run risk of introducing intersections and topological changes in data. The cartographer’s favorite algorithm, reported in Douglas and Peucker [9], can even introduce self-intersections within a single line. There has been some work in computational geometry on algorithms that guarantee to preserve map topology. De Berg et al. [10] give a simplification method that simplifies a polyline without passing over specified points in near quadratic time. Recent work [11] has shown that the map simplification is NP-hard. Moreover, Estkowski [12] has also shown that it is hard to obtain a solution to this problem that


913

approximates the optimal answer to within a polynomial factor. Mustafa et al. [13] propose a Voronoi diagram based algorithm with the goals of avoiding intersections. However, none of the methods for chain simplification guarantees that the output chain will not intersect itself. A. Mantler et al. [14] also propose a Voronoi diagram based algorithm by which user can identify safe sets in a linear time. A safe set is an ordered set of consecutive points defining a fragment of a single polyline that is monotone in some direction and whose convex hull contains no other points. A complex polyline can be divided into a collection of safe sets in which each safe set can be simplified using either a single line or a standard polyline simplification algorithm such as [9], without introducing intersections or topological changes in the collection of safe sets. We generalize Mantler’s algorithm to calculate the minimum number of safe sets to obtain base polyline as described in the next section.

4 Map Representations for Progressive Transmission Popovic and Hoppe [15] first detailed the progressive transmission of triangulated model represented with simplicial complex. Floriani and Magillo [8] generalize this method to multiresolution representation of general vector data such as tetrahedron. In this paper, we apply the concept of simplicial complex to representation of vector map datasets, and show that it is appropriative for progressive transmission of this type of datasets. In simplification of a polygonal chain, a sub-chain is replaced by a new chain with the same start and end vertex but with fewer interior vertices. This procedure can be treated as a modification on original 1-dimensional mesh. We can reverse the modification and use it to generate original polygonal chain from its simplification. The modification is conforming and non-redundant. A polygonal chain can be represented by its simplification with few vertices as possible, namely base polyline, and a sequence of refinement modification {M1,M2,…,Mh}, such that, for each 1-simplex γ ∈ M i− , ,exactly one of the following two conditions holds: either γ ∈ Γ 0 , or γ ∈ M i+ for exactly one i ≠ j . Intuitively, the previous condition means that each line segment must either appear in the base polyline Γ 0 , or be created by some modification. i = 1,..., h

Definition 7. A modification Mj directly depends on a modification Mi if Mj removes some line segments inserted by Mi. More formally, Mj depended Mi if there exists some line segments which belongs to both M −j and M +j .  If the transitive closure ≺ of the direct dependency relation is a partial order, then we say that the triplet Μ = ( Γ 0 , {M 1 , M h }, ≺) is a multiresolution polygonal chain. We call Γ0 the base polyline, since it is the polygonal chain with the fewest vertices, and modifications in set {M1, M2, …, Mh} tend to refine it. The collection of all cells γ contained in Γ 0 ∪ {M i+ | i = 1, h} and not contained in ∪{M i− | i = 1, h} exactly is original polygonal chain, and its cells are not further refined by any other modification. The Hesse diagram of the partial order defining a multiresolution mesh is a Directed Acyclic Graph (DAG) in which the nodes describe modifications, and the arcs describe links of direct dependency between modifications. By convention, arcs

914

H. Zhan and G. Li

are directed from Mi to Mj if Mj directly depends on Mi. This choice reflects the intuitive idea that Mi should be performed before Mj. Consider starting from the base polyline Γ0 and applying all modifications of set {M1, M2, …, Mh} to it in a sequence corresponding to a total order extending ≺ . Regardless of the specific order, all such sequences produce the original polygonal chain. The intermediate collections of cells generated in the process correspond to representations at intermediate LODs, and they depend on the specific sequence used. All possible sorted sequences extending ≺ give all possible intermediate LODs. We can truncate the refinement process described above by selecting a proper subset of the modifications in {M1, M2, …, Mh}, which is closed with respect to the partial order. A subset S of the modifications in a multiresolution polygonal chain is called closed if, for each modifications M i ∈ S , all modifications Mj, such that Mi depends on Mj, are in S. A closed subset of modifications can be applied to the base polyline Γ0 in any total order extending ≺ . Each closed subset S provides a set of cells ΓS, which is the result of applying its modifications to Γ0. If ΓS defines a chain, then we call it an extracted chain. Data structures for multiresolution chains must represent, either explicitly or implicitly, the modifications and the partial order. The encoding of the base polyline Γ0 is not treated here because we assume that Γ0 is small, and that it can be encoded through standard data structures for polygonal chain such as array. In what follows, we will demonstrate the details about how to obtain the base polyline Γ0, encoding modification, and encoding dependencies. (1) In order to evaluate the space complexity of the data structures and computational cost of the query primitives operating on them, we consider following parameters for a multiresolution chain Μ = ( Γ 0 ,{M 1 , M h }, ≺) : (2) the number h of modifications; (3) the total number c of line segments appearing in all modifications, i.e., cardinality of the set of 1-simplexes ∪ hi=1 ( M i− ∪ M i+ ) ; (4) the number a of links of direct dependency among modifications (i.e., number of arcs in the DAG or tree).

the the

the the

The main task of this step is to find the simplification with the fewest vertices of original polygonal chain while preserving topological consistency, described in subsection 2.2, between them. We generalize the Safe-Sets algorithm to compute the minimum number of safe set of polygonal chain, i.e., the algorithm of min-SSs. In algorithm of Safe-Sets, Voronoi diagram of all input points and its dual Delaunay triangulation are used to identify a collection of safe sets. If a Delaunay edge ed connects two consecutive points vi and vi+1 on the polyline, then the Voronoi edge ev dual to ed is the window from vi to vi+1. If there is no Delaunay edge connecting vi and vi+1, then there is no window. Each window is oriented. You can only look through it from vi to vi+1. See figure 5. A cone is a region of the plane bounded by two rays anchored at a point. The outer cone from vi to vj is defined as the largest cone anchored at vi that will pass through all the windows between vi and vj. The inner cone from vi to vj is defined as the smallest cone anchored at vi that contains all of the points from vi to vj. See figure 5.


915

The first step of this algorithm is to compute the Voronoi diagram of points. Once we have got the Voronoi diagram, we start at one end of some polyline and add consecutive points until the inner cone is no longer contained with the outer cone. This can be done in linear time. So we get the first safe set of original polyline. By same step we can get all safe sets. Any simplification of the points contained in each safe set produces polyline that are homotopic to the input polylines in the region of the plane minus all input points [14].

Fig. 5. Definition of window (left), Inner and outer cone(right)

Subsets of safe sets are also safe sets because the convex hull of a safe set constructed by the algorithm will not contain any points that are not in the safe set. We can find the minimum number of safe sets by a greedy approach after we obtain all safe sets of a polygonal chain. We start at one end of first safe set and add consecutive points while the set is monotone and there are no extra in the convex hull. For computing the minimum number of safe set efficiently, we incorporate this process into the procedure of computing safe sets. The whole procedure is shown as follows: (1) Compute the Voronoi diagram of input points, including vertexes of all polygonal chain in a map and points that model special positions inside the regions of the map; (2) Select one end of some polygonal chain and add consecutive points until the inner cone is no longer contained with the outer cone; (3) Add consecutive points while the set is monotone and there are no extra points in the convex hull; (4) Select next consecutive points, repeat step (2) and step (3) until reach the other end of the polygonal chain; (5) Select one end of another polygonal chain, repeat step (2) to step (4) until have worked with the last chain in the map. A complex polyline is divided into a collection of safe sets in which each safe set can be simplified using a single line. In principle all the single lines in a collection compose the base polyline, but for the purpose of visual consistency, a standard polyline simplification algorithm such as Douglas-Peucker is further used to compute the base polyline within an error ε . The refinement modification of the base polyline can be computed by the standard polyline simplification algorithm with an error ε’, where ε’ 1 − ε w(y) = 0, othterwise ⎩ 1 ε−y , y < ε .

(7)

(8)

944

2.4

L. Ren et al.

3D Location Modifying

In this section, we will first introduce the mapping between 2D points and 3D points, then combine this mapping and the 2D mapping in Section 3.3 to modify all points in the generic model. Points in 2D and those in 3D can be related to each other by projection matrix, model-view matrix and viewport matrix [13]. T Let R be a rotation matrix and t = [tx ty tz ] a translation vector, the mapping will be [14]: ⎡ ⎤ ⎡ ⎤ ⎡ ⎤⎡ ⎤ X x f 0 0 1 0 0 0 T T ⎢ R −R Y ⎥ t ⎢ ⎥ . ⎣y ⎦ = ⎣ 0 f 0⎦⎣0 1 0 0⎦ (9) T ⎣ 03 1 Z⎦ 1 0 0 1 0 0 1 0 1 Both of these matrices can be computed from the extracted camera setting. Furthermore there are two utility function called gluProject and gluUnproject to help us do this task. Every 3D point (X, Y, Z) in the generic model is projected to a 2D point (x, y) with a depth value z. The 2D mapping algorithm will then find the corresponding 2D point (x , y ) in the captured image. Combining this new 2D location with the depth value got before, forms a triad, which can be inversely projected to a 3D point. After changing the position of all 3D point, the model will exhibit similar shape to that in the captured image. As in Hilton’s method, our method also employs the depth value in the generic model as an approximation of that in the new model. But since the distance between the camera and person, about three meters, is large relative to the difference of corresponding points in the two models, this is a reasonable approximation. Modification based on pictures of a single view only changes the shape orthogonal to the viewing direction and is incomplete. Full adjustment is the combination of the modifications obtained from two views. Repeating this procedure for every point in the generic model will give us the new model.

3


We implemented the above algorithm in a PC with 512M memory and 1.2G CPU. Low level algorithms are implemented in C whereas the GUI part is written in Python. The generic model is captured by 3D whole body scanners. We filled the holes in it with Blender4 , and cut it into segments with a script. When capturing reference pictures, we fixed the distance from the camera to the center of the person being photographed at 3m. As head, feet, and hands were too small in the pictures, there was not enough information to apply the above algorithm to it. They were simply scaled and translated. The results is shown in Fig. 4. Besides the obvious resemblance of height and weight between the result model and the picture, local features such as delicate contour details in arms and legs 4

http://blender.org/

Personalized Model Deformation Based on Slice Space Representation

(a)

(d)

(b)

(e)

945

(c)

(f)

Fig. 4. Results. (a) generic model. (b), (c), and (d) result model from front, side, and new view. (e) and (f) silhouettes extracted from front and side pictures.

are also captured. Neither seams nor overlaps exist. And because of the structure inherited from the generic model, the result is ready to be animated. Our method does not require the strict topological restriction in [8]. Neither does it require the pose in the picture to be similar to that in the generic model, which helps the potentially wide use of our method.

4


This paper presents an improved method for building articulated human body, which deforms the generic model with features extracted in individual pictures. Our contribution is the novel slice space representation conforming to the natural feature of human body, which captures more details in individual pictures and preserves the connectivity of surface in more cases.

946

L. Ren et al.

Although it gives satisfactory result in most case, sometimes it cannot handle the shoulder joint very well. Solving this problem completely will require incorporation of other specialized deformation techniques such as pose space deformation [15] or difference coordinates [16], which is our future work. Integrating our modeling method with path-planning techniques [17,18,19] generate realistic human animation of individuals in pictures is also a promising direction.

Acknowledgements This project is co-supported by 973 project (grant no: 2002CB312100) and Key NSFC project on Digital Olympic Museum(grant no: 60533080).

References 1. Magnenat-Thalmann, N., Seo, H., Cordier, F.: Automatic modeling of virtual humans and body clothing. Journal of Computer Science and Technology 19 (2004) pp.575–584 2. Devernay, F., Faugeras, O.D.: Computing differential properties of 3-d shapes from stereoscopic images without 3-d models. In: Proc. of Computer Vision and Pattern Recognition. (1994) 208–213 3. Sloan, P.P., Rose, C., Cohen, M.: Shape by example. In: Symposium on Interactive 3D Graphics. (2001) 4. Funkhouser, T., Kazhdan, M., Shilane, P., Min, P., Kiefer, W., Tal, A., Rusinkiewicz, S., Dobkin, D.: Modeling by example. In: ACM SIGGRAPH 2004. (2004) 5. Hilton, A., Gentils, T.: Capturing human models to populate virtual worlds. Technical report, CVSSP, University of Surrey, UK (1998) 6. Hilton, A., Beresford, D., Gentils, T., Smith, R., Sun, W.: Virtual people: Capturing human models to populate virtual worlds. In: IEEE Computer Animation. (1999) 7. Lee, W., Goto, T., Magnenat-Thalmann, N.: Making h-anim bodies. In: Avatars2000. (2000) pp.37–44 8. Lee, W., Gu, J., Magnenat-Thalmann, N.: Generating animatable 3d virtual humans from photographs. In: Computer Graphics Forum. Volume 10 of 3. (2000) pp.1–10 9. Magnenat-Thalmann, N., Laperrière, R., Thalmann, D.: Joint-dependent local deformations fo hand animation and object grasping. In: Proc. Graphics Interface. (1988) 26–33 10. Jianhua, S., Magnenat-Thalmann, N., Thalmann, D.: Human skin deformation from cross sections. In: Proc. Computer Graphics International ’94. (1994) 11. Babski, C., Thalmann, D.: A seamless shape for hanim compliant bodies. In: VRML 99: Fourth Symposium on the Virtual Reality Modeling Language. (1999) 21–28 12. Thalmann, D., Shen, J., Chauvineau, E.: Fast human body deformations for animation and applications. In: Proceedings of Computer Graphics International 1996. (1996) 166–174 13. Neider, J., Davis, T., Woo, M.: OpenGL Programming Guide: The Official Guide to Learning OpenGL. first edn. Addison-Wesley, Reading Mass. (1993)

Personalized Model Deformation Based on Slice Space Representation

947

14. Pollefeys, M.: Visual 3d modeling from images. Tutorial Notes (2002) 15. Lewis, J.P., Cordner, M., , Fong, N.: Pose space deformations: A unified approach to shape interpolation and skeleton-driven deformation. In: ACM SIGGRAPH 2000. (2000) 16. Alexa, M.: Differential coordinates for local mesh morphing and deformation. The Visual Computer 19 (2003) 105–114 17. Xu, W., Pan, Z.: A novel method for generating curved path human walking animation. In: Proceedings of 2nd Internatinoal Conference on Image and Graphics. (2002) 993–998 18. Zheng, L., Zhang, M., Tang, B.: Using motion graph for motion editing in path synthesis. In: 9th International Conference on Computer Graphics and Artificial Intelligence. (2006) 19. Ge, Y., Pan, Z., Xu, W.: Vision-based path planning in intelligent virtual environment. In: 6th International Conference on Computer Graphics and Artificial Intelligence. (2003) 25–34

Flexible Camera Setup for Visual Based Registration on 2D Interaction Surface with Undefined Geometry Using Neural Network Ary Setijadi Prihatmanto1, Michael Haller2, and Roland Wagner1 1

Kepler University of Linz, Institute of FAW, Altenbergerstrasse 69, A-4040 Linz, Austria {asetijadi, rrwagner}@faw.uni-linz.ac.at 2 Upper Austria University of Applied Science and Digital Media, Hagenberg, Austria [email protected]

Abstract. Camera setup, calibration and visual based registration of Augmented Reality (AR) based tabletop setups can be a really complicated and timeintensive task. Homography is often used liberally despite its assumption for planar surfaces, where the mapping from the camera to the table can be expressed by a simple projective homography. However, this approach often fails in curved and non-planar surface setups. In this paper, we propose a technique that approximates the values and reduces the tracking error-values by the usage of a neural network function. The final result gives a uniform representation of the camera against combinations of camera parameters that will help in the multi-camera setup. We present the advantages with demonstration applications, where a laser pointer spot and a light from the lamp will be tracked in non planar surface. Keywords: camera calibration, visual based registration, tabletop application, function approximation, neural network.

1 Introduction Tabletop augmented reality (AR) applications with multiple cameras and projector systems are becoming more and more popular. The current generation of projectors allows a fast combination of multiple projectors that can be used simultaneously. Molyneaux and Kortuem demonstrate in [1], how the future environment can be more accommodative to the ubiquitous display paradigm rather than conventional collection of displays. Bimber and Raskar describe in [2] a novel approach to create an application that has taken augmented reality beyond traditional eye-worn or hand-held displays. They call this technology Spatial Augmented Reality (SAR). Its popularity is also boosted by the fall in cost and availability of projectors, high-resolution cameras, personal computers and 3D graphics cards. Moreover, new generation of projection such as a Dome projector [3] gives new opportunities to create new interaction metaphors. Despite the availability of other tracking technology, it has been shown in the recent years that vision-based tracking systems have a tremendous potential. The most Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 948 – 959, 2006. © Springer-Verlag Berlin Heidelberg 2006

Flexible Camera Setup for Visual Based Registration on 2D Interaction Surface

949

common method for visual-based pose registration is the use of a homography matrix along with quadratic lens distortion model. Despite its success in the past, it suffers from inherent limitations. Hence, a more efficient and flexible solution that gives the possibilities for automation and adaptability is required to support the display ubiquity paradigm and the trend to utilize non-planar surface such as dome projection surface[3]. This paper describes the technique to use a vision-based system for visual based registration on 2D interaction surface with undefined geometry in the tabletop application. Our technique utilizes the fact that tabletop application can control what information that can be perceived by camera to create a more adaptable and easy to configure vision-based sensor. It also utilizes the universal approximation nature of multi-layer perceptron so the system can adapt to as wide as possible geometry of the interaction surface. Furthermore, by maximizing the above utilization, the technique is designed so that the system can be configured without the knowledge of the internal camera parameters. Moreover, the result gives a uniform camera representation despite the various possible camera parameter combination. This feature makes fusion policy among cameras in multi-camera setup can be managed in more tangible manner. Although this paper emphasizes on tracking purposes, it will be shown that the extension to more general purposes is also possible. In the next section, we review the related work on general camera calibration and related work on the application of neural network in connection to camera usage. Next, we describe the proposed framework starting by explaining the general overview of the problem. It is then followed by a description on the proposed method. In section 4, we demonstrate our results. Next, we illustrate various possibilities of extension and finally, we discuss the future work.

2 Related Work Good and efficient camera calibration and visual registration have become more and more important over the last decade. As described by Baratoff et al., the calibration of an AR system can be very complex and has to be done with care [4]. A good calibration is important when we need to reconstruct a world model and interact with the world based on information that comes from a vision system tracked by a camera. Tsai’s techniques can be seen as one of most influential papers in the last decade [5] [6]. The goal of the camera calibration in a conventional sense is to find the internal quantities of the camera that affect the imaging process. In contrast, we do not model the geometry of the camera. Neither do we have to know the camera parameters nor do we have to know the relationship between the camera and the projected image. We only generate an approximation of the function that is triggered by the camera existence according to the application context by using the neural network’s universal approximation feature to coup the non-linear dynamic of the imaging process due to distortions and/or non-planar geometric shape. There are only few reported publication regarding the use of neural networks in camera calibration [7, 8, 9, 10, 11, 12]. These techniques used neural-networks either to learn the mapping from 3D world to 2D image without specifying the camera

950

A.S. Prihatmanto, M. Haller, and R. Wagner

models[12] [7], as an additional stage to improve the performance of other existing techniques [9, 8], or to approximate an imaging process of camera i.e. camera representation. In [11], carefully design neural network structure is developed so each weight has its own physical meaning and different role during the training. The physical meaning of the weight of neural network is correlated to camera parameters value. It has more robust approach than the previous work, but still it gives no considerable advantage over the standard camera calibration procedure. In the next section, we proposed a method to use neural network to create a camera representation that is used to solve visual registration problem. The MLP is used only to deal for what is intended to do which is to approximation a non-linear function. The function is carefully chosen so the universal approximation capability of MLP can be maximaly utilized over all possible data region. The proposed method also addresses issues of scalability, reliability and robustness. This gives considerably advantage over the standard camera setup.

3

Proposed Framework

Fig. 1 (a) illustrates the typical setup of a tabletop application. It consists of one (or more) projector(s) and one (or more) camera(s) for tracking users’ interactions (e.g. for tracking a laser pointer interaction). The interaction range and space of the application is usually limited due to the resolution of the projectors. Furthermore, most applications limit the interaction to a 2D interaction. Fig. 1 (b) shows the transformations between the spaces in a typical tabletop application. Let C ⊂ Z × Z to be the camera image space, M ⊂ Z × Z to be the application image space, W ⊂ \ 3 to be the world space, and S ⊂ \ 2 to be the 2D virtual interaction surface. Let f : C → M denotes the mapping that map a pixel in the virtual interaction surface to the corresponding pixels in the camera image; f −1 : M → C denotes the mapping that map a pixel in the camera image to the corresponding pixels in the virtual interaction surface. Mapping { g , h, i} and its inverse are the usual transformation between respective spaces. One of the promising features of a tabletop application is, however, that the interaction surface usually comprises an overlapping of the input and the output interaction space. Thus in this case, the interaction can be defined directly inside M and be detected visually by the camera in C . Consequently, what we need to implement the interaction is only to develop the mappings {f, f -1} (cf. Fig. 1(b)). Before we have a discussion over the main algorithm of our approach, it is common in general practice of computer vision system, especially in tabletop application to have an indication whether a pixel in camera is part of interaction surfaces. This indication is called Screenmask. Additionally, when we deal with multi camera setup, we also need quantities that can be compared from one camera to the other for the same interaction surfaces. This quantity is in the form of some kind of camera confidence function. A camera confidence function is a function that gives indication on how good camera sees a point in the considered surface in compare to other camera.


951

C ⊂ '×' W ⊂ \3

S ⊂ \2

M ⊂ '×'

(a)

(b)

Fig. 1. The setup of the Office of Tomorrow project1; (a) Usually the tabletop setup consists of multiple projectors and multiple cameras; (b) Transformation between spaces in tabletop application

Hence, we have noticed that the proposed camera calibration method needs to compute: (1) a function s ( x) to specify whether the points in camera image space is part of interaction surface image; (2) mappings {f, f -1} ; and (3) a function δ ( x) to specify a degree of confidence of mappings in specific points in camera setup relatives to other camera. 3.1

Screenmask Generation

Often we want to limit the processing only to the region that is really needed. For this reason, we need an indicator on whether a pixel is part of the region. In tabletop application with a 2D interaction space, it is useful to limit only to the region under consideration since it will preserve computing power and limit possible interference from the surrounding environment. Let s ( x ) : Z × Z → [ 0, 1] ∈ Z indicate whether the point x is in the region which defines an interaction surface, J ⊂ C . It is defined as

⎧0, x ∉ J s( x ) = ⎨ ⎩1, x ∈ J

(1)

We generate the s( x ) function by using the ability to control what is projected onto the surface. As depicted in Fig. 2, we firstly project a black image and capture it as a background image for the reference image. Secondly, we reduce the noise and increase the contrast. Next, we project a white image onto the surface and capture it again (a), followed by a conversion to a grey colored image and a threshold operation which is applied to the image (b). The next operation is the segmentation operation based on white color (c). Since we usually place manually the camera, the surface can be seen as dominant as possible in the camera image space. 1

www.coeno.org

952


Fig. 2. The automation of the Screenmask generation

The result represents the Screenmask. The corresponding matrix representation is a popular representation due to its simplicity although it takes a huge chunk of memory and there are other possibilities to be used, e.g. a neural network classifying function. The availability of Screenmask will also help in generating data sample for calibration of mapping {f, f -1} by minimizing interference from environment, especially in automatic setup. 3.2 Neural Network Based Function Approximation

Universal approximation is a nice property of multilayer perceptron (MLP) neural networks that has unique feature to the universality of approximation using standard fixed mathematical model such as polynomial approximation. The most important difference of the approximation function using neural network is that they are better able to cope with the curse of dimensionality in the sense that the approximation error becomes independent of the dimension of the input space (under certain conditions), which is not the case for polynomial expansions. The approximation error for MLPs with one hidden layer is of order of magnitude O(1/nh), compare to O(1/np2/n) for polynomial expansions where nh denotes the number of hidden units, n the dimension of the input space and np the number of terms in the expansion. Models that are based on MLPs will be able to better handle larger dimensional input spaces than polynomial expansions, which is an interesting property towards many real life problems where one has to model dependencies between several variables. One of the problems of using MLPs is the problem to decide the number of units in hidden layer. Although it has been proven that neural network with one hidden layer is in fact a universal approximator, the number of neuron unit in hidden layer that is needed is dependent on the smoothness properties of the function approximated and the desired error of approximation. Maiorov et al. show in [13] that any continuous function on the unit cube in \ d can be uniformly approximated to within any error by


953

two hidden layer MLP with 3d units in the first hidden layer and 6d + 3 units in the second hidden layer. Hence, we are using a four-layer with a six neurons unit in the first hidden layer, and fifteen units in the second hidden layer with the hope that it will maximize the use of sigmoid activation function. Hence, having calibration data, the mapping f can be generated by applying backpropagation algorithm to the neural-network. Moreover, with the same calibration data only with different order, the invese mapping f -1 also can be generated with the same function structure as the neural network for mapping f. 3.3 System Calibration

Sample data that is needed to be generated is a set of pair interaction surface image point ( xs , ys ) and its corresponding camera image point ( xc , yc ) . Since we can control which image has to be projected, we can generate the calibration image by putting a special designed calibration image to the interaction surface, detect it by the camera and calculate the corresponding camera image points.

(a)

(b)

Fig. 3. (a) Vision algorithm for sample data gathering; (b) Example of a calibration scene and the accuracy of the neural network approximation

Moreover, by using the above techniques, the sample data density will change according to the degree of importance. The near region, since it has more accurate reading, only needs more sparse data. The far region of the interaction surface will generate more sample data. A region with more spatial dynamic will generate data according to the bumpiness of the region. For calibration, we use the usual dot pattern as depicted in Fig. 3(a). We use the same algorithm as it has been used for creating the Screenmask to detect the dots (cf. Fig 2). After the dots are detected, we simply calculate the center of each dot by

954


⎛ xsc ⎞ 1 γ ⎛ xsi ⎞ ⎜ ⎟ = i∑ ⎜ ⎟, ⎝ ysc ⎠ γ =1 ⎝ ysi ⎠

(2)

where ( xsc , ysc ) is the center point of each dot segment, γ is the size of the segment in pixel values and ( xsi , ysi ) pixel position of ith segment member.

4

Implementation and Experimental Result

We implemented the neural network by using open source FANN neural network library [14]. In the experiment, we are using a desktop PC with 3 GHz, Pentium IV, for the projection we used a single-projector setup with a resolution of 1024 x 768 pixels. The typical learning time for a calibration image with 20 pixels radius dot was computed in around 117 seconds. In this case, we used the MPL network with 21 neurons in hidden layers. Fig. 3(b) illustrates the accuracy of the MLP-based mappings implementation. The red dots represent sample data generated for learning, and the blue dots represent an approximation achieved by the neural network. Notice that the right part of the camera space is naturally a region with better accuracy, and left part considered lower. It is shown that the region with lowest accuracy expectation i.e. image region of far top of the computer screen can be well approximated despite the limited amount of camera image pixels represents the area. 4.1 Automatic Calibration Algorithm

We automate the calibration process by using the same algorithm as the manual calibration model. In contrast, we project the individual dots one-by-one to the surface and apply the algorithm to each image. A delay is needed between the projection of a single dot on the surface and the capturing of the camera to grab the dot image. In our setup using a 3 GHz Pentium IV PC with a USB camera, we used a frequency of 0.5 seconds to project one dot image after the other. Notice that in our system, we use various number of dots based on its size. Smaller dot give more sample data with the expense of processing time. The typical setup is 20 pixels radius dot so for an interaction surface with 1024 x 768 resolutions will gives 540 dots, thus it took about 270 seconds for the automatic projection of the entire dot pattern. Unfortunately, we cannot put more than one dot images at a time, since we cannot differentiate which dot in the camera image surface corresponds to the dot in the calibration image. This feature gives the system high flexibilities that the camera does not need to see any additional cue with special feature. 4.2 Experimental Result

We developed a demo-application that tracks the movements of a laser pointer [15]. Thus, the users can manipulate the cursor by moving the red laser pointer. We used two different combinations of camera-interaction surfaces, a planar (cf. Fig. 4(a)) and non-planar (cf. Fig. 4(b)), and two different cameras with different properties, i.e. a low-end USB webcam (640x480 pixels) and a firewire Imaging Source camera with a


955

resolution of 1024x768 pixels and a Fujinon fisheye lens with 180° field of view. The non-planar surface is basically beyond the intended design which is a smooth and continuous surface. It contains a discontinuity and sharp edges. These direct usages of mapping f showed interesting mapping accuracy of the approach.

Fig. 4. In the demo-application, we used a planar wall surface (a), a distorted environment (b), a non-expensive USB webcam (c), and a firewire camera with fisheye-lens with 180° (d)

Fig. 5 depicts the accuracy results. Blue dots represent a calibration dot in application image space. Red dots represent approximation accuracy. The image (a) presents the accuracy by using a USB camera with no special lens and the planar surface of Figure 5 (a). Notice that the red dots are neatly overlapped by the blue dots. Figure (b) depicts the accuracy of the USB camera on the non-planar surface of Figure 5 (b). Notice that the approximation is practically similar. It is only a little bit off in the far left region of the surface, which is a normal since we stretch the application surface to such a distortion.

(a)

(b)

(c)

(d)

Fig. 5. Approximation accuracy for planar & distorted environment on wall surface with normal camera (a) and (b), fisheye lens (c) and (d)

956

6


Multi Camera Setup in 2D Interaction Surface

One of the main problems in the setup of the system with more than one camera is on how to manage the information come from more than one camera with various combinations of camera parameters. Considering Equation 2 which shows the quantity γ , i.e. the size of a dot segment. In our approach, this quantity can be a good basis to develop the third estimate, ∂( x ) . The apparent size of dot by the camera consistently indicates how good the estimation by mapping f will be. Bigger dot sizes, which are related to closer distance between the points to the camera, give more degree of confidence to the estimation result. This γ value for each of the camera that is looking at the same point in the system gives a kind of probability function of estimate of that point for each camera. Let ∂ i ( pi ) denote for camera-i degree of confidence γ for a calculated point pi = fi ( x ) with x is a detected camera image pixel by camera-i. A probability for point n is given as ⎧ ∂ i ( pi ) , ⎪

n = pi

ρ (n ) = ⎨ ∑ ∂ i ( pi ) ⎪ ⎩

0,

(3)

elsewhere

Hence, like the mapping f for each camera, the degree of confident function ∂( x ) is also being presented by the neural network based approximation function, which results in a smooth representation function over C . Notice that the neural network has the same structure of mapping f . Given the probability function ∂ i ( pi ) denotes for camera-i degree of confidence

γ

for a calculated point pi = f i ( x ) with x

is a detected camera image pixel by

camera-i. The best estimates for x is xˆ will be xˆ = Ε [ ρ (n )] =

∑ ∂ i ( pi ) ⋅ pi ∑ ∂ i ( pi )

(4)

Additionally, to coup with various possible camera resolutions, we can start with the idea by considering the fact that the camera with lower resolution has a lower degree of confidence than the camera with higher resolution with the factor as the ratio of the size of each camera image area. Let resolution for camera-i is specified by parameter ζ i = width × height / k where k is a scaling factor. Given probability function ∂ i ( pi ) denotes for camera-i degree of confidence γ for a calculated point pi = fi ( x ) with x is a detected camera image pixel by camera-i. Thus, the best

estimates for x is xˆ can be given by

xˆ = Ε [ ρ ( n ) ] =

∑ ζ i ⋅ ∂ i ( pi ) ⋅ pi ∑ ζ i ⋅ ∑ ∂ i ( pi )

(5)


7

957

Discussion

Although the above technique is designed specifically to solve the registration problem on the 2D interaction surface, it has been found that it can directly be used for solving 2D object augmentation problem. Furthermore, having the knowledge the geometry of the interaction surface and the pose of the camera relative to the interaction surface are known by a simple additional step, another kind of visual based registration problem can also being solved. Even the development of stereo camera pair can be done with the degree of complexity as simple as the complexity of the setup for one camera. Suppose we have a 2D object, such as 2D image, texture or video images etc., is defined in virtual interaction surface M, by using directly inverse mapping f -1, we can simulate the imaging process of the camera which put the 2D object inside camera image space C as seems as the 2D object is lied in physical surface S. Fig. 6(a) illustrates the augmentation achieved for 2D texture into environment scenes. It is illustrated in that the original scene can be altered by augmentation of 2D object into the image of original scene with correct perspective. Notice that the screen monitor is not a planar surface and the augmentation also shows such quality. The ability to be

(a)

(b)

(c)

(d)

Fig. 6. The example of extensions of the proposed method; (a) 2D object augmentation problem; (b) 3D object augmentation problem; (c) The camera pair for stereo camera experimentation; (d) Experimental data from points of a plane are seen by the stereo camera pair

958


used for tracking accurately and augmenting 2D object for more wide class physical surface is one of the strength of the proposed method over the standard method. A fiducial marker based applications is a popular augmented reality application. To be useful, the proposed framework should be able to do the same. Knowing the knowledge the geometry of the interaction surface and the pose of the camera relative to the interaction surface means that we have a knowledge of set of mapping { f , g , i} . Hence, we can use that knowledge to implement 3D object augmentation like Magic Book. Fig. 6(b) illustrates the implementation of the proposed method for solving a 3D object augmentation problem. Having similar setup like the 3D object augmentation problem above for two cameras with fixed position to each other, the stereo camera pair equation can be generated instantly. Fig. 6(c) shows the stereo camera pair that we have used and Fig. 6(d) illustrates experiment range data of a plane for the stereo camera pair.

8


For a 2D interaction surface with unknown geometry, the framework gives a straightforward solution to generate mapping between camera image space and interaction surface and still able to capture the spatial dynamic of the surfaces. Moreover, the technique gives a basis to create simple setup to manage system with multi-camera setup, especially for the future tabletop application where more than one projectors and cameras are needed. However, implementation of the technique into various applications is still needed to understand the complete behavior of the proposed method. In theory, the neural network approximation error can be minimized as small as possible. This allows developers to focus more on computer vision algorithm which can produce good visual cue detection with small error detection, rather than fighting with calibration errors. Nevertheless, more thorough analysis and experimentation need to be made to gain complete understanding the comparative advantage in the accuracy gain by the proposed technique over the standard method for various scenarios. Acknowledgments. This research is partly funded by the Austrian FFG consortium (FHplus framework, project no. 811407) and Voestalpine Informationstechnologie.

References 1. D. Molyneaux and G. Kortuem, Ubiquitous displays in dynamic environments: Issues and opportunities, Ubiquitous Computing 2004, 2004. 2. O. Bimber and R. Raskar, Modern approaches to augmented reality, SIGGRAPH 2005, 2005. 3. R. Raskar, "http://www.Mpcnet.Co.Jp/prodct/proj/index.html," 2006. 4. G. Baratoff, A. Neubeck and H. Regenbrecht, Interactive multi-marker calibration for augmented reality application, International Symposium on Mixed and Augmented Reality (ISMAR'02), 2002, 107.


959

5. R. Y. Tsai, An efficient and accurate camera calibration technique for 3d machine vision, IEEE Conference on Computer Vision and Pattern Recognition, 1986, 364-374. 6. R. Y. Tsai, A versatile camera calibration technique for high-accuracy 3d machine vision metrology using off-the-shelf tv cameras and lenses, IEEE Journal of Robotics and Automation RA-3 (1987), no. 4, 323-344. 7. M. Lynch and C. Dagli, Backpropagation neural network for stereoscopic vision calibration, SPIE Internationa Society of Optical Engineering, 1991. 8. J. Wen and G. Schweitzer, Hybrid calibration of ccd camera using artificial neural network, International Join Conference on Neural Network 1 (1991). 9. D. Choi, S. Oh, H. Chang and K. Kim, Nonlinear camera calibration using neural network, Neural, Parallel and Scientific Calibration 2 (1994), no. 1, 29-41. 10. M. Ahmed, E. Hemayed and A. Farag, Neurocalibration: A neural network that can tell camera calibration parameters, IEEE International Conference on Computer Vision (ICCV'99), 1999. 11. M. Ahmed and A. Farag, Locked, unlocked and semi-locked network weights for four different camera calibration problems, IEEE International Joint Conference on Neural Networks (IJCNN'2001), 2001. 12. Q. Memon and S. Khan, Camera calibration and three dimensional world reconstruction of stereo vision using neural networks, International Journal of Systems Science 32 (2001), no. 9, 1155-1159. 13. V. Maiorov and A. Pinkus, Lower bounds for approximation by mlp neural networks, Neurocomputing 25 (1999), 81-91. 14. S. Nissen, A. Spilca, A. Zabot, D. Morelli, E. Nemerson, Freegoldbar, G. Megidish, Joshwah, M. Pereira, M. Vogt, S. Hauberg, T. Leibovici and V. D. Massa, Fast Artificial Neural Network Library, http://leenissen.dk/fann/, 2006. 15. J.-Y. Oh and W. Stuerzlinger (Editors), Laser pointers as collaborative pointing devices, 2002.

Painterly Rendering with Vector Field Based Feature Extraction Chen Pang, Mingli Song, Jiajun Bu, Chun Chen, and Dong Wang Zhejiang University - Microsoft Joint Lab of Visual Perception, Key Lab of Ministry of Education Zhejiang University, Hangzhou, P.R. China {pangc, brooksong, bjj, chenc}@zju.edu.cn

Abstract. In this paper, a novel technique is presented to incorporate vector field based feature extraction schemes into painterly rendering. This approach takes a raster photograph as input and automatically creates a hand-painting style picture. Via techniques formerly used in image segmentation, a vector field representation are generated, identifying color and texture variations at each pixel location, and a series of brush strokes are created with sizes and alignments controlled by the vector field and color matched from the original picture. Moreover, different scale parameters could be utilized to produce several vector fields depicting images features of the original photograph from rough outline to detail. The final output could be rendered first by brushstrokes in the coarsest scale and refined progressively. Unlike conventional techniques that used taking account only of local color gradients, this approach employs multi-scale feature extraction scheme to guide stroke generation with image structure on larger scale.

1

Introduction

The automatic creation of artistic paintings from photographs has long been a favored topic of non-photorealistic rendering. Artists create representations of their view of the real world with expressive paintbrushes, usually with some level of abstraction. Researchers have made efforts to simulate this process and automatically produce images bearing visual resemblance to artistic works. Most methods in this class create a set of brush strokes from the source image and render them on a simulated canvas. The two steps address two major issues concerned: brush stroke generation and rendering. Some earlier systems created brush strokes randomly[1], later ones used local color variations[2], but a better representation of structural features in the source image usually leads to finer results. Computer vision techniques provide plenty of methods to extract useful information from raster images, ranging from color variations to object structures. When these features are employed as guidance of generating strokes, the resulting stroke set could be seen as an abstraction of

This paper is supported by the National Science Foundation of China (60573176).


Painterly Rendering with Vector Field Based Feature Extraction

961

the source image. The stroke rendering process varies from simply flood filling the area the stroke covers, to mapping real paintbrush texture, and further to sophisticated physical simulations of specific paint media. With the aims of utilizing image features on various scales in the stroke generation process, we incorporated into painterly rendering a recently proposed vector field based edge detection and image segmentation scheme[3][4]. These systems create 2D vector field representations of the source image as a preprocessing step for finding object boundaries. The vector fields contain quantitative information of image boundaries at each pixel location, which is essential in simulating many painting styles because the orientations of strokes in real paintings are also generally affected by adjacent edges. The quantitative representations of image features as vector fields therefore provide a superior basis for stroke alignment. In this paper, we take a raster photograph as input, and automatically generate a painting like image consist of several layers of brush strokes. The system starts by constructing multiple Edgeflow vector fields [3][4] from coarse to fine scales. At each scale and at each pixel location, a flow direction pointing to the closest boundary are obtained. Therefore by following the perpendicular direction of the vectors, strokes can be aligned to their adjacent edges. The overall painting process runs afterwards, first generating and rendering the strokes created from the vector field of the coarsest level, and update the canvas progressively on each finer scale, where minor edges cause inconsistencies with those of the previous layer. The next section introduces some techniques related to our work. Section 3 presents the multi-scale edge flow vector generation scheme. Section 4 discusses the overall painting process, followed by a detailed description of the brush stroke generation and rendering mechanism in Sect. 5, where image features represented in vector fields are mapped to characteristics of composed strokes. We provide some sample results in Sect. 6 and then conclude this paper.

2

Related Work

This paper follows a long line of works on stroke-based non-photorealistic rendering that began with Haelberi[1], who have shown that abstract paintings could be simulated using an ordered collection of brush strokes. Meier[5] created painterly rendered animations with a particle system. Litwinowicz[6] used clipped strokes with a given center, length, radius and orientation to produce impressionist-style video. Gooch et al.[7] pointed out image features obtained from segmentation could guide stroke generation. Kov´ acs[8] applied multi-scale edge and ridge detection[9] to control stroke sizes and painting directions. This approach bears some resemblance with the one in this paper, but in [8] strokes are modeled simply as rectangles and their orientation correspond only to the local color gradient of its nearest edge. Much of the inspiration of this paper comes from Hertzmann[2], who produced images with long curved brush strokes built up by several layers. Brush strokes are modeled as a series of B-splines positioned to a grid mapped on the image, and

962

C. Pang et al.

created with guidance of local color gradients. He also introduced rendering an image with multiple layers, starting from the coarsest level containing large strokes depicting the sketches of the image, to progressively finer scales of smaller strokes. Some other works focus on simulating artistic media, including brush, ink and the canvas. Curtis et al.[10] did some very successful simulations of western watercolor painting using a fluid model. Chu and Tai built a brush dynamics model[11] and a ink flow model[12] that provide a remarkable resemblance of eastern calligraphy and ink paintings. Improvements in image feature extraction techniques are also concerned. We focus our interest especially in boundary detection and image segmentation, because object edges require special consideration in various art forms. Plenty of methods on edge detection worked by convolving the image with predefined filters[13]. Detection of boundaries in these approaches is equivalent to finding the local maxima in gradient magnitude. Detecting and analyzing edges in multiple scales as well as resolutions is another widely studied technique, both from fine to coarse[13] and coarse to fine scales[14]. Lindeberg[9] introduced scalespace edge and proposed techniques on estimating optimum scales. Tabb[15] suggested an interesting idea of building a vector field for edge detection, and later in [3], Ma designed a Edgeflow vector field such that the direction of vectors at each pixel location points to the adjacent edge, and leaving scale as the only significant user parameter. Sumengen[4] augmented this approach by constructing multi-scale Edgeflow vector fields to reinforce major edges of an image, and to suppress insignificant boundaries.

3

Creating Multi-scale Vector Fields

As a preprocessing step, vector field representations of the source image is created for painterly rendering. Ma proposed in [3] the Edgeflow vector field scheme as an indirect method for boundary detection. It utilizes a predictive coding model to identify and integrate the direction of changes in image attributes. The result is a flow direction that points to closest boundary at each pixel location, with magnitude suggesting the possibility of finding an edge nearby. By propagating the flow following its direction, edges can be found at where two opposite directions of flow encounters. After boundary detection, image segmentation are completed by edge connection and region merging. The only user defined parameter significantly influencing the result is the scale value, which relates to standard deviation of the Gaussian. We are especially interested in the part of the work that a vector field representation of the original image is created that contains detailed edge information. By following the perpendicular directions of edge flow vectors strokes can be aligned to their nearby edges. Sumengen[4] extended Ma’s vector field design to multi-scale edge detection. The edge flow vector field is generated from finest scale and selectively updated with values of coarser ones. At each step, edge flow vectors whose magnitude are below certain level directly take the value from coarser scale, meaning empty areas are filled with vectors from larger scales. This way it can handle better some


963

boundaries that cannot be detected in fine scales, such as shades and blurred edges, yet can be captured at coarser levels. In addition, edge flow vectors with coherent directions are summed up to represent major edges. The result is a vector field that expresses image feature from multiple scales. To generate multiple vector fields, we takes much of the essence of Sumengen’s approach[4] but explicitly process oppositely, from coarse to fine scales, so that the painting process can proceed starting with larger strokes of more general level and refining by progressively smaller strokes. Begin on the coarsest scale, an Edgeflow vector field is generated as described in [3], and utilized to paint the first layer of the output image. The algorithm for generating vector fields on subsequent scales in this paper is almost identical to the one proposed by Sumengen[4], except that at each step the vector field is updated with the previous one on a coarser scale. This scheme suppresses edges that are detected only in fine scales because of noise or clutters, and favors those exist at a range of levels. The resulting stroke set thus correspond to salient image structures. Every intermediate result after one update are utilized to create a layer of the output image.

4

Overall Painting Process

A painting is built up by rendering multiple layers, based on the idea proposed in [2]. Artists often draw on a blank canvas with rough sketches first, creating the outlines by following the main contours of the objects, and then continues by adding smaller strokes to refine the details. Computer algorithms[2] based on the same motivation can also yield desirable effects. We start with a blank single color canvas, renders the stroke set generated from one vector field as a layer, and repeat the process to paint all layers from coarsest to finest scales with progressively smaller strokes. On each scale, only at places where the vector field suggests inconsistency with previous scale new strokes are created and the canvas is updated. The algorithm is adapted from the one presented in [6] to work with vector fields. Starting points of strokes are positioned with a jittered grid. program PlaceStrokes {Place stroke for one layer}; const GridSize; var I, J, ImageWidth, ImageHeight : Integer; S, T:VectorField; R:Real; begin for I:=0 to ImageWidth step GridSize for J:=0 to ImageHeight step GridSize begin Jitter [I,J] randomly within the grid; Let S be the vector field at the current scale;

964

C. Pang et al.

Let T be the vector field at the previous scale; if S(I,J) does not equal T(I,J) then Generate the stroke at [I,J] with radius R; end end. In the vector field generation process, vectors with very small magnitudes are directly replaced by their equivalents in coarser scale[4]. At these image locations, little color variations can be found in local areas. Therefore strokes of coarser scales covering these area already provided enough detail of the original image, # » eliminating the need of further updates. In the algorithm, such vector S(x, y) # » on scale s is obtained by taking the value from the next coarser scale, T (x, y). Hence the condition test to generating a new stroke is checking whether their values equal. The radius of generated strokes is consistent in the same layer, and provided as an user parameter. Radius R associated with each layer should correspond to the scale of that layer. Coarse scales have larger radius values.

(a)

(b)

(c)

Fig. 1. Example of painting multiple layers with progressively smaller strokes. (a) A source image. (b) Output image after rendering the first layer. Large strokes follows the edges of grapes. (c) Output image after successively rendering three layers.

5

Generating and Rendering Strokes

With the vector field on each scale, we generate a set of B-spline strokes and renders them on a layer of the result image. Each individual stroke is a weighted curve that follows the direction perpendicular to edge flow vectors, and tends to be parallel to its adjacent edges in the original image. Each stroke takes a constant color. To obtain this color, a reference image is created by convoluting


965

Fig. 2. Example image demonstrating placement of strokes. The two images on the right are the first and second layers of rendering result. Rectangle region on the original photo is magnified to show that brush strokes on the grassland are aligned parallel to major edges and shadows, while the ones above the grassland apparently carries more curves.

the original image with a Gaussian kernel of standard deviation corresponding to the scale used in creating vector fields, e.g. by multiplying the scale value by a constant factor. Then a stroke take the color at its starting point in the reference image, same as the method described in [2]. The stroke generation algorithm creates a list of control points from a given loca#» tion [x0 , y0 ] in the image. With the vector field S , from any pixel location [x, y], the location of the next control point is obtained by adding to the location coordinates # » # » a vector with the perpendicular direction of S(x, y), and length of R. If S(x, y) has #» the value of [S(x), S(y)], the vector s with value [−S(y), S(x)] is perpendicular to # » S(x, y). Thus the location of the next control point is given by: #» s [x , y ] = [x, y] + R × #» . (1) s where R is a user parameter that has larger values on coarser scales. We repeat propagating control points this way until any one of the following conditions is meet: – The stroke length reaches its maximin value. – Color difference between the reference image at [x, y] and the stroke color is greater than that between the reference image at [x, y] and canvas at [x, y]. – The vector at [x, y] is zero. – The angle between the vectors at [x, y] and at the start location of the stroke is larger than a predefined value A1 .

966

C. Pang et al.

(a)

(b)

Fig. 3. Example of a photograph rendered with three layers. Brush strokes curving to the edges and color variations on the flowers are still perceptible.

Fig. 4. More examples of results

– The angle between the vectors at [x, y] and at the previous location is larger than a predefined value A2 . The first two conditions is suggested and explained in [2], where color difference is defined as ((r1 − r2 )2 + (g1 − g2 )2 + (b1 − b2 )2 )1/2 . The other three are specifically suitable for vector fields. When the propagation stops, a brush stroke is determined


967

by the list of control points. Limiting the angle between the vector at the two ending of a stroke is optional. Without it, circular brush strokes are allowed in the result. The throttle parameter A2 mentioned in the last condition also leads to different visual effects. A small value will cause the resulting strokes be shorter and straighter, while a relatively larger value, e.g. π, only prevents the stroke from crossing image boundaries and restricting its curvature to a reasonable degree. Figure 2 demonstrates stroke alignments controlled by vector fields.

6

Results

In this section, we demonstrate some of the results from our implementation. Experiments shows that three to four layers with the final scale and grid size both set to 1 pixel provide satisfactory results. Over rendering usually leads to realistic appearance that bears too little artistic effects. Results are presented in Fig. 2 and Fig. 3. Both intermediate outputs and final results shows that brush strokes are aligned in the presence of multi-level image features.

7

Conclusion

This paper presents a painterly rendering technique that incorporated image feature extraction methods based on multi-scale vector field into the stroke generation process. Several layers of B-spline strokes are created according to quantitative edge information presented in multi-scale vector fields, and represent salient boundaries better than methods that takes into account only local color variations.

References 1. Haeberli, P.: Paint by numbers: abstract image representations. In: SIGGRAPH ’90: Proceedings of the 17th annual conference on Computer graphics and interactive techniques, New York, NY, USA, ACM Press (1990) 207–214 2. Hertzmann, A.: Painterly rendering with curved brush strokes of multiple sizes. In: SIGGRAPH ’98: Proceedings of the 25th annual conference on Computer graphics and interactive techniques, New York, NY, USA, ACM Press (1998) 453–460 3. Ma, W.Y., Manjunath, B.S.: Edgeflow: a technique for boundary detection and image segmentation. IEEE Transactions on Image Processing 9(8) (2000) 1375– 1388 4. Sumengen, B., Manjunath, B.S.: Multi-scale edge detection and image segmentation. In: EUSIPCO ’05: Proceedings of the 13th European signal processing conference. (2005) 5. Meier, B.J.: Painterly rendering for animation. In: SIGGRAPH ’96: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, New York, NY, USA, ACM Press (1996) 477–484

968

C. Pang et al.

6. Litwinowicz, P.: Processing images and video for an impressionist effect. In: SIGGRAPH ’97: Proceedings of the 24th annual conference on Computer graphics and interactive techniques, New York, NY, USA, ACM Press/Addison-Wesley Publishing Co. (1997) 407–414 7. Gooch, B., Coombe, G., Shirley, P.: Artistic vision: painterly rendering using computer vision techniques. In: NPAR ’02: Proceedings of the 2nd international symposium on Non-photorealistic animation and rendering, New York, NY, USA, ACM Press (2002) 83–ff 8. Kov´ acs, L., Szir´ anyi, T.: Painterly rendering controlled by multiscale image features. In: SCCG ’04: Proceedings of the 20th spring conference on Computer graphics, New York, NY, USA, ACM Press (2004) 177–184 9. Lindeberg, T.: Edge detection and ridge detection with automatic scale selection. Int. J. Comput. Vision 30(2) (1998) 117–156 10. Curtis, C.J., Anderson, S.E., Seims, J.E., Fleischer, K.W., Salesin, D.H.: Computer-generated watercolor. In: SIGGRAPH ’97: Proceedings of the 24th annual conference on Computer graphics and interactive techniques, New York, NY, USA, ACM Press/Addison-Wesley Publishing Co. (1997) 421–430 11. Chu., N.S.H., Tai, C.L.: Real-time painting with an expressive virtual chinese brush. Computer Graphics and Applications, IEEE 24(5) (2004) 76–85 0272-1716. 12. Chu, N.S.H., Tai, C.L.: Moxi: real-time ink dispersion in absorbent paper. ACM Trans. Graph. 24(3) (2005) 504–511 13. Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8(6) (1986) 679–698 14. Bergholm, F.: Edge focusing. IEEE Trans. Pattern Anal. Mach. Intell. 9(6) (1987) 726–741 15. Tabb, M., Ahuja, N.: Multiscale image segmentation by integrated edge and region detection. IEEE Transactions on Image Processing 6(5) (1997) 642–655

Reducing Time Cost of Distributed Run-Time Infrastructure* Zhong Zhou and Qinping Zhao School of Computer Science and Engineering, BeiHang University, Beijing 100083, P.R. China [email protected]

Abstract. RTI(Run-Time Infrastructure) provides services for HLA-based distributed simulation, and decides to a great extent the simulation scalability and efficiency. Distributed RTI has good scalability, but its time cost is always higher because of the architecture complexity. BH RTI is based on a distributed RTI architecture. Some techniques are used to overcome the problem of time cost. LoI-based data delivery algorithms are presented to speed up data delivery. PDU(protocol data unit)-function registry and RTI process model are designed to promote the efficiency of data packet processing. Experiment results illustrated that distributed BH RTI depressed the negative influence of architecture complexity and achieved a relatively lower time cost.

1 Introduction High Level Architecture (HLA) is the prevailing standard of distributed simulation. It consists of three parts, framework and rules, object model template (OMT) and interface specification[1]. Run-Time Infrastructure (RTI) is prescribed to implement the interface specification. It provides services for HLA-based simulation, and decides to a large extent the simulation scalability and efficiency. The standard of HLA 1.3 has been widely used in military fields[1]. However, when HLA was approved as IEEE 1516 standard in September 2001, a lot of changes were made[2,3,4]. Some RTIs have been developed such as DMSO RTI NG[5,6], pRTI[7], MÄK RTI[8], BH RTI[9], KD-RTI[10] and starlink[11] etc.. Because HLA doesn’t specify RTI implementation and interoperability, RTI design differs much. Different RTI cannot interoperate without the participation of each producers[12]. RTI architecture and performance are one of the main concerns in HLA/RTI applications. This paper presents a distributed RTI architecture implemented in BH RTI. It supports multiple HLA standards by shielding interface differences in Local RTI Component(LRC). Then some techniques to overcome the problem of time cost in distributed RTI are presented. In the end experiment and result analysis are given. *

This paper is supported by the National Grand Fundamental Research 973 Program of China under Grant No. 2002CB312105.


970

Z. Zhou and Q. Zhao

2 Problem Initiation There are many changes in HLA interface specification transition. Messages from SISO CFI workgroup said that next evolution of HLA standard would possibly be made in 2006 or 2007. Some means have been put forward to interoperate HLA 1.3 and IEEE 1516 federates. The Pitch AB inc. developed 1516 adapter [13]. It provides HLA 1.3 APIs based on 1516 RTI implementation. Then HLA 1.3 federates can run on pRTI 1516. The MÄK Technologies Inc. said that MÄK RTI 1.3 and MÄK RTI 1516 were built from the same code base. The two can retain compatibility between 1.3 and 1516 federates[14]. We developed RTIBridge [16] based on bridge federate[15] to link 1.3 and 1516 federates. RTIBridge can link federates between HLA 1.3-based BH RTI, DMSO RTI NG and IEEE 1516-based starlink. RTIBridge covers only part interfaces, and is only suitable for small scale simulations because of the bottleneck of bridge federate. Unpredicted revision may be made to HLA standard in future. There exist several versions of HLA standard. These problems bring much work for interoperation, other than transplant or modification. The trouble also exists in code preservation and tedious test procedure. In this background, the interoperation problem should be taken into consideration in further RTI development.

3 Distributed RTI Architecture RTI implementation has no fixed architecture. Current RTI architectures mainly have three types: central, distributed and hierarchical, according to the relationship of CRC (Central RTI Component) and LRC. DMSO RTI is the first RTI, and had been published for several years. It has a great influence on the field. A large portion of current RTIs are based on a traditional DMSO RTI like architecture as Figure 1. Considering the interoperation of heterogeneous systems, we made an effort in distributed RTI architecture and developed BH RTI.

RtiExec

FedExec

Federate(s)

LRC

...

Federate(s)

LRC

Inter-Process Communications

RTI Provided

Federate Provided

Fig. 1. Traditional DMSO RTI like architecture

The distributed RTI architecture owns several RTI nodes in peer to peer. Each RTI maintains only the required data to serve connected federates. BH RTI is divided into two processes, LRC and rtiexec (Figure 2). LRC provides a programming library following the standard. It is responsible for interface implementation based on the rtiexec. Process rtiexec is designed to provide common services of distributed simulation for LRC. The standard differences are shielded in RTI-LRC services. Then minor modification is required when the standard changes. New LRC can be developed based

Reducing Time Cost of Distributed Run-Time Infrastructure

971

on RTI-LRC service for a revised HLA standard or even another standard. With this distributed RTI architecture, much work on the standard transition can be cut off, and federates of different standard are easier to interoperate.

Fig. 2. Software Architecture of Distributed RTI

Although distributed architecture is good at scalability, time cost is a main negative problem for more layers and architecture complexity. Liu compared time cost of three RTI architectures[11] in which the time cost of distributed RTI is narrated as T(distributed)=T(sender_localLRC)+T(globalComputing)+T(localLRC_remoteLRC) + T(remoteLRC_receiver) The time cost of distributed RTI is higher than the other two with the comparisons. To be more precise, the time cost of this distributed RTI architecture should be T(distributed)=T(sender_localLRC)+T(globalComputing)+T(localLRC_RTI)+T(send er_rti)+T(RTI_RTI)+T(remote_RTI)+T(remoteRTI_LRC)+T(remoteLRC_receiver) The constitution of total time cost is rather complicated, including three network cost factors, T(localLRC_RTI), T(RTI_RTI), T(remoteRTI_LRC), together with four process cost factors. However, this is just analysis in theory. Each part of the cost may vary much in different implementations. The following parts present techniques applied in BH RTI to overcome the problem of time cost.

4 LoI-Based Data Delivery Algorithms Data delivery in HLA requires attribute set matching for publish/subscribe. Sometimes region overlap is also required when DDM is used. Existing publish-subscribe mechanisms can only judge whether a message is relevant to a subscriber or not. Aiming to solve the relevance evaluation problem, a new relevance evaluation mechanism Layer of Interest (LoI) was proposed in our previous work [17][18]. LoI defines a relevance classifier based on the influence of spatial distance on receiving attributes and attribute values. Some related LoI variables are listed in Table 1. Based on LoI, new data delivery algorithms are presented in this section. In our recent research, we drew two important deductions about the LoI relationship among the three parts, LoIs of the publisher, the subscriber and messages.

972

Z. Zhou and Q. Zhao Table 1. Symbols in the LoI Mechanism

Symbol

Definition

Pm(i ) pm( i , o )

LoI of publisher over object class i with m-size attribute set

η (ji , o )

LoI of attribute update/reflect with j-size attribute set of object instance o of object class i

S k(i )

LoI of subscriber over object class i with k-size attribute set

sl(i , o )

LoI of remote object instance o of object class i with l-size attribute set

LoI of local object instance o of object class i with m-size attribute set

Deduction 1. A publisher can only send attribute updates of LoI η (j i ,o )

≤

Deduction 2. A subscriber can only receive attribute reflects of LoI η (j i ,o )

pm( i ,o ) . ≤

sl( i ,o ) .

In the new publish-subscribe environment, the publisher works with Pm( i ) of object class i and pm( i ,o ) of local object instance o. And a subscriber works with S k( i ) of object class i and sl(i ,o) of remote object instance o. The four LoIs denote the dynamic detail relevance in publish-subscribe sides. Messages will be tagged with LoI η (j i ,o ) . η (j i ,o ) represents the fundamental relevance of messages. Then new data delivery algorithms below can be obtained, according to Deduction 1 and 2. Algorithm 1. (Algorithm for sending data) LoI_UAV FOR each attribute update of local object instance o int l = η (j i ,o ) , compute according to the Definition. IF (l

≤

p m( i ,o ) )

//attach l to the update packet; update.loi = l; multicast the update packet to the subscriber group; END FOR Algorithm 2. (Algorithm for receiving data) LoI_RAV FOR each attribute reflect of remote object instance o int l = reflect.loi; //get η (j i ,o ) IF (l

≤

s l( i ,o ) )

//the reflect packet is wanted by the subscriber accept the reflect packet; callback the corresponding user function; END FOR

Reducing Time Cost of Distributed Run-Time Infrastructure

973

Complex and costly attribute set matching is simplified, and the data sending and receiving become more efficient. Receivers can also perform precise attribute set matching as Algorithm 3. Algorithm 3. (enhanced receiving data) PreciseLoI_RAV FOR each attribute reflect {HVPj{}, loi} of remote object instance o int l = reflect.loi; IF (l

≤ sl

( i ,o )

)

WHILE(int h = 1; h d th ⎧0 ⎪ . ψ =⎨ d th − d min arctan( ) otherwise ⎪ d th ⎩

(2)

984

B.-S. Shin, H. Ahn, and G.Y. Kim

Fig. 4. Determining the direction of walk-through using the range data from ultrasound sensors. The next direction Dnext can be decided with Dcurr, Dmax and dmin.

2.2 Stereoscopic Vision System We develop a stereo vision system for helping a visual handicapped person. The system is composed of a hardware device and several software components. The two stereoscopic cameras are connected to the portable station. Software components include Color Detector, Face Recognizer and Object Finder/Tracker. The Color Detector analyses an input image from a camera and outputs a voice message. For the purpose of learning color values, we make 16 color plates, capture the images and extract the average color values, A1..16, of each image. In the color detection step, we use eigencolor normalization [7] for compensating the adverse effect due to shape distortion for 2D planar image. And we extract the major color value, C, from a histogram of the image and find a color code using Eq. (3).

Ccode = arg min i =1..16 | Ai − C | .

(3)

The Object Finder/Tracker is used to guide handicapped persons toward an object that they want to take. The scenario is as follows: (1) it manually captures an image and points an object, (2) automatically finds and tracks the object using Snake [8], and (3) computes the object's position and orientation from stereo matching. In learning step, it performs SIFT (scale invariant feature transformation) to extract some descriptors from training set. It performs SIFT on a target image and compares the stored descriptors and newly extracted ones in recognition step. The Face Recognizer is used to identify persons around the user. The Face Recognizer is composed of face detection module and recognition module. We apply an efficient and robust face detection algorithm to extract facial regions in a sequence of images when persons are moving in front of the camera. Popular techniques to detect faces in an image often involve skin-color segmentation [9]. Skin-color segmentation is fast, however, it can only be done in well defined surroundings. We want our detection algorithm to be robust even under totally different lighting conditions. So, instead of using skin-color, we implemented the Haar face detection algorithm [10]. Fig. 5. shows an example of detected faces using Haar face detection algorithm.

Wearable Multimodal Interface for Helping Visually Handicapped Persons

985

Fig. 5. An example of applying Haar face detection algorithm. Rectangles represent facial region on images.

To recognize the detected facial regions robustly, we applied embedded HMMbased face recognition algorithm [11]. This algorithm is known to be more efficient and robust to scale changes than the classical eigenface methods and the previous HMM-based algorithms.

3 Experimental Results Fig. 6 shows the configuration of our system for visually handicapped person. Main processing is performed on ARM9-based embedded system equipped with RS232C/USB interface and 1GB memory. Operating system is embedded Linux. Output of this system is represented as a series of voice message or vibration pattern with earphone and vibration jacket. Eight ultrasound sensors are arranged with three rows, top row has two sensors to detect objects hazardous to user’s head, middle row has four sensors to compute safe path to target point, and bottom row has two sensors to recognize small obstacles on the floor, staircases, or cliffs. The number of sensors is trade-off manufacturing cost and the size of sensor array for accuracy. Eight sensors are sufficient for this application. The sensor array scans a scene 2.5 times per second. For safe work-through, computed direction vectors are quantized into four directions and mapped to voice messages or vibration patterns representing “rotate-left”, “turn-left”, “turn-right”, and “rotate-right”, respectively. It produces additional commands such “start”, “stop”, “watch your step” (in staircase or cliff), and “take care of your head” according to some situation. Fig. 7 depicts an example of performing our obstacle detection and avoidance feature. A user recognizes a wall in front of him and changes his direction to avoid collision to the wall. Fig. 8 shows an example of a detected face and the recognized face retrieved from the facial DB. For evaluation of the recognition algorithm, we learned 50 persons from a public facial database provided by AT&T. 50% of the facial images of each person were used for learning and the remaining 50% were used for the evaluation. From our evaluation the recognition rate was 99% for the public database. Experimental results show that detecting the facial regions takes 40ms~120ms while using 320×240 images and face recognition requires about 100ms for sample data. Fig. 9 shows an example of finding and tracking an object with our method. Tracking time is about 1 second.

986


ARM-9 based Wearable PC

pre-calibrated stereo cam-

Sensor controller (USB interface)

Back pack Orientation tracker

Vibration Jacket (RS-232C)

Sensor array

Ultrasound sensor

Wearable system

vibra-

Fig. 6. Configuration of prototype system

Fig. 7. An example of using obstacle avoidance system

Fig. 8. Face recognition using an embedded HMM. It can detect a facial region (left) and recognize the face using face database. Actual output is converted to voice message and offered to the user.

Wearable Multimodal Interface for Helping Visually Handicapped Persons

987

Fig. 9. An example of finding target objects using SIFT features (left) and tracking and object using Snake

4 Conclusion We designed and implemented a system to provide alternative sensory for visually handicapped persons with wearable multimodal interface. It acquires visual and spatial information with a variety of sensors, converts them to high-level information, and represents the information with voice command or vibration patterns. The system is mainly composed of two sub-systems, obstacle detection and avoidance module and stereo vision module. The first is formed with multiple ultrasound sensors and a tracker and it helps handicapped person to access to target place easily and safely. The second is stereo vision system to provide ability to track some target objects, to recognize persons around a user, and to understand color and gray-level of a specific object. Since our system is designed for wearable computers, it is very easy for visually handicapped persons to operate the system.

Acknowledgements This work was supported by IITA through IT Leading R&D Support Project.

References 1. Borenstein, J., Ulrich, I. : The GuideCane : A Computerized Travel Aid for the Active Guidance of Blind Pedestrians, Proceedings of the IEEE International Conference on Robotics and Automation (1997) 1283-1288 2. Mori, H., Kotani, S. : Robotic Travel Aid for the Blind: HARUNOBU-6, Proceedings of the 2nd European Conference Disability, Virtual Reality & Assoc. Tech. (1998) 193-202 3. Borenstein, J : The NavBelt - A Computerized Multi-Sensor Travel Aid for Active Guidance of the Blind, Proceedings of the CSUN's Fifth Annual Conference on Technology and Persons with Disabilities (1990) 107-116 4. Jiang, K., Seveviratine, L. D., Earles, S. W. E. : A Shortest Path Based Path Planning Algorithm for Nonholonomic Mobile Robots, Journal of Intelligent and Robotic Systems (1999) 347-366 5. Divelbiss, A. W., Wen, J. T. : A Path Space Approach to Nonholonomic Motion Planning in the Presence of Obstacles, IEEE Transactions on Robotics and Automation, VOL. 13, No. 3(1997) 443-451

988


6. Hu, H., Gu, D., Brady, M. : Navigation and Guidance of An Intelligent Mobile Robot, Second Euromicro Workshop on Advanced Mobile Robots (EUROBOT '97) (1997) 104-111 7. Lin, Z., Wang, J., Ma, K. : Using Eigencolor Normalization for Illumination-Invariant Color Object Recognition, Pattern Recognition, Vol. 35 (2002) 2629-2642 8. Kass, M.m Witkin, A., Terzopoulos, D. : Sankes: Active Contour Models," Internation Journal of Computer Vision, Vol. 2 (1988) 321-331 9. Majoor, T. : Face Detection using Color-based Region of Interest Selection,. Technical report, University of Amsterdam, Amsterdam, NL, 2000 10. Lienhart, R., Maydt, J. : An Extended Set of Haar-like Features for Rapid Object Detection, IEEE ICIP, Vol. 1 (2002) 900-903 11. Nefian, A.V. and Hayes, M.H. : An embedded HMM-based approach for face detection and recognition, Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing, Vol. 6 (1999)

Personalized Primary Port: Analysis of User View for the Smart Environment Wei-jin Park1, Junhwan Kim1, and Doo-Hwan Bae2 1

WRG, 22th Floor, Parkview Office Tower, Jeongja-Dong 6, Bundang-Gu Seongnam-Si, Gyeonggi-Do, 463-863, South Korea {jin.park, james.kim}@wrg.co.kr 2 Dept. of Computer Science, Korea Advanced Institute of Science and Technology 373-1, Kusong-dong, Yusong-gu, Taejon 305-701, Korea [email protected]

Abstract. In this paper, the user side view of smart space based on a hand-held device was proposed. We gave precedence to the personalized primary port (PPP) ─currently the cell phone─which is the most popular personal belonging with a powerful functionality among the ubiquitous computing environment. It can be used as the gateway to ubiquitous computing for the smart environment. The implementation focus is a combination of short-term-based sensing using an accelerometer and long-term-based user preference sensing through a software robot. Keywords: Smart space, personalized hand-held device, software robot, sensor.

1 Introduction Since the concept of ubiquitous computing (ubicomp) was presented by Weiser [1], many different kinds of approaches and domain applications have been provided. One of the application views is the smart environment technology. Smart environments integrate technologies in pervasive computing, artificial intelligence, interfaces, mobile communications, sensor networks, and wireless networking into our everyday setting, including homes, classrooms, offices, laboratories, shops, cars, etc. [2]. The phrase “smart space” has also been used in some research projects such as the NIST Smart Space Project [3] and the CSIRO Smart Space [4]. The words ‘space’ and ‘environment, are often used interchangeably and are relatively abstract concepts of general sites or places [2]. There are various reasons for developing a prototype that involves hand-held devices, which are sensed and activated in the user context: (1) There are different levels of priority among multiple computers per user in the ubiquitous computing environment. Some computers play more significant roles than others or a specific computer can be the primary. (2) Currently, hand-held devices, especially cell phones, can be regarded the nearest computing, communicating, content-creating devices available to users anywhere. Ubiquitous computing should give a more concrete form to involving hand-held devices at this time. Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 989 – 997, 2006. © Springer-Verlag Berlin Heidelberg 2006

990

W.-j. Park et al.

(3) Even though many context-aware frameworks have been presented in the last few years, what is usually missing is the notion of characteristic features of contexts that are accumulated during long time intervals. (4) It is necessary to extend the concept of limited space of smart environment to every general space where we live.. If a user in a ubiquitous environment has more than one computer, there should be priorities or precedence set among computers. We define the concept of primary computer among many computers and deal with the role, interaction, and coordination of the primary computer. This research aims to develop a smart space application with short-term sensing and long-term preference capture from a personal point of view. The implementation of smart space can provide popularized personalization, which can resolve the conflict between personality and popularity. This paper provides insights on the current and future role of hand-held device systems for the smart environment. The technical goal is preservation and transmission of user preference during context transfer so that the user’s intention can be recognized in the personalized primary port (PPP) as though it had originally appeared there. The structure of the paper is as follows. Section 2 presents the current research status and a review of the smart environment-related research. Section 3 describes our concept of the hand-held centric smart space and PPP. Section 4 introduces the prototype implementation, including personalized user interface using software robot (sobot) and sensor interface embedded into the hand-held device. Section 5 gives a summary of the paper.

2 Related Research It is generally difficult to specify the smart space because every researcher has his/her own focus of interest and, in some sense, we already have been living in certain level of smart space. We can find lamps with sensors; ovens capable of cooking automatically following instructions from the Internet; and smart functions using embedded computers in a car, furniture, and so on. Related research regarding personalized smart environment based on hand-held device is presented in three categories: (1) smart environment, (2) personalization in ubicomp, and (3) hand-held application in ubicomp. In a broad sense, smart space means ubiquitous computing environment. In a narrow sense, smart space refers to specific spaces of intelligent interplay to maximize the convenience of people without limitations of time and space among every object that is part of a future ubiquitous environment. It is also a new space created by computing for intelligence, communication for access, and service for delivery of information and knowledge. To specify a more meaningful ubicomp environment or smart space, it is necessary to specify or define a degree/level value attached to the definitions describing the terms ubicomp and smart environment technology. It is a fact that some objects already have a certain level of smartness and we have already been surrounded by these. A smart space is an environment with numerous elements that "(1) sense + (2) think + (3) act + (4) communicate + (5) interact with people” in a way that is robust, self-managing, and scaleable [CSIRO]. In research projects such as Boulder’s Adaptive House and Buxton’s Reactive Environment, their smart space is only

Personalized Primary Port: Analysis of User View for the Smart Environment

991

focused on a single specific space, and the different spaces developed are isolated, looking like small islands in an ocean. However, a person’s daily activities often cover multiple spaces or environments. [MA 21]. The smart space of NIST ITL and DARPA focused on ‘smart’ than on other elements. For example, for the phrase semantic web, pervasive computing is widely used. A set of associated and integrated smart spaces as a unity can be regarded a higher level space─i.e., a smart hyperspace or a hyperenvironment. And the smart “hyperspace” can be seen as a subset of the “hyperworld.” FLAME2008 described an approach, focusing on ontology design, user model, contexts and situations, and how all these are used to provide a user with personalized situation-dependent services. Shahi’s personal operating space is an entity formed for the personalization of space-specific services─i.e., by treating a user’s personal mobile device as an identity for personalization. One of the personalized applications, mPERSONA, is a flexible personalization system for the wireless user that takes into consideration user mobility, the local environment, and the user and device profile. Ozen described the architecture for mobile network operators to deliver highly personalized information from XML resources to mobile clients. The smart card-based personalization approach shows that the use of a smart card to host the personal profile engine answers both privacy and ubiquity requirements. Until recently, research regarding smart environments or ubiquitous space has been limited to describing a specific space and has been incapable of handling general spaces in our everyday lives. This work provides ubiquitous computing with particular emphasis on the role of primary hand-held device, which can be used as the gateway to smart space. Other related research differs from ours in the following sense: (1) Experience vs perception: In the smart space of NIST, a perceptual interface was set up in a limited space and personality was recognized. If the results of perception could be transmitted through the mobile device to the service in space by the mobile device as medium for perception, experience could take the place of perception. The implementation of smart space had centered around the cell phone to recognize personality. (2) Our hand-held centered smart space (HHCSS) differs from the previous one in the following aspects: - Primary reasoning engine to protect the user in the smart flow - HHCSS have interaction modes with objects based on user experience - Position and orientation of the user can be used by some service through the hand-held device but not restricted to the certain bounded space like a room or house.

3 PPP in the Hand-Held Centric Smart Space The Jansen formalized the smart space consisting of three entities–user, sensor, actuator–that are relevant to creating a programming model for pervasive space. Based on context, a user has a different set of desires about the behavior of space. Actuators, whose invocation has an intended effect on a particular domain observed by a sensor, influence the state of the space [14]. A user interacts with the space and has a set of long-term preferences and short-term intentions. The sensor senses the

992

W.-j. Park et al.

user’s current status and any status brought by any other behavior in the smart environment and provides information to the system about the current state of space. It is a fact that computers are becoming smaller and smaller and can be attached and embedded into almost all things─from man-made devices to natural ones in the world. The design modeling of smart-space is divided into the four levels: 1) framework (e.g., smart classroom framework, NIST smart space framework); 2) device (e.g., smart furniture of Hosei University); 3) component (e.g., springs); and 4) elements (e,g., rectangular cells on the mask). The Smart_Space, Smart_Flow, and Smart_Component classes are subclasses of the Smart_Artifact class. The sensor, actuator, and user are the subclass of Smart_Feature. The sensor is specialized into an HW_Sensor and an SW_Sensor.

Fig. 1. (a) User is encapsulated by the PPP in the loop of Smart flow. (b) Object relations in smart space for describing PPP.

PPP is the generalized concept of current hand-held device─the cell phone is the 2006 version of PPP. It may disappear into our clothes or embedded in other belongings, all through the wearable computer paradigm with functionality as follows. (1) Identifier: To identify a person, an identification (ID) card, such as a driver’s license, is mainly used. However, this may be cloned illegally and has no other functionality beyond the ID. Nowadays, diverse ways, with pros and cons from the points of view of security or cost, are being used, including other parts of the body such as the fingerprint. Future electronic purse including ID may converge into an invisible pervasive wearable device. (2) Preference management: If PPP can sense the user’s action and store it for future use, an analysis may be done to capture user preferences. (3) Convergence of digital belongings: Recently, the functions of personal devices and belongings are becoming more integrated─for example, the digital camera and MP3 player or the portable memory stick may be the default function of cell phones. The cell phone will have more functionality, including devices whose life cycles are not long and are easily out of fashion. (4) Security: In the ubiquitous computing environment, the user may be exposed to unauthorized computing device anywhere. (5) Basic actuation: This function can be covered by the PPP. Examples are emitting flavor, automatic scheduling in diary, etc. (6) Controlling other devices: For example, a noncontact car key using smart card may be developed. It can be used instead of a 3-D wireless mouse or joystick or keyboard because some small device has no input device.


993

4 Personalized Primary Port (PPP) Prototype The PPP can be used to capture not only short-term-based user context but also longterm-based preference. This section describes the PPP prototype from the user point of view. 4.1 Short-Term User Action Sensing on PPP by Sensor Application with an accelerometer in the PPP has two main features. One is static application of measuring the tilting angle to the ground and applying it to a UI manipulation. The other is dynamic application of measuring the signal on a specific bandwidth and applying it to calculate human step and kinetic energy. The acceleration, speed, and displacement of the object can be measured through the accelerometer, making it possible to calculate the motion of the object.

Fig. 2. Short-term sensing framework by hardware sensor: (a) acceleration sensor-based application, (b) wave from user gesture using an accelerometer, (c) wave and relevant action of user, (d) prototype implementation, and (e) process of sensor attachment into the cell

The PPP can recognize some user gestures. Waves in the figure show some examples of user gestures: (1) 9 clock direction to the 3 clock direction; (2) heart shape; (3) up, down, left, or right; (4) diagonal direction; (5) drawing a rectangle; (6) two-time shacking for the manner mode; and (7) sample emergency call. The PPP can recognize user movements such as stopping, walking, and running and how much they have moved. The figure shows capture of acceleration signal at different speeds. There are several attachment methods at this stage: pocket in pants, pocket in shirts, necklace type, and attachment to belt. In this prototype, the necklace type was used because it does not depend on the mobile phone type as shown in the figure. Fig.17 (?) shows the embedding sensor into the cell. The external attachment consists of main chip sets. A precise periodic clock should be provided to the MCU of

994

W.-j. Park et al.

the solution as (1) the MCU captures acceleration data and (2) stores data to the serial EEPROM. After opening the folder of the phone and executing the application, data from (3) stage can be transferred to the sobot application. The internal attachment needs a smaller number of main chip set and space. Flash memory in the cell-phone is used to store captured data and communication between sensor and phone performed through the I2C. Data on sensed and measured movement can be accessed by the sobot application. 4.2 Long-Time-Based User Experience Sensing The user context manager, which aims to process long-term user experience, has the sensor (accelerometer) for user status, especially gesture and movement sensing. It has several pattern managers for the user’s long-term preference analyses. For instance, a device configuration may ask “whether it has DVB-H functionality or not;” the browsing pattern is similar to a cookie in the Web browser. The calling pattern is “who does the user like or do not like to call” or “when and how much time does the user like to call in a specific context.” Another pattern may mean “when does the user go to bed at night”. When an event occurs that is matched with the case in the event database, the proper UI is shown for the user. The gray box is the PPP which has the user context manager and sobot processing module.

Fig. 3. Processing framework of the PPP

All ubiquitous services should be through the PPP as it has control over other devices so that user preferences are directly reflected and services and events are selected and customized. There are two cases of coupling─tight and loose. In the loosely coupled case, the other device can refer to the user preference in the PPP but it is the device’s

Fig. 4. Kinds and levels of personalization


995

choice how much context information in the PPP to take or not to take. The figure shows the GUI of the sobot (software robot) interacting with the user. The sobot interface consists of five parts: icon, background, speech balloon, button list, and sobot. Figure 4 shows the user profile consisting of short-term sensing context and longterm preferences for managing personalization through the PPP. The Y axis is the level of personalization and the X axis is the term duration of the personalization. The simplest example of widespread personalization is “personal ring tone” and “appearance” of the cell. Figure 5 shows the composition structure of the sobot, illustrating its characteristics and functionality.. The PPP provides users with sensuous interface using sobot which can be the virtual identity of the user for the smart space. The sobot character can be used by linking with the web or from a game character, which has several parameters compatible with those of the game character. This sobot upgraded automatically has intelligence and user’s preference. Besides the basic function like spam mail and message filtering, schedule management, and help for the cell phone menu item, sobot plays the role of an agent, which recognizes and expresses the current context. As a biological model, it is an electronic pet with intimate interaction with the user; as a motion model, it has its own characteristic response to the user, considering his/her preference; and as a growing model, by user manipulation, it can be growing. Thus, appearance, intelligence, preference, and game character parameters can vary, depending on the interaction with the user.

Fig. 5. Display and structure of lifelike character in the sobot architecture

As a sample response, sobot notifies the user when a message is received from someone and opens it when the user gets to open the message. The response depends on the sender. If a male user gets the message from a female sender, the sobot may ask, “Is she your girl friend?”. The sobot thread is executed as a subprogram, but the programming environment for a hand-held device is so limited. The over-ray function for displaying outputs of two programs simultaneously may not be available. Three kinds of Avatar system implement the hook in the PDA/phone system kernel. The kinds of hook are the keyboard hook, the digitizer hook, and the screen hook. The keyboard hook is for receiving keyboard input; the digitizer is for receiving pen input in the phone OS, and the screen hook is for getting a memory buffer value before displaying it in memory buffer. The calling hook makes it possible to display the output of more than one program. The PPP prototype presents a framework of integration of sensing from the hardware and the software. Not only can current shortterm context but also long-term user experience be captured by the PPP. Since the PPP is not yet part of a user’s identity but rather just a kind of mobile device, the benefits of interaction based on PPP may not be easily seen.. However, the concept of

996

W.-j. Park et al.

PPP is still feasible because the cell phone is the most common belonging that has the communication and computing capabilities and that uses other recent technologies such as multimedia content viewing. Sensor attachment is possible only with the small variation of the existing model because, in MEMS manufacturing technology, size does not matter. Currently, the sensing interface is limited to only the area of sensing of the project. User reaction to the sobot is restricted to the push button that the sobot displays or to gestures with the phone. Several kinds of extension are in progress such as the diverse interaction method that includes voice interaction.

5 Conclusion The perceptual interface through a mobile device, especially a person’s PPP (currently the cell phone), can be the most likely gateway to smart space. In this research, the concept of user side view of the hand-held centric smart space and PPP was introduced. For the implementation prototype of the PPP, the capture of user preference (through an investigation of long-term user experience) and user context (through the hardware sensor in the PPP) was presented. Future research includes exploring advanced combination technology between mobile device experience and numerous services in smart space.

References 1. Weiser M. 1993. Hot topics: ubiquitous computing. IEEE Computer. 2. Jianhua Ma. 2005. Ubiquitous intelligence: the Intelligence Revolution. Wise Media, http://cis.k.hosei.ac.jp/~jianhua 3. Abowd GD., Atkeson C, and Essa I.. 1998. Ubiquitous smart spaces. A white paper submitted to DARPA (in response to RFI). 4. http://www.smartspaces.csiro.au/ 5. Abowd GD, Iftode L, Mitchel H. 2005. The smart phone: a first platform for pervasive computing. Pervasive computing. p 18-19.. 6. Gellersen HW, Schmidt A, Beig M. 2002. Multi-sensor context-awareness in mobile devices and smart artifacts. Mobile Networks Appl. 7 (5). 7. Benbasat AY, Paradiso JA. 2001. An inertial measurement framework for gesture recognition and applications. Paper presented at the International Gesture Workshop, GW 2001, London, UK. 8. Hansen TR, Eriksson E, Lykke-Olesen A. 2005. Mixed interaction space–designing for camera-based interaction with mobile devices. CHI (ACM Computer-Human Interaction) 2005 Apr 2-7, Portland, Oregon, USA. 9. Dannenberg RB, Amon D. 1989. A gesture-based user interface prototyping system. In: Proceedings of the 2nd ACM SIGGRAPH, User Interface Software and Technology. 10. Hinckley K, Pierce J, Horvitz E, Sinclair M. 2005. Foreground and background interaction with sensor-enhanced mobile devices. ACM TOCHI 12 (1). 11. Mantyjarvi J, Kela J, Korpipaa P, Kallio S. 2004. Enabling fast and effortless customization in accelerometer-based gesture interaction. Paper presented at the Third International Conference on MUM.


997

12. Jianhua MA, Yang LT. 2005. Towards a smart world and ubiquitous intelligence: a walkthrough from smart things to smart hyperspaces and bicKids. J. Pervasive Comput. Comm.1(1). 13. Shahi A, Callaghan V, Gardner M. 2005. Introducing personal operating spaces for ubiquitous computing environments. Paper presented at the Third Inernational Conference, PERMID, 8-13 May 2005. 14. Jansen E, Abdulrazak B, Yang H, King J, Helal A. 2005. A programming model for pervasive spaces. Paper presented at the Third International Conference on Serviceoriented Computing, The Netherlands, December 2005. 15. Van der Meer S, Jennings B, Barrett K, Carroll ,R. 2003 Design principles for smart space management. Paper presented at the First International Workshop, MUCS, Waterford, Ireland, 11 December 2003. 16. Tokuda H. 2003. A prospect of ubiquitous computing environment: new applications and issues. Paper presented at the IEEE International Workshop on Networked Appliances, Tokyo. 17. Potonniée O. 2002. Ubiquitous personalization: a smart card-based approach. Paper presented at the 4th Gemplus Developer Conference, Singapore, 12-14 November 2002. 18. Panayiotou C, Samaras G. 2004. mPERSONA: personalized portals for the wireless user: an agent approach. Mobile Networks Appl. 9: 663–677.

Mobile Information Presentation Schemes for Supra-adaptive Logistics Applications Bj¨ orn Schwerdtfeger, Troels Frimor, Daniel Pustka, and Gudrun Klinker Fachgebiet Augmented Reality, Department of Informatics, TU M¨ unchen, Germany {schwerdt, troels, pustka, klinker}@in.tum.de http://campar.in.tum.de

Abstract. In supra-adaptive logistics systems, operative workers must be able to quickly and precisely retrieve objects from large storage areas with minimal training. We explore concepts toward providing workers with location-based wayfinding and picking information on mobile and spatial displays, varying both visualization schemes (1D: textual descriptions, 2D: map-based overviews and 3D: AR-based arrows) and display devices (head-attached, hand-held and spatial displays). In this paper we describe the system and report on first user evaluations. Keywords: Augmented reality, order picking, commissioning, mobile information presentation, user interface evaluations.

1

Introduction

With markets becoming increasingly global and supply and demand chains undergoing rapid changes in quality and quantity, today’s logistics systems need to become supra-adaptive, i.e., they have to be able to adapt with minimal effort to global dynamic changes [1]. As one consequence, logistics companies need flexible workers who can adapt to new working conditions and environments quickly and with minimal training – e.g. for order picking. Mobile and flexible information presentation is a key issue toward reaching this goal. 1.1

Item Picking in Logistics Applications

In order picking tasks, workers collect sets of items from an assortment in a warehouse according to a work order and deliver them to the next station in a precisely designed material flow process [2]. The efficiency of such picking processes is divided into time measurements of four interleaved tasks: the base time for getting the next order information, the dead time during which a worker interprets and understands the order as a 3D navigation and picking task, the way time during which the user physically moves to the selected item and the picking time to actually grab the item. While the base, way and picking times have been the subject to many optimizations, we focus providing schemes to reduce the dead time. Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 998–1007, 2006. c Springer-Verlag Berlin Heidelberg 2006

Mobile Information Presentation Schemes

999

Traditionally, order picking is accomplished by providing workers with printed-out pick lists of articles, stating their position in the warehouse, the amount to be collected, and short descriptions. To increase efficiency and reduce the number of picking errors, current industrial setups start using two techniques [2]. Pick-by-Light approaches use lamps (LEDs) that are installed with each lot of items on the warehouse shelves. When a new order is processed, lamps corresponding to each item on the list are turned on to show the lot and the amount to pick. A drawback of Pick-by-Light solutions is the chance of errors when lamps fail. Furthermore, the system cannot simultaneously support multiple workers picking items in the same shelf area. In Pick-by-Voice systems, workers wear headsets and receive auditory information about the next item to collect. The advantage is that workers do not need their hands to operate the system. Yet, the system requires considerable mental efforts since workers have to remember what has been said. 1.2

Related Work

To overcome problems with large and complex warehouses or with inexperienced workers, different systems have been developed with the focus of easing the work burden. Several researchers have investigated approaches supporting navigation in large environments. Raskar et al. [3] focus on building context aware systems with a variant of RFID tags that are attached to the items in warehouses and transmit geometric data of the items. This is used together with hand-held projectors to highlight the boxes and items or show other information for the worker. Butz et al. [4] attach visible markers on the points of interest and combine the projector with a camera. M¨ uck et al. [5,6] use ARToolkit markers attached to the shelves. They then use HMDs with attached cameras to register and augment the points of interest in the shelves. This approach is also explored by Reitmayr and Schmalstieg where they augment sign posts into the environment [7]. F¨ ockler et al. [8] use camera-equipped mobile phones as a guide for museum visitors where the devices themselves perform the object recognition. Measuring human performance has become an important consideration in the VR community [9,10]. Smith and Hart have conducted experiments to evaluate the cognitive load of a user with different wayfinding aids [11]. Kopper et al. [12] have evaluated how users can navigate in multiscale virtual environments. Also for virtual environments, Swan et al [13] have evaluated how users performed with different map based techniques on four different virtual environment setups. With respect to augmented reality, Tönnis et al. [14] have evaluated different visualizations for directing a user’s attention in a car.

2


We have conceived several approaches to support two phases of the picking scenario: wayfinding (going to the correct shelf) and picking (finding the correct item on the shelf). The approaches systematically vary along two orthogonal axes: display devices and various visualization schemes.

1000

B. Schwerdtfeger et al.

Fig. 1. Displays and their possibilities to show a visualisation

Extending the classification by Bimber and Raskar [15], we distinguish between the following display devices: Spatial Ubiquitous Displays are all those which are placed in the environment, such as remote screens, projectors or pick-by-light lamps. Hand-Held Displays refer to mobile devices which are held in the hand, attached to the user’s wrist or mounted on a trolley. Such displays include classical print-outs, tablet-PCs, PDAs or hand-held projectors. Head-Attached Displays most commonly refer to head-mounted displays which present information directly in front of users’ eyes. Several taxonomies of visualization schemes exist [16]. They classify visualizations with respect to criteria such as dimension, functionality, intention of use, etc. For logistics applications, we have developed schemes that show information in different numbers of dimensions. One-dimensional information presentation arranges data items along one axis. In our case, this scheme corresponds to the printed-out lists. Two-dimensional information presentation arranges items in a plane. The plane can be positioned vertically in front of a shelf or horizontally on the floor. Information on the plane can be oriented ego-centrically with the user (compass) or with respect to an exo-centric viewpoint (2D map). Three-dimensional information presentation schemes [10] use three axes. Metaphors vary from World in Miniature (3D map) to the AR-based presentation of symbolic icons. Such icons can be placed as Points on a Path in


1001

world-fixed locations (traffic signs and 3D arrow at the position of the next item) or as Floating Signs in a user-related coordinate frame (3d compass with rubber band - connecting the compass to the next item location). Putting it all together, the aforementioned information presentation options span a spectrum of 3 visualization schemes × 3 types of display devices = 9 options each for presenting wayfinding and picking information to mobile workers in logistics applications. Figure 1 gives an overview of all schemes and indicates which visualization can be displayed on which device. We have implemented and evaluated 11 of these schemes, as indicated by circles in Figure 1 and discussed in section 3.

3

Experimental Environment/System Design

This section gives a short overview of the hardware and software wayfinding and the picking setups for our usability experiments. Options for visualizing information were developed and discussed in an interdisciplinary team of usability and augmented reality experts, psychology students, and logisticians. 3.1

Hardware Setup

Tracking. To provide real-time optical see-through augmented reality and to track user’s locations for evaluation, we used an ART DTrack system with 5 infra-red cameras. The system tracks up to 10 users or objects with submillimeter precision in a tracking area of about 6 × 4 meters. Head-Mounted Display. To evaluate the use of head-attached devices we used a Sony Glasstron HMD connected to a light-weight laptop computer in a backpack. The display provides only monoscopic presentations. In order to avoid wrong depth perceptions, we presented information only to one of the user’s eyes. PDA. We used a Dell Axim X51v PDA for the hand-held presentations. It was attached with a wristband to the participants’ arm. Remote Screens. For the evaluations of spatial displays, we placed 5 monitors next to our shelves. The monitors were connected to stationary computers. 3.2

Software Components

The software system is based on our distributed DWARF middleware [17]. It allowed us to focus system development on components that are central to the underlying research questions while providing only very simple temporary solutions (shortcuts) for other systems components – to be easily replaced with more mature versions when more elaborate industrial setups are considered. Using DWARF, we have implemented the following main services: Visualizations. A different rendering component exists for each display device, configurable for 1D, 2D and 3D visualization schemes via XML descriptions. User Input. System notification after a user has picked an item is currently provided interactively by the supervisor (Wizard of Oz method).

1002


Warehouse Management. A warehouse management system is simulated using pre-defined order lists from XML files. Logging. A logging service writes time stamps to a log file, indicating when a test person starts and completes a task (wayfinding, picking). 3.3

Wayfinding Setup

For the wayfinding experiments, the warehouse consisted of 4 double-sided shelves with 5 × 5 boxes on each side, resulting in a total of 200 boxes (see Fig. 2a). The shelves were numbered sequentially.

Fig. 2. Warehouse setup for the Wayfinding test with a) spatial b) hand-held and c) head-attached displays

Spatial Displays. (Fig. 2a): We implemented one-, two-, and three-dimensional visualizations for spatial displays (Fig. 3). The 1D visualization showed item locations as a textual list on the closest monitor (Fig. 3a). For the 2D visualization, we used an ego-centric you-are-here strip-map (Fig. 3b). As 3D visualization, we implemented the Points on the Path metaphor. We displayed 2D arrows on the screen, yielding a 3D meaning according to the monitor’s position and orientation. The visualization is shown on Fig. 3c. Hand-Held Displays. (Fig. 2b): As for head-attached displays, the 1D visualization showed a text-based list, and the 2D visualization used the you-arehere strip-map metaphor. We decided not to implement and test 3D AR-based visualizations because hand-held displays could not be tracked by our system inbetween the shelves. Head-Attached Displays. (Fig. 2c): The 1D visualization shows a textual list, and the 2D visualization uses the you-are-here strip-map metaphor. This was favored over the compass, as [6] showed good results using the strip-map for picking applications. For the 3D visualization we used the ego-centric Points on the Path metaphor and set up a virtual traffic sign in front of the next shelf (see Fig. 4a+b), using occlusion relationships between the shelves and the sign as an additional depth cue. Since users had trouble finding augmentations within


1003

Fig. 3. Visualization schemes on spatial displays. a) Text-based list (1D): amount, id, shelf, box number (letter = column, digit = row), b) You-are-here strip-map (2D), dot: user position, square: position of the next item, c) Points on a Path (3D).

Fig. 4. Wayfinding scenario. Virtual traffic sign to indicate the next position. a) Seen through a HMD occluded by s real shelf. b) Just the augmentation.

the small field-of-view of our HMD, we added a rubber-band visualization. It consists of a compass-like arrow about one meter in front of the user. It points to the next relevant augmentation and is extended with a rubber band of flexible length. 3.4

Picking Setup

The warehouse for the picking test uses a shelf, 5 boxes wide and 10 boxes high. The subjects stood in front of this shelf during the entire test. We did not expect seeing considerable variations across mobile device types and thus tested this scenario only for the Head-Attached Device. As 1D visualization we again chose the textual list. For the 2D visualization we implemented a map for a vertically oriented shelf, similar to the strip-map. To highlight the box in 3D we implemented a 3D arrow slightly immersing into the box (Fig. 5b). This reinforced the effect of depth perception provoked by the occlusion. As recommended by pre-testers, we added a ball in front of each box (Fig. 5b).

1004


Fig. 5. Picking scenario. a) Arrow indicating the box to pick from and a ball in front of each box. b) The arrow seen through an HMD. c) Arrow with rubber band point at target.

4

Test Performance and Results

We designed two separate experiments for the wayfinding and picking tasks, allowing for a summative evaluation [18]. To obtain significant results from a moderately-sized group of subjects, we then separated the dimension of display devices from the dimension of visualization schemes. We thus made three individual evaluations, one for each display device, using a single independent variable in each test: the visualization scheme. The disadvantage of this procedure is that we did not get a unifying conclusion that compared all concepts. We used 18 subjects (age 23-48) in a Within-Subject Design. As the primary dependent variable, we used the time subjects needed to find a target or to pick an object, respectively. More precisely, we measured Way Time + Dead Time for the wayfinding evaluations and Picking Time + Dead Time for picking evaluations. Additionally, we investigated the number of picking errors the participants made. During the experiments an observer watched the test persons and the displayed information. Furthermore, the subjects were interviewed and had to fill out questionnaires. Our hypothesis in each experiment was that the execution speed of the Wayfinding/Picking tasks increases with the dimension of the visualization, i.e.: users are faster when they are provided with 3D information than with 2D and 1D information. 4.1

Wayfinding

For wayfinding, the participants had to go to specified shelves, always starting from an initial position in the lab. Each basic batch of tests consisted of 3 wayfinding orders listing 6 positions, i.e. 18 operations. Subjects had to run a batch for each display device for each implemented visualization scheme (i.e., 3 + 2 + 3 = 8 batches). Subjects thus were requested to perform 8 × 18 = 144 wayfinding operations. To compensate for learning effects all subjects ran the tests in the same sequence: Head-attached, Hand-held and Spatial Display. We could not measure significant time differences between the 1D list and the 2D you-are-here strip-map, independently of the used display device (top row of


1005

Fig. 6. Measured times in the experiment. Diamond: average, line: median, box: standard deviation.

Fig. 6). However, most subjects said that they preferred the list because a single look sufficed for understanding and remembering the task whereas, with the strip-map, they had to check continuously. Interestingly, they found the map to be confusing on the Head-Attached and on the Hand-Held Display, but not on the Spatial Display. We attributed this to the fact that the map uses an exo-centric reference scheme and our presentation did not provide enough unsymmetric information to help users disambiguate between different principal orientations on mobile displays. On the Spatial Displays, this effect was compensated by the fact that the map could be aligned with orientation of the (permanently fixed) display. But even for the visualization on the Spatial Displays, we could not measure significant differences compared to the list. The 3D visualizations were significantly slower than the 1D and 2D visualizations. Especially the AR visualization on the Head-Attached Display was several seconds slower (Fig. 6). Yet, we observed two interesting effects: Firstly, the subjects were almost 50% faster in the third cycle than in the first. This was not only a learning effect, but also a increasing fascination people felt when experiencing Augmented Reality for the first time. Secondly, subjects clustered into two distinct groups of fast and slow users. There were no average users. These sub-clusters correlated with subjects who already prior exposure to computer games or AR/VR applications and those who did not. 4.2

Picking

The picking task required subjects to pick a specified number of items from the correct box on a shelf. Each basic batch of tests consisted of 3 picking orders listing 10 items. Subjects had to run a batch on the HMD for each visualization. Subjects thus were requested to perform 3 × 30 = 90 picks.

1006


We could not measure significant speed differences between the 1D, 2D and 3D visualizations (lower row in Fig. 6). However, failure rate showed clear differences. Subjects made up to 10 times more mistakes with the AR-based visualizations than with the lists and maps. In most cases, subjects picked items one row too high or too low. This indicates a problem with depth perception using current HMDs. For 2D maps, most subjects noted that they took mental efforts to explicitly count the row and column of the visualization and then read the tags on the real shelves. This means that they transferred the information back from the 2D visualization scheme to the 1D list. Something similar happened with the 3D visualization (see Fig. 5a)). In the interview we found out that subjects divided into two camps: One group used the arrow, while the other group counted the balls (as they already did with the 2D visualization) and then used the tags on the shelf.

5

Discussion and Future Work

This paper presents on first results on comparing visualization schemes for order picking across several types of displays. Thus far, we did not find the perfect mobile visualization to provide robust and fast general guidance for the picking process. We measured that neither the 2D strip map nor the 3D visualization is on average faster for Wayfinding or Picking than the 1D textual list. But separating the subjects into AR/VR experienced and unexperienced users showed that there is a group of users, at least in one of our scenarios, which is faster with an AR system. In future evaluations the subjects will have to execute different task in parallel, such as examining the quality of the picked items while picking/wayfinding. Aside from this, we are looking for a suitable classification for users concerning their amenability to spatial concepts, as needed in AR/VR applications and games. Acknowledgments. We would like to thank D. Walch, R. Reif and J. Boppert for their help in understanding the picking process. Thanks go also to all members of the Fachgebiet Augmented Reality for the feedback towards choosing the visualizations, as well as participants of the experiment. Finally, we thank ART GmbH for providing us with supplementary tracking hardware. This work was supported by the Bayerische Forschungsstiftung within the ForLog project.

References 1. G¨ unthner, W.: Bayerischer Forschungsverbund Supra-adaptive Logistiksysteme (ForLog) (2004) F¨ orderantrag an die Bayerische Forschungsstiftung, M¨ unchen. 2. Gudehus, T.: Logistik: Grundlagen, Strategien, Anwendungen. Springer-Verlag, Berlin, Heidelberg, New York (2005) 3. Raskar, R., Beardsley, P., van Baar, J., Wang, Y., Dietz, P., Lee, J., Leigh, D., Willwacher, T.: RFID lamps: interacting with a self-describing world via photosensing wireless tags and projectors. ACM Trans. Graph. 23(3) (2004) 406–415


1007

4. Butz, A., Schneider, M., Spassova, M.: SearchLight - a lightweight search function for pervasive environments. In: Pervasive 2004. (2004) 351–356 5. M¨ uck, B., H¨ ower, M., Franke, W., Dangelmaier, W.: Augmented reality applications for warehouse logistics. In Abraham, A., Yasuhiko, D., Furuhashi, T., K¨ oppen, M., Azuma, O., Ohsawa, Y., eds.: Soft Computing as Transdisciplinary Sience and Technology - Proceedings of the fourth IEEE International Workshop WSTST’05. Advances in Soft Computing, Springer-Verlag (2005) 1053–1062 6. Dangelmaier, W., M¨ uck, B., H¨ ower, M., Franke, W.: AR support for picking. In: Simulation und Visualisierung 2005, SCS European Publishing House (2005) 7. Reitmayr, G., Schmalstieg, D.: Location based applications for mobile augmented reality. In: Proc. of the fourth Australasian user interface conference. (2003) 65–73 8. F¨ ockler, P., Zeidler, T., Brombach, B., Bruns, E., Bimber, O.: PhoneGuide: Museum guidance supported by on-device object recognition on mobile phones. In: Proc. International Conference on Mobile and Ubiquitous Computing (MUM’05). (2005) 3–10 9. Swan, J., Ellis, S., Adelstein, B.: Conducting human-subject experiments with virtual and augmented reality. In: Proc. IEEE Virtual Reality. (2006) 10. Boman, D., Kruijff, E., Joseph, J., LaViola, J., Poupyrev, I.: 3D User Interfaces, Theory and Practice. Addison-Wesley (2005) 11. Smith, S., Hart, J.: Evaluating distributed cognitive resources for wayfinding in a desktop virtual environment. In: Proc. IEEE Symposium on 3D user interfaces. (2006) 3–10 12. Kopper, R., Ni, T., Bowman, D., Pinho, M.: Design and evaluation of navigation techniques for multiscale virtual environments. In: Proc. IEEE Virtual Reality. (2006) 13. Swan, J., Gabbard, J., Hix, D., Shulman, R., Kim, K.: A comparative study of user performance in a map-based virtual environment. In: Proc. IEEE Virtual Reality. (2003) 259–266 14. T¨ onnis, M., Sandor, C., Lange, C., Klinker, G., Bubb, H.: Experimental evaluation of an augmented reality visualization for directing a car driver’s attention. In: Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR). (2005) 15. Bimber, O., Raskar, R.: Spatial Augmented Reality: Merging Real and Virtual Worlds. A. K. Peters Ltd. (2005) 16. Ellis, S.: A taxonomy of visualization techniques for simulation in production and logistics. Proceedings of the 2003 Winter Simulation Conference (2003) 17. Bauer, M., Br¨ ugge, B., Klinker, G., MacWilliams, A., Reicher, T., Riß, S., Sandor, C., Wagner, M.: Design of a componentbased augmented reality framework. In: International Symposium on Augmented Reality ISAR. (2001) 18. Hix, D., Hartson, H.: Developing User Interfaces: Ensuring Usability through Product & Process. John Wiley and Sons (1993)

Interactive Collaborative Scene Assembly Using AR on Mobile Phones Miroslav Andel, Alexander Petrovski, Anders Henrysson, and Mark Ollila VITA, Linköping University, Sweden [email protected], [email protected], [email protected], [email protected]

Abstract. In this paper we present and evaluate a platform for interactive collaborative face-to-face Augmented Reality using a distributed scene graph on mobile phones. The results of individual actions are viewed on the screen in real-time on every connected phone. We show how multiple collaborators can use consumer mobile camera phones to furnish a room together in an Augmented Reality environment. We have also presented a user case study to investigate how untrained users adopt this novel technology and to study the collaboration between multiple users. The platform is totally independent of a PC server though it is possible to connect a PC client to be used for high quality visualization on a big screen device such as a projector or a plasma display.

1 Introduction Recently, mobile phones have developed into an ideal tool for interaction with virtual objects in Augmented Reality (AR) environments. One particularly interesting area of research is collaboration and manipulation with a shared scene on mobile phones. The small size of a handheld device such as a mobile phone naturally provides tangible interaction possibilities. Tangible interaction means that the user can interact with virtual objects in an AR environment by simply moving the phone. By physically moving the phone, a selected object can be translated in the same direction as the physical movement. Tangible rotation can be used as well, by simply rotating the phone. Tangible interaction provides a grab and move metaphor, which feels natural and is easy to adopt for most users. The Bluetooth technology makes it possible to create a wireless close range network between connected mobile phones. The sufficient transmission speed allows users to share data in real time and thus Bluetooth is feasible for AR collaboration. Today’s AR collaboration systems uses special made equipment such as tracked head mounted seetrough displays connected to a computer. Instead of using expensive laboratory systems easy accessible mobile phones can be utilized. The problems arise when a complex scene is shared between all connected devices. Every user’s action is to be viewed simultaneously on all devices’ screens in real time. The issue is to distribute the data at a sufficient rate of speed. Another issue to be solved is when different users want to manipulate with the same object in the scene. Furthermore the computing capabilities of a mobile phone are limited and floating point operations are emulated and therefore slow. Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1008–1017, 2006. c Springer-Verlag Berlin Heidelberg 2006

Interactive Collaborative Scene Assembly Using AR on Mobile Phones

1009

Fig. 1. Screenshots from a four people collaboration session and an overall view

In this paper we present and evaluate our platform for interactive collaborative faceto-face AR using a distributed scene graph on mobile phones and an example application showing the features of our platform called Collaborative Mobile Augmented Reality (CMAR). We believe that CMAR is the first platform to support multi-user collaborative AR scene assembly on mobile phones. In the next section we review related work. We then describe the CMAR platform and its’ implementations, and a user study of our demo application. Finally we present the results, conclusions and future work.

2 Related Work Rekimoto has shown how several users can interact with the same shared scene with his system called TransVision [8]. The system consisted of two sub parts connected by cables, a display and a computer for graphics computations. The display consists of a LCD TV, a CCD camera, a 3D sensor, and two buttons for interaction. D. Mogilev et al. have extended TransVision into AR Pad [5], which allows users to both view and interact with AR content using a combination of camera, button and trackball input. The first light weight handheld multi collaborative AR system for the general public was the ”the invisible train” [9]. Wagner et al. have created a mobile collaborative multiuser augmented reality PDA game where players control virtual trains on a real wooden miniature railroad track. The players maneuver the trains by using the stylus on the PDA as interaction tool. The goal of the game is to prevent collision between the virtual trains. Nowadays, a decade after TransVision mobile phones are very similar to Rekimoto’s system but are more compact and with Bluetooth short range wireless connectivity standard, cables are unnecessary. Mobile phones have their own CPU with sufficient computing power, which makes external computers needless. This increases the portability and by using optical tracking the mobile phone transforms into an ideal handheld AR device. Several AR applications for camera based phones have been developed but very few collaborative ones. The best known application of this kind is ”AR Tennis” [2]. AR Tennis is a face to face collaborative AR game where two players plays tennis against each other. The players can see a virtual tennis court overlaid on the real marker between

1010

M. Andel et al.

them. The mobile phones are used as rackets and the players interact by physically moving their phones to hit the virtual ball across the net. This application is limited to only two users and does not involve any object manipulation except hitting the ball. Palke et. al. implies in their work ”Foot-based mobile Interaction with Games” [7] that interaction with mobile applications is often awkward due to the limited and miniaturized input modalities available. Their approach exploits the video capabilities of camera equipped smart-phones and PDAs to provide a fun solution for interaction tasks in simple games like ”Pong”, ”Breakout” or soccer. To be able to distribute the data between the devices in a smart manner the following distributed scene graph architectures has been examined. Hesina et al. presents Distributed Open Inventor, an extension to the Open Inventor toolkit [4]. The toolkit is extended with the concept of a distributed shared scene graph, which means that multiple devices can modify or use scene graph data. Distributed Open Inventor is a layer on the original Open Inventor and the programmer is not being forced to change the usual work style because of distribution. Similar ideas have been introduced by Martin Naef et al. with The blue-c Distributed Scene Graph [6] which is an extension to OpenGL Performer.

3 CMAR CMAR is a platform for collaborative mobile augmented reality, which allows several users to interact and manipulate with a shared scene. The platform also contains a PC program called CMAR ViSION (figure 1) to visualize the shared scene in high-quality on a big screen device like a plasma screen or a projector. CMAR is implemented in C++ and OpenGL ES 1.1 based on Henrysson’s ported version of ARToolKit to Symbian Series 60 [2]. The coordinate system convention is the same as in ARToolKit1 and the platform is totally independent of a PC server. 3.1 Scene Graph To be able to manage an advanced scene assembly the scene must by represented in an efficient data structure. The narrow bandwidth of the Bluetooth network makes it unsuitable to resend a whole scene representation to all connected devices in every frame. Therefore we are using a tree representation of the scene, a scene graph. This solution where each node is inheriting properties from its parent reduces the traffic over the network radically. Scene graphs are also simple, familiar and an intuitive environment to use for application developers. The few existing non-commercial distributed scene graph libraries are far too complex to port to the Symbian OS. For that reason, we have created our own simple distributed scene graph which provides easy management of large numbers of objects in the scene. This data structure also makes it easier to interact with and modify the scene. Our scene graph uses OpenGL ES which provides a lowlevel API between software applications and hardware or software graphics engines. When 3D hardware in mobile phones becomes a standard feature, this solution will automatically result in increased rendering speed and rendering quality. Our implementation of the scene graph is simple, compact, extendable and optimized for mobile AR 1

ARToolKit, http://www.hitl.washington.edu/artoolkit


1011

Fig. 2. Example of a CMAR network Table 1. CMAR Bluetooth protocol Id 0 1 2 3 4 5 6 7

Package Send client id (server) Update color Update rotation Update translation Select/deselect object Delete object Insert object Print screen

Parameters package id, client id package id, node id, RGB color package id, node id, XYZ rotation package id, node id, XYZ translation package id, node id, client id, current selection id package id, node id package id, object id, XY position package id

purposes. It consists of following node types; group node, root node, transform node, geometry node, camera node, light node and material node. To be able to modify or edit an arbitrary node, every one has it’s own identification number and can be accessed through an access list. This list constitute the base of how the protocol will access the scene graph data structure and perform updates. 3.2 Bluetooth Network and Protocol The network is a Bluetooth multi point based on BoomslangBT2 , where at least one client is connected to a mobile phone server. In CMAR there are two kinds of clients; mobile phones for interaction with the shared scene and PC clients for a high resolution overall view (figure 2). Every device attached to the network has its own copy of the scene graph and is synchronized by updates from the server through a protocol. To keep the scene graph synchronized during interaction, following steps are performed. 1. The unit(s) who wants to modify the scene graph sends a request to the server 2. If the server approves the modifications it broadcasts the update information to all attached devices including itself 3. All units receive and commit the modifications To achieve this, a protocol must be designed. Our simple binary protocol contains a number of packages that can be sent over the network (table 1). When the server connects to a client package 0 is sent, telling the unit which id it has and in that way its’ 2

BoomslangBT, http://www.boomslangstudios.com

1012

M. Andel et al.

selection color. The coordinates in the protocol corresponds to the coordinate system center on the marker. A user can only select one object at a time and this is assured by the protocol. If the user selects an object that is already selected by the same user, the object will become deselected. If the user has an object selected and selects another the first object will become deselected and the second selected. If no object is selected the current selection id is equal to -1. The last package in the protocol makes it possible to simultaneously capture screenshots on all attached mobile phones to the network. 3.3 Selection and Interaction To select an object, instead of calculating distances from the camera vector and check for object intersections, it is possible to store object id in the alpha channel. By backtracking the alpha channel value of the pixel in the crosshair’s center of the framebuffer, it is possible to immediately obtain the selected object which is fast and requires no calculations. It is possible to have an alpha value on an object without having any transparency. Figure 3 illustrates one frame with three opaque objects and corresponding alpha channel. When an object is selected a transparent bounding box appears around the selected object with a predefined user dependent color. For interaction, based on Henryssons et al. user case study [3] we have chosen tangible interaction for the translation, joypad input for the rotation and key based interaction for all other functions. Tangible translation means in practice that a selected object can be moved by physically moving the phone in any direction.

Fig. 3. Picking in the alpha channel, the RGB frame to the left and the corresponding alpha channel to the right

3.4 PC Client - CMAR ViSION The purpose of the PC client CMAR ViSION3 is to be used as a group display for visualization, giving an overall view of the scene on a big screen device. The only form of interaction is that the user can change the view of the real-time high-quality rendering. Therefore no camera is connected and thus the rendering contains no AR, only pure computer graphics. CMAR ViSION uses an almost unchanged version of the mobile phone scene graph except for minor changes which regards only data type substitutions and that it uses standard OpenGL instead of OpenGL ES. The functionality and syntax of the scene graph is still the same as for the mobile phone version. To achieve different quality on the renderings of the two types of clients while using the same scene graph, 3

It is possible to connect several PC clients to the network. The total number of clients including mobile phone clients should not exceed eight though.


1013

two different levels of detail of the objects are created. For the mobile phone client rendering speed is important and having objects with high amount of triangles, will lead to a significant decrease in rendering speed. To best preserve appearance of the 3D objects, a mesh simplification algorithm called ”Surface simplification using quadric error metrics” [1] is utilized by CMAR ModConverter4.

4 User Study The evaluation was carried out in a standard conference room where the participants could communicate without disturbance during the session. The usability and experience of the user collaboration and the CMAR platform were assessed with an individual user evaluation. The purpose with this study was to elucidate characteristics of the platform limitations and advantages and to receive an insight of how this kind of interactive collaboration is adopted. 4.1 Demo Application Our demo application that we based our user study on consists of nine pieces of furniture that can be moved freely. The virtual crosshair in the center of the screen on each mobile phone is used for selection. Selection (or deselection) is done by pressing the keypad button ’5’ when the object is in the crosshair. As indication that the object is selected, a transparent box around the object is visible with each of the collaborators specific and individual color. After selection, moving the selected object is performed by holding down the keypad button ’2’ and moving the phone physically (tangible translation) in the desired direction. The selected object is then fixed to the crosshair until the button is released. It is also possible to rotate the objects around their z-axis by pressing right or left on the joypad. 4.2 Subjects Twenty students from Linköping University, eight males and twelve females, took part in the evaluation. Their age ranged from 22 to 28, averaging at 25. The group sizes vary from two to four collaborators and the test users had technical as well as non-technical background. To test how much experience our test users have of interaction combined with graphics the following question was asked: ”How often do you play games on your mobile phone?” The majority of the test users played games more then once a year and no one played more then once a month. This was good since we wanted to focus our user study based on mobile phone users with basic knowledge. 4.3 Hardware and Setup The hardware consisted of three Nokia 6630 and two Nokia 6680 running Symbian OS v8.0 (Series 60 Feature Pack 2) that ships with a software implementation of OpenGL ES. They have a 220 MHz ARM processor and an integrated 1.3 megapixel camera. 4

This tool is a part of our platform and simplifies OBJ and PLY models to our own compact binary format supported by the scene graph.

1014

M. Andel et al.

Fig. 4. The test setup and a group of four persons collaboration session with the group display to the right

The screen size is 178 x 208 pixels and the video capture resolution is 160x120 pixels. The notebook is an Acer TravelMate 8102wlmi running Windows Xp Pro Service Pack 2 with an ATI Radeon X700 graphics card and integrated Bluetooth. As group display a standard DLP projector was connected to the notebook to visualize the overall view. The test users sat at a table with the projector screen to the left of the table (figure 4). 4.4 Design and Procedure The test users were given instructions over the functionality of the application. By letting them play around, we could actually listen and make records of their very first experience with the furnishing program. As the test users got familiar with the application (usually after a few minutes) a task was given with no time limitations. The task was to furnish their virtual apartment from a random state to a predefined state. A paper printout with the predefined state of the furniture was placed next to the marker. After completing the task the test users completed a questionnaire addressing various aspects of the perceived usability and user experience. The questions in the questionnaire (table 2) had all graded answers ranging from one to five. 4.5 User Comments The test user’s experienced the furnishing application to be fun to use and to collaborate. Some thought that it was a bit tricky to get the furniture to the exact position (as the predefined design planning) by only watching the mobile phone screen, since the tracking is not completely accurate. One test user said: ”With conventional design tools as paper and pen it would not be possible to obtain the 3D effects”. People in the test group liked the idea to use the mobile phone as a collaborative tool, where they could see other users moving pieces of furniture in real-time. All test persons found the tool intuitive and very easy to use. Almost all of the users found the big screen to be very useful to get better overall view and the virtual walk through (by pan, zoom and rotate the scene) provided them very good visual feedback. Some users found it frustrating when they moved the phone too far and lost their tracking. When the phone is far away from the marker, the whole scene starts to wobble due to inaccurate tracking. One user actually experienced slightly cyber sickness.


1015

Table 2. Question protocol Q1 How easy was it to move the furniture to the desired position? A1 1: Very difficult, 5: Very easy Q2 How did the cooperation worked between the collaborators? A2 1: Very bad, 5: Very good Q3 How important is the oral communication in the collaboration? A3 1: Not important, 5: Very important Q4 How would more users affect the effectivity? A4 1: Not at all, 5: Very much Q5 How much did the projector help to place the furniture? A5 1: No help at all, 5: A lot of help

5,0 4,0 3,0 2,0 1,0 0,0 Q1

Q2

Q3

Q4

Q5

Fig. 5. The compilation of the test users answers to the questionnaire. The mean value corresponding to each question is displayed.

4.6 Results For brevity we just summarize some of the main results: all groups were able to complete the task successfully. Almost all of the users thought it was easy to move the furniture to desired position and more than 60% thought that the projector with CMAR ViSION gave a lot of help when positioning. The collaboration between the users worked out well even though the oral communication was not found to be important. Most of the test users agreed that it would be possible to accomplish the task without speaking, but it would probably take more time. The test users also concluded that the effectiveness would be better if their group size was bigger. The main advantage with CMAR ViSION is the help in positioning that the group display gives. The main limitation of the platform is the inaccurate tracking. For a more quantitative summary of the survey see figure 5. The tracking accuracy of CMAR could be improved and applications where quick movements are used may create irritation or work bad. Good tracking is a very good start and more markers will increase the working area. The limited screen resolution is in this case helped out by the group display, which provides high quality graphics in real-time. It would be smart to make use of the vibrator to notify collisions between objects. Sound notifications could also be for the same purpose. The physical manipulation of the phone as an interaction tool is very natural and provides an intuitive approach.

1016

M. Andel et al.

5 Conclusions and Future Work We present a novel implementation of a distributed scene graph structure for interactive collaborative Augmented Reality on mobile phones. In our work we consider the phone to be a tangible interaction tool and use the motion of the phone to translate objects. The platform gives the possibility to create a collaborative Bluetooth network containing up to eight mobile devices. If high quality real time rendering is needed, it is also possible to connect one or more PC applications to the Bluetooth network of phones for pure visualization. This platform is totally independent of a PC server and makes it therefore portable. We also presented a user study of a furnishing application that is based on our platform for group interaction on mobile phones which includes Augmented Reality, 3D graphics and tangible 3D translation. The target group of this platform are the general public with no special training. The essential equipment to run CMAR is existing consumer camera mobile phones and a marker. This makes the technology accessible and cheap. The conclusion of this user study is that untrained users adopt the technology fast and in an intuitive way. They quickly took on the task and the collaboration worked out well and the test users thought that more collaborators will lead to a faster accomplishment of their task. It was easy to move the furniture but sometimes difficult to get the furniture on exact position. The big screen visualisation was found to be helpful in many situations and could be used to walk around in the apartment. For future work it would be interesting to compare CMAR to similar collaborative scene assembly systems on the PC platform. We would also investigate how the tracking could be improved for accuracy. Furthermore, the possibilities to save a scene to file for later collaboration sessions should be explored. How to use collision handling and physics with tangible translation needs to be investigated. In this case haptic feedback through the phone vibrator would be a nice feature to add as collision indicator. Implementing a snap function that aligns the furniture to walls and corners would help users to place the furniture faster and more accurate. Obviously we would also improve the platform according to the comments of the user study to elucidate platform drawbacks. At this moment, the platform does not allow late joining, which could be implemented by requesting a complete scene graph update from the server, and have a buffer to guarantee no pending updates are missed. This feature would also require an Ad Hoc Bluetooth network which could be solved by not letting the phone to stop searching for new CMAR running devices. Later on we have discovered that it can be a bit confusing for users to find out which selection color they have. This can be solved with a legend in the GUI. We should also consider the color choices, the server selection color is white and it can be hard to se a white selected object on white background. It would also be neat to continuously broadcast all camera positions in the local coordinate system to CMAR ViSION, where a grid of all users’ camera views could be viewed.

References 1. M. Garland and P. S. Heckbert. Surface simplification using quadric error metrics. In SIGGRAPH, pages 209–216, 1997. 2. A. Henrysson, M. Billinghurst, and M. Ollila. Face to face collaborative ar on mobile phones. In ISMAR, pages 80–89, 2005.


1017

3. A. Henrysson, M. Billinghurst, and M. Ollila. Virtual object manipulation using a mobile phone. In ICAT 2005: of the 15th International Conference on Artificial Reality and Telexistence, pages 164–171, 2005. 4. G. Hesina, D. Schmalstieg, A. Furhmann, and W. Purgathofer. Distributed open inventor: a practical approach to distributed 3d graphics. In VRST ’99: Proceedings of the ACM symposium on Virtual reality software and technology, pages 74–81, New York, NY, USA, 1999. ACM Press. 5. D. Mogilev, K. Kiyokawa, M. Billinghurst, and J. Pair. Ar pad: An interface for face-to-face ar collaboration, 2002. 6. M. Naef, E. Lamboray, O. Staadt, and M. Gross. The blue-c distributed scene graph. In EGVE ’03: Proceedings of the workshop on Virtual environments 2003, pages 125–133, New York, NY, USA, 2003. ACM Press. 7. V. Paelke, C. Reimann, and D. Stichling. Foot-based mobile interaction with games. In ACE ’04: Proceedings of the 2004 ACM SIGCHI International Conference on Advances in computer entertainment technology, pages 321–324, New York, NY, USA, 2004. ACM Press. 8. J. Rekimoto. Transvision: A hand-held augmented reality system for collaborative design. In Proceedings of Virtual Systems and Multimedia (VSMM) ’96, 1996. 9. D. Wagner, T. Pintaric, F. Ledermann, and D. Schmalstieg. Towards massively multi-user augmented reality on handheld devices. In Third International Conference on Pervasive Computing (Pervasive 2005), Munich, Germany, may 2005.

Design of the Multimedia Communication Protocol and System for Wireless LAN Qingzhang Chen1,2, Jianghong Han1, and Keji Mao2 1

College of Computer and Information, Hefei University of Technology Hefei ,Anhui, China [email protected] 2 College of Information Engineering, Zhejiang University of Technology Hangzhou, Zhejiang, China

Abstract. The protocol of wireless LAN —IEEE 802.11 put forward DCF/PCF to offer the real time transmission, but it did not meet the need of multimedia transmission, so a new MAC protocol is developed to solve this problem. In this paper we discuss the way of implementing and processing the multimedia data streams in WLAN. We regard the transmission of multimedia as multi-channel accessing with synchronization to study it. We start from the parameter QoS and mapped the multimedia transmission data streams to the different priorities, and use negotiation approach to allocate the resource.

1 Introduction 1.1 Wireless LAN Now we are in the age of communication revolution. In the recent decades, traditional wireless network system developed very quickly, and many kinds new technology appeared, just like wireless LAN. In order to give users a better wireless communication service, IEEE draft out the protocol of 802.11. The 802.11 MAC protocol supports two kinds of operation modes, PCF and DCF. PCF provides an entering mode that can avoid the competition, and DCF adopts a kind of CSMA that also can avoid conflicting based on entering competition. These two modes can trade off, that means a DCF competitive period follows a PCF noncompetitive period. 1.2 Multimedia Communication IEEE 802.11 use RTS/CTS to avoid collision, but the duration of RTS and CTS are same in length, so only the problem of base-to-peer hidden terminal can be solved, but to the transmission shape of multipeer, it can not solve the problem of hidden terminal and collision. Besides, IEEE 802.11 thought that PCF/DCF could provide the real transmission in some degree, but it didn’t equal to the character of multimedia transmission, so a new kind of MAC protocol is also needed. Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1018 – 1025, 2006. © Springer-Verlag Berlin Heidelberg 2006

Design of the Multimedia Communication Protocol and System for Wireless LAN

1019

1.3 Project Aim 1.3.1 Putting Forward a Kind of Multimedia Transmission Protocol to Apply the MAC in the Wireless LAN In this paper we regard the transmission of multimedia as multi-channel accessing with synchronization to study it, and in every contention period one multimedia transmission competition by one group that maybe formed by many members. The socalled multi-channel access with synchronization means that a sender needs one or more contention free transmission authority for channel before transmitting, and transmitting different streams by these channels synchronously, every stream need a different transmission time. 1.3.2 The Design of QoS The way of TDMA, FDMA and CDMA are all belong to static allocation methods. They should be improved to meet the need of multimedia transmission of QoS. We start from the parameter QoS and mapped the multimedia transmission data streams to the different priorities. As soon as priority model come into existence, we will use negotiation in the admission and resource management of streams handover. When high priority requests enter the system, and there is not enough resource to meet the need of new requests, we will give some new resource to the high priority requests by lowing low priority users’ service quality.

2 The Design of System 2.1 Overview Actually we also need driver and firmware. Drivers offer service to the system to control the wireless LAN adapter. Firmware takes change to the practice process of the wireless LAN adapter, including the carrier wave and sense in physics and receiving and transmitting data, and also provides service to drivers. 2.2 Firmware In firmware, after initialization we should wait a new event, for the convenience of implementing, we use an endless circulation to poll every queue, check the happen of event and call for processing it. 2.3 Drivers In drivers, we use windows operating system as platform, and program the drivers of wireless network according to NDIS (Network Driver Interface Specification). 2.4 Multimedia Service 2.4.1 The IEEE 802.11 Standard of Real Time Service Besides using the technology of CDMA/CA offer the basic visiting control, IEEE 802.11 also supports the Point coordination function (PCF for short).

1020

Q. Chen, J. Han, and K. Mao

The main principle of the PCF is establishing a polling mechanism, PC (Point Coordinator) will poll by a polling list on which recorded all the workstation that need to transmit real time data, and the workstation which have been polled can transport frame. According to the standard of IEEE 802.11, the polling list that PCF maintained are not compelled, and the polling list only know which workstation can be polled, it doesn’t give the information of that how many real time data that the workstation needed to be transmitted. Workstation also cannot dynamically request the time and length for polling, every workstation have one frame time. Allotting the resource on average so side by side cannot reflect the real request. 2.4.2 Processing the Multimedia Data According to the character of multimedia data, we divide the emergency of the data into three types in the rough, there are real time data, harmonizing data and random data. According different request, every workstation can put forward specific request to the three types. In order to describe the parameter QoS in quantity, we use smooth model to describe the problem, to the general situation that there may be several workstations, we put forward the following expression. U={ui| i∈N} ui means the workstation I. U means the all workstations that in the same infrastructure. Every workstation should obey the smooth model, and according this put forward the parameter of the data streams. The four parameter of the smooth model are {ni, Ti, Pi, Wi} ni means when the workstation I wants to establish a multimedia transmission, the quantity of the multimedia transmission data streams that it needs. Ti means the data streams ni of workstation I belongs to which type of data. {tij| j=1, 2, 3, …, ni ， tij∈ stream type} Pi means the data transmission jitter ni of the workstation I. {pij| j=1, 2, 3, …, ni, j pi ∈ N}。 Wi means the max. and the allowed min. that the bit rate of the data stream needed of the workstation I. {(xij, yij )| j=1, 2, 3, …, ni }. According to the smooth model, {ni, Ti, Pi, Wi} means the demand of one workstation that wants to establish realtime multimedia. For example, the basic demands of online movie are video, sound and letter. So you must put forward the request parameter when you are establishing the online service, the smooth model maybe expressed like this {3, {1, 2, 3}, {1, 3, 7}, {{150}, {20, 13}, {NULL}}}. 2.4.3 Analysis Before the establishing of every online multimedia, the workstation will put forward the requested parameter that depend on the fore-mentioned model, then checked and affirmed by IBSS PC. Because of the hardware limit of the physics, we must use a fair and dynamic allot principle to the parameter request of users. The principle of allocating is adopting a kind of admission control testing model. So we have to analyze the behavior of three different kinds of streams in PCF.


1021

2.4.3.1 Real Time Data. If we observe the whole transmission program IEEE 802.11 from the transmission period, and add the time of DCF to PCF regarded as one period of transmission, we call it super-frame (DCF time + PCF time= K, K∈ constant). If we look one super-frame as QoS transmission unit, on the base of period, we also can quantify the parameter of jitter as period parameter in order to translate the request of jitter into synchronic mechanism. If we can look jitter as bit rate guaranteed, lingering reference variable; if we look the period as the basic unit of transmission, jitter means the allowed period parameter that can meet the requirement. That is to say, if the period counted by jitter, and the finish of period also is the synchronic point of the stream. Through the period transition, the real time data stream can be quantified to the number of frames that must be transmitted in every period. And in every period must be a PCF period in order to insure the certain number of frames should be transmitted. Admission test should fetch the smallest common factor from the quantified period of all period parameter, and then looks the min. common multiply as the longest period. All the allowed bandwidth limited streams in the longest period and also accord with their jitter request can get reservation. A simple math expression like this, Order L= LCM({Pij}) Σnij=0(Wij/L)+ DCF time< 1 Get Σnij=0( Wij /L)< 1- DCF time(= max of PCF period time). After meeting the need of bandwidth, the PRI of transmission adopts prescheduling on demand. In advance, we maintain the PCF polling list by jitter to decide every time unit save to which stream to transmit a frame. On every time point we directly appoint the use attach of channel, and it doesn’t have to be counted again. The keystone of scheduling mechanism is maintaining the polling list, and polling list must record the transmission order, destination and data and so on of every stream in every big period in detail. Every time there is a new request of online, it must find a transmission time from the polling list and does not affect all the online quality. 2.4.3.2 Harmonizing Data. Unlike the real time data, this is a data stream that the requirement of the bandwidth are variable depends on the network under allowed range. Without prejudice to the basic requirement, adjusting the use of bandwidth, but it also need PCF time, so admission control model like the real time data streams model, only at the time of scheduling, can consider the limit of bandwidth, to seek the time point and allowed transmission quantity accord with the parameter jitter. 2.4.3.3 Random op Data. Random data has no time emergency, so there is no jitter as period parameter, and also it dose not need admission control and scheduling. The CDMA/CA mechanism which operating in DCF period offers the request of random data. Here we do not offer the efficiency evaluation, only suppose under the request of plentiful bandwidth remained, PCF can engross the longest time, and the time of DCF must meet the need of transmitting the longest frame, such shortest DCF time including DIFS、 RTS/CTS、 SIFS、 max frame length and ACK time.

1022


3 Implement 3.1 Implement the Simple PCF Mechanism In general wireless LAN adapter, only the DCF transmission mechanism realized, to the PCF are mostly not. In order to implementing the multimedia corresponding mechanism we must add PCF model by ourselves, especially, about the usage of beacon frame, probe response frame in CF-parameter, and also the minuteness of polling list and the running mechanism of poll must be finished. Another keystone is the design of point coordinator, for the sake of convenience, we adopt Ad-hoc network frame, and use one IBSS Creator as point coordinator to imitate the centralized control model of access point. Under such a principle and subordinate frame, point coordinator shoulders the work of admission control、 schedule and poll.

Element ID

Byte: 1

Length

1

CFP Count CFP Period

1

1

CFP MaxDuration

CFP DurRemaining

2

2

Fig. 1. CF Parameter Set element format

From figure 1 we can observe the every parameter of CF Parameter, and the fixation of CFP Period and CFP Max. Duration separately express how many beacon period as one CFP period and the maximums allowable duration of one CFP period. Deciding the two number properly can help to the efficiency of the whole mechanism to the bandwidth. Beacon use CFP Count to declare the beginning of PCF after several beacon periods, CFP DurRemaining descended timely the parameter of beacon frame transmitted by DCF period to zero. The above CFP parameters mentioned are all decided by point coordinator. And use beacon timer to timing transport beacon frame. On one hand, all the mobile stations timing synchronization, also can harmonize the subrogation of DCF and PCF. Before the beginning of PCF period, we must decide the polling list. If it involves admission control and scheduling, it will wastes more time to count time and memory size, so it is dealt with by drivers and not realized by the firmware on the network adapter. The firmware process of point coordinator only charge to affirm the object that drivers wanted to be transmitted or the object of polled transmission data to transmit in time. Common workstations are in passive situation, only when they received the frame of poll, they can transmit the data, or when they received data frame, they can response ack frame. In the whole period of PCF, it use piggyback to transmit the messages of poll and ack to save the bandwidth. Besides the transmission of frame, CFP also have to cooperate null-function frame to use in the condition of that there is no data frame to be transmitted, and the messages of piggyback should be transported.


1023

3.2 Offering QoS Service 3.2.1 Admission Control The quality of the service is very important to the high quality video, sound and the multimedia applied in WLAN. IEEE 802.11 standard has no criterion to the request of the avoid-competition bandwidth, that is to say, how to let point coordinator know which user should what time to transmit the data? If there is only have a bandwidth remained non-competition mechanism, and there is no chance to use it. It is also useless. In order make good use of PCF period, we should have a seeking way to the bandwidth, and let every bandwidth user who want to the bandwidth take part in the competition fairly, it will not lose the bandwidth remained mechanism because of the competition. Obviously, we remain enough bandwidth to the PCF period, and the use the remainder DCF period to deal with the reservation request. But IEEE 802.11 standard did not define such request/response frame. Fortunately, the frame type that management frame defined did not use up, and the remainder used to enlarge the type. We can dauntlessly use these two to define the offered frame type, and we call QoS request frame and QoS response frame, permeating the information exchange of these two groups to obtain the remained bandwidth mechanism. The frame type of QoS request frame / response frame is also management frame, but sub-type (4 bit ) separately use 1110 and 1111, that is the whole field of frame control separately add 0x00E0 and 0x00F0, and the results as these two type of frame type. The information of the body of QoS request frame are the QoS parameter of the data streams, and at most n data streams can be allowed. The information of the body of QoS request frame is very easy, response to the request whether allowed of not, when it is allowed it can be allocated several data streams’ ID. 3.2.2 The Maintenance of Admission Test and Polling Test According to the traditional IEEE 802.11 standard, the speed of wireless LAN adapter has two types, 1Mbps and 2Mbps. But on the implement of the network adapter, 1Mbps is used and 2Mbps is not for insurance and the validity of the data. Millions of bits can be transmitted in one second, and the longest frame is 2304 bits, every five beacon periods as one new start of PCF period, every beacon period transmitted in 100 microsecond. So the period of PCF is 5x105μs, if every second transmit 106bits, every period of PCF can hold 217 longest frames, except the DCF period time, and the time of the IFS and back-off. The efficiency of 200 frames/PCF period is prospective. 100kbps= 105 bits/106 μs = 5x104 bits/5x105 μs = 21 MPDU/1 PCF To the request of users’ bit rate, we can change it into the number of frame that every non-competition period should be transmitted. This the fact speed contrast to the request of the users.

1024


The change of time parameter according to the PCF period, every PCF period less than one second, so we suppose every time parameter as one, means every PCF period as one synchronization, if the point of synchronization can meet the number of transmitted frame means QoS is satisfied. For example, bit rate= 100kps, jitter= 2 come into being in the MAC, the time that the first frame begin to transmit to the 42th frame finished transmitting must be in the two period of PCF. Implementing the QoS on the wireless LAN adapter to maintain the link list of array of polling list, PollingListTab[1]、 PollingListTab[2]…to PollingListTab[n] means the time synchronization period of 1,2,to n data streams, and clustered by the way of link –list. In every node’s information, there are the number of frames to be transmitted and the total number to be transmitted and destination address and data stream model and synchronous remained time. Every time deal with the request of the bandwidth should according to this table to count. 3.2.3 Sequence Every time IBSS PC transmit the beacon frame, it will check the polling list table according to the remained CFP duration. The frame has smaller period has higher priority, one by one in the same period and the workstation that has more frames need to be transmitted has higher priority.

4 Conclusions About the development of the wireless LAN adapter, firmware and drivers are the closed two parts, if you want to develop the wireless LAN multimedia communication, you must firmly grasp these two parts, and experience be sufficient. On the implement of multimedia, the core of a new operating system has been support in some degree, such as Windows 2000 /XP, I believe that it will bring great help to the development of the wireless multimedia communication.

Acknowledgements The Project Supported by Zhejiang Provincial Science Foundation of China (Y105388).

References 1. Sunghyun Choi and King G. Shin, A cellular wireless local area network with Qos guarntees for heterogeneous traffic, Mobile Networks and Applications 3 (1998)89-100 . 2. Andrew Muir and J.J. Garcia-Luna-Aceves, An efficient packet sensing MAC protocol for wireless networks, Mobile Networks and Applications 3 (1998) 221-234 . 3. Andrew Muir and J.J Garcia-Luna-Aceves, Group Allocation Multiple Access with Collision Detection, Computer Engineering Department University of California Santa Crus, CA 95064 . 4. Macro Conti, Claudio Demaria and Lorenzo Donatiello, Design and performance evaluation of a MAC protocol for wireless local area networks, Mobile Networks and Applications 2 (1997) 69-87 .


1025

5. Paramvir Bahl, Imrich Chlamtac and Andras Farago, Optimizing Resource Utilization in Wireless Multimedia Networks, 1997 IEEE . 6. Cheng-Shang Chang, Member, IEEE, Kwang-Cheng Chen, Member, IEEE, Ming-Young You, and Jin-Fu Chang, Fellow, IEEE, Guaranteed Quality-of-Service Wireless Access to ATM Networks, IEEE JOURNALON SELECTED AREAS IN COMMUNICATIONS . VOL. 15. NO. 1. JANUARY 1997 . 7. Chane L. Fullmer and J.J. Garcia-Luna-Aceves, Complete Single-Channel Solutions to Hidden Terminal Problems in Wireless LANs, Computer Engineering Department Unitersity of California, Santa Crus, California 95064 8. Brocha Epstein and Mischa Schwartz, Reservation Strategies for Multi-Media Traffic in a Wireless Environment, 1995 IEEE . 9. Part11 : Wireless Lan Medium Access Control(MAC) and Physical Layer(PHY) specifications, LAN MAN Standards Committee of the IEEE computer Society . 10. AMD, Wireless LAN DSSS PC Card Reference Design Application Note, application note . 11. Microsoft Driver Develop Kit. Microsoft, 1999 .

Representation of Color in Ancient Noh Masks Xin Yin1, Yasuo Miichi2, and Hiromi T.Tanaka1 1 Ritsumeikan University, Nojihigashi 1-1-1, Kusatsu, Shiga, 5258577 Japan 2 Noh Mask Maker

Abstract. Noh is a traditional Japanese dramatic art. The color of the Noh mask constantly changes in appearance, because of natural factors. In this paper, a digital technique for representing the original color of Noh masks is introduced. At first, an image which shows the normal on surface is generated from the constructed 3D Noh mask mesh. And then, the color of the Noh mask is updated by reference to the color of the surrounding points. As the result, the original color is obtained. Finally, the implementation is carried out using a current Noh mask and an ancient Noh mask. The technique introduced in this paper can also be used to represent the original color in other 3D objects. Keywords: Representation, 3D object, Noh mask, Color, Retinex algorithm.

1 Introduction Objects of cultural assets which are exposed to the natural environment can undergo changes in appearance. The original colors may fade and some parts of the object may be lost. For this reason, much research has been undertaken into the preservation of cultural assets. Various digital representation techniques have been developed for preservation of cultural assets.

Fig. 1. Noh masks


Representation of Color in Ancient Noh Masks

1027

Noh is a famous Japanese traditional drama in which the faces of the players are covered with masks. Noh masks are vulnerable to environmental weathering which can result in fading and loss of the surface pigments. Two Noh masks are shown in the Fig. 1. The left mask is made about thirty years ago and the right one is made about three hundreds years ago. The representation of original color in Noh masks is a challenging area of research.

2 Previous Work Long ago, the restoration of cultural objects was based on the rich knowledge and experience of craftsmen. More recently, digital techniques using image processing software have been developed, but like the craftsman of former times, the operator applies his or her knowledge and experience in restoration the color of the object. One example of this type of technique is [3], by which the color of an ancient “hell scroll” was represented perfectly. In addition, some color representation techniques based on computer graphics (CG), such as [2], have been developed. In this technique, the three-dimensional (3D) shape of the object is measured by a machine. Next, the artist draws pictures based on the historical record, which are then mapped onto the surface of the 3D model by computer. The color of an ancient statue was reproduced using this technique. This technique allows manual adjustment of detail based on the experience of the operator. But these techniques pain the color on the surface of object and lost some real information of color on the surface. Automatic or semi-automatic color-representation techniques have also been developed. Reference [8] used a technique which makes use of color information remaining on the object. A color that is lost may be reproduced by reference to the color information surrounding the missing section. Reference [7] developed a system for representing the color base on the experience of the exporters. Reference [1] introduced some digital techniques which can clean the surface and repair the crack pattern on the objects. And [4] developed an algorithm, based on the ability of human eyes to adjust to environmental lighting conditions, for removing the influence of dirt on computer simulations of ancient drawings. This technique has mainly been used in the color representation of oil paintings. These techniques mainly represent the color on 2D images. We propose a technique for representing the color on 3D objects in this paper. Usually, for representing the color in the culture assets, the investigation of the pigment on the surface and segmenting the pigment field is needed. But there are some problems are not solved well. The same pigment can show different color if the pain method is different. As this reason, it is very difficult to understand original color in the culture assets even we know what type pigment is used. By another hand, segmenting the pigment field well is not easy. The segment technique is developing now. To avoid these problems, we develop a technique to represent the color based on the information of pigment on the 3D surface directly. The color variation process is different, the craftsmen always represent the color by referencing the color where is look new on the surface. This means that the different color on the surface implies the some rules of color variation process. If these color information on the surface is used well, the original color can be represented. The

1028

X. Yin, Y. Miichi, and H.T. Tanaka

Measure 3D data Get normal information

Get color information

Sample the color of surrounding point Update the color of target point Get results with original color Fig. 2. Computer process

technique represented in this paper needs not segmenting the pigment field detail. The point where the color need updated is called target point, and the surrounding points are called sample points. Shown as Fig. 2, the first step was measurement of the 3D data and color of the Noh mask. Next, we sample the variation in color over the 3D surface based on the surface normal variation, then, the color of the Noh mask is represented using a digital algorithm. Finally we show some implementing results and discuss this technique.

3 Background Knowledge on Noh Masks Noh masks are made from wood on which a design is painted ([5]). In creating a Noh mask, the first step is to carve and polish the wood to provide detail, so that a shape similar to that of the human face is obtained. Next, pigments are painted onto the wooden mask. As shown in Fig. 3a, the Noh mask has two surfaces, the back and the front. The back surface is nearest to the face of the player, and is painted uniformly because it cannot be seen. The front surface is painted with various types of pigments. There are two types of coating: the undercoat and the topcoat. The undercoat is painted in a uniform color and represents the color of the skin. The topcoat is painted in different colors to show the hair, eyes, mouth and so on. Finally, certain other pigments are applied to the surface so that the Noh mask can be seen well on the stage. A glue is used to cause the pigments to adhere to the wood. Light passes through the glue and is reflected outward by pigment particles. As shown in Fig. 3b, when the glue deteriorates, some of the pigment particles are lost, with the result that the formerly smooth surface of the mask becomes rough and decreases in brightness. If the deterioration extends deep into the mask, some areas of pigment are lost completely, and the color of the mask cannot be seen. The pigment particles are mixed slowly and the surface become dark. One subject of representing the color is taking off this dark efficiency.


1029

Fig. 3. Pigments of Noh mask

4 Representation of Noh Mask Original Color The techniques used in ancient times for fashioning and painting Noh masks are not well understood. In addition, the variation of pigments is a complex process. For these reasons, it is very difficult to represent color completely. The key idea for representing the color in this paper comes from [4] which use Retinex (come from “retina” and “cortex”) algorithm ([6]) to represent the color on 2D image. To update the color of a target point, the Retinex algorithm uses the color information found in the vicinity of the point. The influence of one pixel on another varies according to the distance between them. Similar to the Retinex algorithm, we also use the color information found in the vicinity. Then compute the original color based on this information. The detail will be introduced as follow.

(a)

(b) Fig. 4. Measure 3D mesh and color

1030


4.1 Measure 3D Mesh and Color Because there are some techniques can measure and construct 3D mesh of object well, we use these techniques directly. The 3D surface data were obtained using a 3D object measurement machine (Vivid 910). The data got by the Vivid 910 are about thirty patches. Then use exist software to compose these patches together and fill the hole on surface. Fig. 4a shows the final 3D mesh surface. We will use the 3D mesh surface to calculate the image which shows the normal information on 3D surface in the section 4.2. Because the Noh mask is a 3D object, the normal of the 3D object can cause color variations when the color of the mask is measured; the same color may show a different value if the view point is different. For example, there are some highlight and shadow when take photo of a mask shown as Fig. 1. To avoid the highlight effect, the mask is put into a white dome. In this white dome, the light is diffused and there is not highlight on the mask when take the mask photo. One photo taken in this dome is shown in Fig. 4b. Since the mesh is 3D and the color image is 2D, the registration process between the 3D mesh and the 2D image is needed. Three reference points are used. Two points B C θ E A

Fig. 5. Sample route

are the center of the eyes. One point is the center of the mouth. Then the 3D mesh is rotated and scaled until these three points are match between the mesh and the color image. The result is shown in Fig. 4a and 4b. The mesh and the image are seen from same view point. 4.2 Sample on 3D Surface As mentioned before, when update a point (called target point) color, the around point (called sample point) color information is used. To get uniform samples, a sample rout shown as Fig.5 is used. The color of point A is updated based on the colors of the surrounding points (circle points). Reference [9] has introduced the concept of aesthetic curves. We assume that these curves are beautiful because color information


1031

obtained as along with these curves are uniform. To decrease the compute cost, the expression of the curves is simplify and is represented in polar coordinates as follow:

⎧ R 2 = a • R1 . (1) ⎨ ⎩ θ =b Where R is the length between the point A and sampled points. For example, R1 is the length of AB and the R2 is the length of AC. Shown as Fig. 5, θ is the variation in angle. a and b are constants. In this paper, a is 0.9 and b is π/6. As shown in Fig. 5, the surrounding points (circle points) are distributed according to this curve, and the color of point A is calculated from the color of the surrounding points (B, C etc.). But the surface of the object is 3D. It is need adjust the sample method on the surface according to the normal of the surface. Shown as in Fig. 6a, A1 is the point which needs updating the color. B1 is the sample point which is computed by the expression (1). B2 is the projection of B1 on the surface according to the vector to the camera. As the result, the sample point on the 3D surface is the B2. In Fig. 6b, three Normal

Camera

A1

B1

Surface B2

(a)

(b) Fig. 6. Sample on 3D surface

states of the route on the different position of the surface are shown. Then according to these adjusted routs, the color of the target points is updated. For generating the route, the information of the normal on surface and the vector to the camera are need. These are draw into normal image shown as Fig.7a and the camera-vector image shown as Fig. 7b. The values of the RGB color show the xyz values of the vectors. By this translation, the 3D problem is translated to the 2D problem. As the result, the process of representing the original color becomes simple. The sample rout can be adjusted from these vector images. 4.3 Update the Color This color updating is calculated in the logarithmic field using the LMS color system. RGB color values are converted into LMS values in the logarithmic field, and the color of each point is then updated along the sample route. Finally, the LMS color

1032


(b)

(a) Fig. 7. Vector images

values in the logarithmic are converted back to the RGB system. The discussion at next is in the LMS color system. The speed of the pigment variation on surface is different, so this different information can be used to represent the original color of the mask. Usually, the process of the pigment variation is similar on the different superficial position. If the position is near, the pigment variation is more similar. As this reason, when update the color of a target point. The color of the sample points along with the rout will give some contribution to the color of the target point. The near points give more contribution. Based on this idea, the next expression is constructed: N

NP ( x, y ) = OP ( x, y ) +

∑ i =0

OPi ( x' , y ' ) − OP( x, y ) k N −i +1

.

(2)

Here, NP(x,y) is the new color of the target point (point A in Fig. 5); OP(x,y) is the old color of the target point; OPi(x’,y’) is the old color of the sample points (points B and C in Fig. 5). i is the index of the sample points along with the sample route. N is the total number of the sample points and is 12 in this paper. If the index i is small, the distance between the sample point and the target point is large. k is a constant and is the parameter that control the amount of influence one point has on another. In this paper, k is 2.0. Expression (2) sums the influence of all the sample points. The influence is big if the sample point is near to the target point. Using this expression, the color of target point is updated based on the color of sample points. This compute process is done by some times. If the same pigment shows the same color, it is mean that the original color is got and the compute process is finished. Despite the pigment variation process is not understood well, the difference of the color on surface can help us to know something about the pigment variation. The compute process that mentioned above can infer the original color of the pigment automatically.


1033

5 Implementation and Discussion Implementation is carried out using two Noh masks. One is the mask that is made about thirty years ago. The other mask is famous one that is made about three hundreds years ago. Shown as in Fig. 8a, the current mask color is a little dark. After the cleaning process, the color becomes new and is bright shown as Fig. 8b. This mask is made thirty years ago. The maker of the mask says that the color is similar to the original one. When this mask is cleaned, the photo of the mask is used directly as mentioned above.

(a)

(b) Fig. 8. Implement of a current Noh mask

(a)

(b)

Fig. 9. Implementation of a famous ancient Noh mask

Another mask is made about three hundreds years ago and nobody know the original color. At this case, the photo is projected to a column surface and a texture is got. The sample algorithm and cleaning process is carried out same as above. And then the result texture is mapped on the 3D mesh surface. Fig. 9a show the current

1034


Noh mask with color texture and Fig. 9b show the color restored one. All of the color is bright and look like a new one. Especially, the color of the skin and the red color of the lip is look well. The key idea of the technique proposed in this paper is using the different color information on the surface which is similar to the weathering process of pigment. By this method, the original color of the Noh mask can be got. This technique need not segment the pigment field and can be applied on the 3D object easily. These are the main advantages of this technique. In some cases, the pigment of Noh mask is lost. The technique presented in this paper can not represent the original color of Noh mask in this case. So it is need to analysis the original pigment types and represent it by developing a new technique.

6 Summary In this paper, we describe the color representation of a Noh mask by use of a sample route which takes into account the 3D shape of the object. The effectiveness of this technique is demonstrated by the results obtained. The technique can also be used to represent color in other 3D objects. To further develop this work, we intend to calculate the influence of weathering on the pigment colors in order to obtain a better result. We will also attempt to develop a new rendering technique which takes into account the pigment microstructure of the Noh mask. Acknowledgments. This work was supported partly by "Kyoto Art Entertainment Innovation Research" of Centre of Excellence Program for the 21st Century by Ministry of Education, Culture, Sports, Science and Technology of Japan.

References 1. M. Barni, F. Bartolini, and V. Cappellini: Image Processing for Virtual Representation of Artworks. IEEE MultiMedia, Vol. 7, No. 2, pp.34-37, (2000) 2. CAD CENTER CORPORATION: http://www.cad-ceter.co.jp/casestudy/detail.php?ID= cg0067. (in Japanese) 3. DNP DIGITALCOM CO. LTD, and NHK. Representation the color of “Hell Scroll", http://www.dnp.co.jp /nenshi/nenrin/lab/syufuku.html (in Japanese). 4. F.Drago and N.Chiba: Locally Adaptive Chromatic Representation of Digitally Acquired Paintings. The International Journal of Image & Graphics, Vol.5, No.3, pp.617-637, (2005) 5. S. Ihara: Seigen Ihara and Noh mask. http://www5.ocn.ne.jp/~seigen/ (in Japanese) 6. E. H. Land and J. J. McCann: Lightness and the Retinex Theory. Journal of the Optical Society of America, Vol. 61, No. 1, (1971) 7. X. Li, D. Lu, and Y. Pan: Color Representation and Image Retrieval for Dunhuang Fresco Preservation. IEEE MultiMedia, Vol. 7, No. 2, pp.38-42. (2000) 8. S. Schar, H. Bieri, X. Jiang: Digital Representation of Medieval Tapestries. In the 6th International Symposium on Virtual Reality, Archaeology and Cultural Heritage, VAST 2005, pp.235-240, (2005) 9. N.Yoshida, T.Saito: Aesthetic curve segment. IPSJ SIG technical report, Vol.2005, 2005CG-121, (2005) (in Japanese)

Modeling and Immersive Display of Cultural Heritage* Lei Zhao, Duan-qing Xu**, Yi Li, and Yi-min Yu College of Computer Science, Zhejiang University, Hangzhou, China [email protected] Abstract. In order to display virtual cultural heritage environment in CAVE based on multiple cheap PC, we implement a collaborative display platform called Multipro. The application developer doesn’t need to know anything about stereo display principles and the platform architecture. We reconstruct the model of No285 cavern of Dunhuang and Hemudu virtual environments by a variety of software and techniques. The paper also addresses the issues of the modeling and display of immersive virtual heritage environments (VHEs).We in detail describe the methods we use to model the heritages and how we fill the holes in the raw heritage models and remove noise of the models. This work is being developed in the context of the National Grand Fundamental Research 973 Program of china. Keywords: CAVE, Virtual Reality, Hybrid Rendering, Cultural Heritage, Rendering System.

1 Introduction In conventional computer graphics, the computer-generated scene is viewed through a window, thus, users have an outside-looking-in experience. VR offers the means to leap through that window, allowing users to participate in the computer-generated scene with an inside-looking-out experience [1]. Many control tasks can be intuitively accomplished by walking or flying through the environment; and, manipulation of objects is done by hand and finger gestures. Users can also receive feedback from the virtual environment as audio, tactile or force cues. So the VR has been applied to various areas, this paper is to utilize the VR device to display the cultural heritages in order to protect them. With the development of computer software and hardware, the personal computer is very cheap according to graphic workstation. In these circumstances, constructing a hardware system consisting of cheap personal computer can reach a high performance. In order to display virtual heritage environment based on multiple cheap PC, we implement a display platform called Multipro, which provide a platform to display immersive application, which can be extended to be used in the areas of urban planning/architecture and cultural heritage education. Multipro is a PC cluster based system developed by C++ and Qt library, Multipro is responsible for communication among multiple machines. Benefiting from modular design, the application developer doesn’t need to know anything about stereo display *

This work is supported by the National Grand Fundamental Research 973 Program of China (2002CB312106). ** Corresponding author. Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1035 – 1044, 2006. © Springer-Verlag Berlin Heidelberg 2006

1036

L. Zhao et al.

principles and the platform architecture. Besides, different applications can be switched at run-time of the platform, without the inconvenient switch of application processes. In this paper, we build a CAVE virtual environment using multiple personal computers. The model of these objects can be acquired by different methods such as photograph, scanner. The unearthed relic model can be from the 3d scanner, which generates a huge volume of point samples with complex details and texture. Rendering or manipulating this mass of data is a main challenge for the community. In order to render this point cloud, a common way is to reconstruct triangle meshes from the samples using mesh reduction [2]. Although the capabilities of computer graphics hardware have dramatically increased in the past decade, the handling of scene complexity is still one of the most fundamental problems in computer graphics. Although sophisticated hardware implementations are available, these algorithms are not capable of displaying highly complex scenes in real-time. So we introduce a new efficient screen space approach for highly complex models real-time rendering, this method can render large complex model acquired by scanner with high detail texture. The creation of realistic 3D models has, typically, been a time-consuming laborintensive manual process, requiring well-trained artists. Recent advances in computer graphics and vision allow the creation of realistic models based on photographs that are of very high quality. Many methods [3] have been developed to perfect and display such models. We take a lot of time to scan the frescoes and figures of Buddha of Dunhuang, also take photos of this frescoes and figures of Buddha. We reconstruct the model of No 285 cavern of Dunhuang and Hemudu virtual environments by a variety of software and techniques. In this paper we focus our attention on two areas, realism and speed. For realism, we introduce a hybrid rendering method for realistic display of existing heritage model based on triangle-based, and a point-based (PBR) method to render a relic. By introducing view-dependent display techniques we achieve unprecedented realism for VHEs. The software could potentially be used for some of the applications we examine; each approach has different tradeoffs. The following sections will in detail describe the methods we use to model the heritage and how we fill the holes in the raw heritage models and remove noise of the model. Multipro system is also discussed .The whole work is being developed in the context of the National Grand Fundamental Research 973 Program of china. The aim of the project is to protect the cultural heritage by realistically recreating archaeological site that can be displayed in a range of immersive VR systems using a high-quality, view-dependent algorithm, the observer can observe the heritage in near distance in virtual environment. Finally, Dunhuang No 285 cavern and Hemudu are as examples demonstrated in the virtual environment.

2 Reconstruction of a Realistic Virtual Heritage Environment The model of cultural heritage is acquired by scanner, the texture of the model can be from photograph, and then the photograph as texture is mapped onto model. The raw data provided by a 3D scanner may contain several artefacts, including noise, difference in resolution between near and far surfaces, and distortions due to intensity

Modeling and Immersive Display of Cultural Heritage

1037

bias. The following sections in detail describe data acquisition and how to remove noise and fill the holes in the raw models. 2.1 Data Acquisition The selected cultural heritage is, Duhuang, located in Gansu province in northeast china. This cultural heritage site was chosen due to its cultural, educational, and symbolic significance, since ultimately our goal is to use this VE reconstruction as an educational and presentation tool in a museum. This site now exists, but in order to protect them, we need reconstruct them so that the people can observe the cultural heritage through virtual display environment, so we reconstruct it by scanning the real heritage. A single scan usually contains holes due to occlusion, and samples near and far objects at different resolutions. Thus, to get a more complete model, or to obtain a more uniform resolution, we need to scan the cavern from multiple locations. Automatically determining the best set of locations for scanning is a hard problem [4]. The simplest approach is to select the scanning locations annually, trying to minimize the number of scans necessary for a good coverage of the environment, and making sure that there is some overlap between the scans [5]. Other low-end systems [5] rely on a human operator, who uses an interactive tool to select matching features on each scan. From the matching features, a rigid transformation is computed that aligns the two scans [7]. Automatic feature detection, although desirable, is a hard problem, and is currently an active area of research [6]. Optical methods may be passive or active. Passive methods do not interact with the object being scanned, and include computer vision techniques such as shape-from-shading for single images, stereo triangulation for pairs of images, and optical flow and factorization methods for video streams. Passive methods require little special purpose hardware, they rarely construct accurate models, and their main application is object recognition, so we use active method to scan the heritage model. 2.2 Data Filtering The raw data provided by a 3D scanner may contain several artefacts, including noise, difference in resolution between near and far surfaces, and distortions due to intensity bias. Many researchers have studied the artefacts related to triangulation scanners [8]. Here we focus on artefacts related to optical time-of-light scanners [5]. One artefact related to optical time-of-flight scanners is intensity bias: points with too low or too high intensity tend to have unreliable radius. One approach to minimize the intensity bias of the scanner is to scan a calibration pattern, and build a correction table. Correa et al. [5] use a calibration image that changes linearly from black to white, and is surrounded by a white background. The image is placed on a flat surface, and then scanned. From the scan, a bias correction table that is used for correcting subsequent scans is built. One approach to deal with noise is to average samples from overlapping scans. Another approach is to resample the points based on a local estimate of the underlying surface. We adopt the latter to filter the heritage models from scanner.

1038

L. Zhao et al.

2.3 Filling Small Holes In locally smooth surfaces, holes can be filled using surface interpolation methods. Wang and Oliveira [9] describe a procedure for automatic identification and filling of holes in point clouds. The method consists of first creating a triangle mesh using a surface construction algorithm. Holes are then automatically identified by analyzing the resulting mesh searching for cycles of non-shared edges. Note that since it is not possible to distinguish between an under-sampled region and a real hole in the surface, user interaction is required to guarantee proper reconstruction. Once a hole has been identified, the missing region can be interpolated from its neighborhood. Whereas several surface fitting techniques can be used to reconstruct missing regions, for most of them, the resulting surfaces do not pass through the original samples, which tend to introduce discontinuities. Avoiding discontinuities between the original and filled areas is important in order to guarantee the quality of the reconstructed models. The hole filling strategy proposed by Wang and Oliveira [9] consists of adding new samples to under-sampled regions by re-sampling an interpolated surface reconstructed using moving least squares [11], a variation of the original least squares method that guarantees the interpolation of the original samples. An important requirement during the introduction of new samples is to enforce that the sampling density inside the hole matches the sample density of the interpolation context. This is necessary to guarantee that the surface reconstruction algorithm used will not just leave even smaller holes. The next one is perpendicular to the plane. Once the plane has been computed, the samples belonging to the corresponding interpolation context are projected onto it. The problem of deciding the re-sampling positions inside the hole is then reduced to the 2D problem of finding a set of (x; y) positions inside the projection of the hole that preserves the sampling density of the projection of the interpolation context. These positions are computed, and the new samples are obtained by re-sampling the interpolated surface at these points. Once these samples have been added, a new triangle mesh is created for the object using the surface reconstruction algorithm described in [10]. In addition to reconstructing geometry, it is also necessary to reconstruct color and texture. The moving-least-squares procedure can also be used for the reconstruction of smoothly varying colors. 2.4 Texture Acquisition The texture can reduce the amount of triangles of heritage model, and can increase the sense of realism. How a surface point reflects light depending on its normal, the incident light direction. The emerging light direction, and the light’s wavelength is known as the bidirectional reflectance distribution function (BRDF) of a surface. Measuring BRDFs accurately is a hard problem [12]. The simplest approach for acquiring the texture of an environment is to take pictures of it, and then map these photographs onto the previously acquired geometry. To map a photograph to the geometry, we need to know the camera projection parameters (intrinsic parameters), and the position and orientation of the camera when the photograph was taken (extrinsic parameters). Real cameras do not perform a perfect perspective projection as a pinhole camera does, and


1039

present many kinds of distortions, e.g., radial distortion. One solution to this problem is to model the action of the distortions, and to find an inverse mapping. A widely used model was proposed by Tsai [13]. Tsai’s camera model has five intrinsic parameters, namely the focal length, the first-order coefficient of radial distortion, the coordinates of the image center, and a scale factor. One way to calibrate these intrinsic parameters is to take a photograph of a planar checkerboard pattern with known geometry, and then find the image location of the checkerboard corners with sub-pixel accuracy. Implementing these algorithms, which require full non-linear optimization, is fairly complex. Luckily, high quality implementations are available from Willson [14]. After calibrating the intrinsic camera parameters, the process of acquiring the images goes as follows. First, we take photographs of the heritage and cavern, keeping the same camera settings for all photographs to avoid having to recalibrate the intrinsic parameters. Then, for each photograph, we first remove its radial distortion, using a warp based on the coefficient found above, and then find the position (translation) and orientation (rotation) of the camera when we took the photograph relative to an arbitrary global coordinate system It is hard to automatically solve the image-geometry registration problem. By specifying pairs of corresponding points between the images and the geometry, it is possible to find the extrinsic camera parameters [14]. The approach taken by McAllister et al. [15] is to keep the center of projection (COP) of the camera coincident with the COP of the 3D scanner, while acquiring panoramic images. This simplifies the registration problem by only requiring the computation of a rotation. Furthermore, this enforces that no occlusion artifacts arise. Correa et al. [5] uses an interactive program to specify corresponding points, typically requiring only 7 to 10 correspondences to obtain a good calibration of the camera. One advantage of this approach is that it allows the user to take pictures from any position. Once all the parameters have been found, it is straightforward to map the colors from the photograph to the scan (or scans) that it covers. For each 3D point in a scan covered by the photograph, we find its 2D projection on the camera plane, and assign the corresponding pixel color to it (Figure 2). To support view-dependent effects (such as highlights), we can store multiple color samples per point, and at runtime find the color of a point by interpolating the closest color samples. The texture can reduce the amount of triangles of heritage model, and can increase the sense of realism. How a surface point reflects light depending on its normal, the incident light direction, the emerging light direction, and the light’s wavelength is known as the bi-directional reflectance distribution function (BRDF) of a surface. Measuring BRDFs accurately is a hard problem [12]. The simplest approach for acquiring the texture of an environment is to take pictures of it, and then map these photographs onto the previously acquired geometry. To map a photograph to the geometry, we need to know the camera projection parameters (intrinsic parameters), and the position and orientation of the camera when the photograph was taken (extrinsic parameters). First, we take photographs of the heritage and cavern, keeping the same camera settings for all photographs to avoid having to recalibrate the intrinsic parameters. Then, for each photograph, we first remove its radial distortion, using a warp based on the coefficient found above, and then find the position (translation) and orientation (rotation) of the camera when we took the photograph relative to an arbitrary global coordinate system It is hard to automatically solve the

1040

L. Zhao et al.

image-geometry registration problem. By specifying pairs of corresponding points between the images and the geometry, it is possible to find the extrinsic camera parameters [14]. The approach taken by McAllister et al. [15] is to keep the center of projection (COP) of the camera coincident with the COP of the 3D scanner, while acquiring panoramic images.

3 System Overview The software engineering aspects of this work are excellent. The whole system is based on dynamic link library. The display platform provides an interface definition which can be derived by developers of the application to implement their own code, the code of special application is compiled into a file of DLL. The display platform will let user to load a given DLL to run, and DLL as a call-back module is invoked by the display platform. This platform is based on Microsoft OpenGL library. Multipro can be run on multiple PC, so this system can make full use of the power of cheap pc since now the price of single pc has decreased so much. Every PC generates two view images, one for each eye, switched by electronic glasses. Existing stereo development libraries, e.g. CAVELib, Avocano, require fully or partially understanding, at least caring of stereo display principles. Furthermore, most of existing libraries are used to develop a full application with all stereo display functions, e.g. multi-tasking, shared memory, stereo displaying, and so on. Multipro is a PC cluster based system developed by C++ and Qt library. Multipro is composed of following modules of controller, renderer, Multipro Library, communication layer and application module. Benefiting from modular design, the application developer doesn’t need to know anything about stereo display principles and the platform architecture. Besides, different applications can be switched at run-time of the platform, without the inconvenient switch of application processes. But Multipro is not responsible for rendering the scenes, just responsible for network communication and collaboration of multiple PC, so we have to do extra things in order to display immersive interactive scenes. Multipro is not responsible for rendering, so the developer need to customize a specified rendering code to render scenes according to different application. Figure 1 is the framework of a collaborative system platform Multipro.

Fig. 1. The framework of a collaborative system platform (Multipro)


1041

4 Rendering As VR technology becomes commonplace, there has been a proliferation of VR in fields such as design, education, and entertainment or, in other words, areas where VR applications are more easily available to and accessible by the general public. In the field of education, VEs have been developed to help teach concepts that are hard to learn or difficult to visualize otherwise. In design, VR has been used where conventional media are ill-suited to represent the work processes in ways that make them easy to visualize. In both cases, VR, with its immersive and interactive properties, can offer possibilities and solutions that are otherwise very difficult to obtain. For these reasons, we have chosen two application domains that relate to learning and working in VR, an archaeological reconstruction and automobile appearance display project, in this paper we will focus on the archaeological site reconstruction. In the first case, the Society of Dunhuang Studies has been involved for years in display frescos and the cavern on which the frescos are drawn, reconstructing the archaeological site in order to make it more accessible to visitors (Fig. 2). We use cubic display environments called CAVE to display the frescos of Dunhuang. These frescos are scanned form real wall of Dunhuang, which consists of huge points data and high detail texture. For the second application, we display an outdoor complex scene which consists of lot of objects, and these objects can be from scanner or photograph or triangle model. For scanned model we adopt PBR-based rendering method, in order to speed up the rendering we introduce a new efficient screen-space rendering algorithm to render point-based objects. Our algorithm refers to the randomized z-buffer algorithm. The main idea of the algorithm is utilize hierarchical data structure to represent the models. (Every model is presented by uniform spatial data structure which is multiresolution hierarchy based on bounding spheres, see [13]). Before the scenes are rendered, the visibility test is conducted using the bounding sphere of every object in the scenes. Our algorithm only renders all visible object nodes, we adopt conservative visibility test, so the rendering quality of the scenes is not damaged .we will adopt different rendering policy for different object nodes. Our algorithm conducts hierarchical visibility culling and multi-resolution rendering for every object nodes. We render object nodes from scanner data represented by triangle primitive using the method similar to QSpat. The object nodes represented by triangle primitive are reconstructed in pre-process phase using Octree, we utilize back-face culling, occlusion culling, frustum culling to reduce the number of triangles according to the position of viewpoint dynamically. Finally we utilize screen space approach to generate a set of point-sample, and construct an image of this object from a dynamically chosen set of surface sample points. All the set of sample points of object nodes represent the complex model, hence not every single triangle must be handled separately during rendering. First, given number of point-samples are chosen so that they cover the projections of visible objects in the image plane. This can be done in highly output-sensitive time. Here, the sample point selection is the key to avoid expensive geometric queries. Thus, for choosing sample points, a probability density proportional to the projected area of the surfaces of the objects has to be used. However, a sample selection using this viewpoint-dependent probability density does not seem to be possible in sub-linear

1042

L. Zhao et al.

time. Therefore, we use a multi-resolution hierarchy based on bounding spheres. In a preprocessing step, we construct hierarchical data structure based on bounding spheres. For each viewpoint, a suitable set of visible object is chosen dynamically from the hierarchy. The projected area of object bounding box to be used to determine the number of sample points to be chosen from each object: The set of point sample of the whole scenes consists of all visible objects set of point sample. In most cases, it is sufficient to consider only the distance between an object and the viewer in order to estimate the projection area. The sample points are then chosen within a specific object according to the number of point sample our sample algorithm traverses the hierarchical data structure, records the number of inner node plus leaf node in every layer, if the number is more than the object’s expected set of point sample, this layer’s nodes including inner node is the set of point sample we want. Using pre-computed search structures, we can show that this multi-resolution data structure does not harm the image quality. In a second step, the algorithm conduct occlusion test with z-buffer between the chosen sample points and renders the resulting image using the visible points.

5 Interaction and Animation Interaction is the key element of the virtual environment. If there is no interaction with virtual environment, the environment will lack no sense of realism. So we introduce interaction into the Hemudu scene, the observer can interact with the virtual environments through a data gloves, the user can take a harpoon to fish. We use FOB(The Flock of Birds) to track the position of eyes and hand. FOB is a six degreesof-freedom measuring device that can be configured to simultaneously track the position and orientation of multiple sensors by a transmitter. Each sensor is capable of making from 20 to 144 measurements per second of its position and orientation when the sensor is located within ± 4 feet of its transmitter. We use Cybe-Grasp to track the action of the finger, and the user can get force feedback form the objects of scenes by wearing the Cybe-Force device. By introducing viewpoint dependent texture mapping technique the user also observe the relic in near distance. We use Motion Capture to capture the animation. Motion Capture is the process of capturing the movement of a real object and mapping it onto a computer generated object. Motion Capture creates synthetic actors by capturing the motions of real humans. In this case, special markers are placed over the joints of actors. Then, a special hardware samples the position and/or orientation of those markers in time, generating a set of motion data, also known as motion curves. Our prototype system renders the key-frame animation generated by Motion Capture.

6 Results For the Dunhuang 285 cavern, the overall photography, calibration, modelling and texturing required about 8 person-months. The total number of triangles is about 3M. Figure.2. A) the fresco and figure of Buddha of Dunhuang displayed on the single PC, C) The female ancient about 7000years ago is pounding rice with a pestle


1043

(animation) in Hemudu virtual scenes in CAVE virtual environments, D) virtual display of the No 285 cavern of Dunhuang in CAVE virtual environments.

7 Conclusions and Futureworks We reconstruct the model of No285 cavern of Dunhuang by a variety of software and techniques, we implement a collaborative display platform called Multipro, which provide a platform to display immersive application which can be extended to be used in the areas of urban planning/architecture and cultural heritage education. Multipro is a PC cluster based system developed by C++ and Qt library. As mentioned above, Multipro is not responsible for rendering the scenes, just responsible for network communication and collaboration of multiple PC, so we have to do other things in order to render special application data, we plan to implement the rendering engine which provide the developer a interface. Rendering engine should has the following features: It is Simple, easy to use, and Supports vertex and fragment programs (shaders), provides automatic support for many commonly bound constant parameters like worldview matrices, light state information, object space eye position etc; It Supports the complete range of fixed function operations such as multi-texture and multi-pass blending, texture coordinate generation and modification, independent color and alpha operations for non-programmable hardware or for lower cost materials.

Fig. 2. A) The acquisition of photographs, B) Calibration of the photograph, C) The female ancient about 7000years ago is pounding rice with a pestle (animation) in Hemudu virtual scenes in CAVE virtual environments, D) virtual display of the No 285 cavern of Dunhuang in CAVE virtual environments, E) The model of Buddha

1044

L. Zhao et al.

References [1] Bishop, G., Fuchs, H., et al. Research Directions in Virtual Environments. Computer Graphics, Vol. 26, 3,Aug. 1992, pp. 153--177. [2] Chamberlain,xB., DeRose, T., Lischinski, D., Salesin, D., Snyder, J.: Fast Rendering of Complex environments Using a Spatial Hierarchy.In: Proc. Graphics Interface ’96, 132141, 1996. [3] Gorther, S. J., Grzeszczuk, R., Szeliski, R., Cohen,M. F.: The Lumigraph.In: SIGGRAPH 96 Proceedings, [4] S. Fleishman, D. Cohen-Or, and D.Lischinski. Automatic camera placement for imagebased modeling. Computer Graphics Forum, 19(2):100–110, 2000 [5] W. T. Corrêa, S. Fleishman, and C. T. Silva. Towards point-based acquisition and rendering of large real-world environments. In Proceedings of the 15th Brazilian Symposium on Computer Graphics and Image Processing, 2002. To appear. [6] G. M. Cortelazzo, C. Doretto, and L. Lucchese. Free-form textured surfaces registration by a frequency domain technique. In Proceedings of the International Conference on Image Processing, pages 813–817, 1998. [7] B. K. P. Horn. Closed form solution of absolute orientation using unit quaternion. Journal of the Optical Society A, 4(4):629–642, Apr. 1987. [8] P. Hébert, D. Laurendeau, and D. Poussart. Scene reconstruction and description: Geometric primitive extraction from multiple view scattered data. In Proceedings of IEEE Computer Vision and Pattern Recognition, pages 286–292, 1993. [9] J. Wang and M. M. Oliveira. A hole filling strategy for surface reconstruction from range images. Technical Report TR02.07.18, SUNY at Stony Brook, 2002. [10] M. Gopi and S. Krishnan. A fast and efficient projection-based approach for surface reconstruction. High Performance Computer Graphics, Multimedia and Visualization,1(1):1–12, 2000. [11] P. Lancaster and K. Salkauskas. Curve and Surface Fitting. Academic Press, London, 1986. [12] H. Lensch, M. Goesele, J. Kautz,W. Heidrich, and H. Seidel. Image-based reconstruction of spatially varying materials. In Rendering Techniques 2001, pages 103–114,2001 [13] Szymon Rusinkiewicz and Marc Levoy. Qsplat: A multiresolution point rendering system for large meshes. In Proceedings SIGGRAPH 2000, pages 343–352. ACM SIGGRAPH, 2000.

Personalized Content Presentation for Virtual Gallery* Wonil Kim1, Hanku Lee2,**, Kyoungro Yoon3, Hyungseok Kim2, and Changduk Jung4 1

College of Electronics and Information at Sejong University, Seoul, Korea [email protected] 2 School of Internet Multimedia Engineering at Konkuk University, Seoul, Korea [email protected], [email protected] 3 School of Computer Engineering at Konkuk University, Seoul, Korea [email protected] 4 School of Information and Communication at Korea University, Korea [email protected]

Abstract. Utilizing Virtual Reality technologies for virtual museum brings new ways of interactive presentation of the contents. In addition to interactivity, personalization is an important emerging issue in digital content management especially with virtual reality. For the virtual museum or gallery, selection and presentation of personalized content will improve user experience in navigating through huge collections like Musée du Louvre or British Museum. In this paper, we present a personalization method of massive multimedia content in virtual galleries. The proposed method is targeted for the pictures that could be characterized by its large amount of source in galleries. The method is based on classified image features which are extracted using standard MPEG-7 visual descriptors. Using Neural Networks, the best matching pictures are selected and presented in the virtual gallery by choosing similar styles from the extracted preference features. The simulation results show that the proposed system successfully classifies images into correct classes with the rate of over 75% depending on the employed features. We employ the result into a virtual gallery application which gives opportunities of automatically personalized gallery browsing.

1 Introduction As the era of information flood begins, everyday we encounter much more digital data than the traditional textual data on the Internet and TV. The attention and focus of information shift from textual forms to digital multimedia forms. Digital multimedia brings a new perspective on information presentation accompanied by technological advances including virtual reality. Interactivity has been a quite popular issues in multimedia content management. It has enabled a range of new multimedia applications such as interactive TV, virtual museum and virtual heritage [18]. With this digital form of the media, conventional museum, gallery or cultural sites could be extended from current forms enriched by additional objects which could not be presentable in physical * **

This paper is supported by Seoul R&BD program. Author for correspondence: +82-2-2049-6082.


1046

W. Kim et al.

location due to limitation on space and security issues. With these new services the quantity of digital media being presented to user is drastically gained. In this paper, we deal with a method to manage massive amount of multimedia contents for easy access by the user. Among many media types, we focus on the picture which typically shows the problem of efficient presentation in virtual gallery due to its massiveness in quantity. In addition to interactivity, personalization is an important emerging issue in digital content management. For the virtual gallery, selection and presentation of personalized content will improve user experience in navigating through huge collections like Musée du Louvre or British Museum. The virtual gallery presents personalized contents among millions of pictures according to user preference. A preference of user can be given to the virtual gallery in a form of visiting history and evaluation on different types of pictures. In this paper, we present a personalization method of massive multimedia content in virtual galleries. We adopt MPEG framework as current MPEG standard incorporates issues of personalization within MPEG-7 and MPEG-21 [19]. Personal preference is a complex and difficult aspect to be measured. In this paper, we propose a method to measure preference by its styles. Digital contents are automatically classified by their styles. By specifying preferred styles, users can choose and watch their favorites from the sea of digital contents. The main purpose of this paper is to apply MPEG-7 standard to digital multimedia images by utilizing MPEG-7 descriptors. By analyzing MPEG-7 descriptors, we create a prototype system that can be used to categorize digital picture styles under visual environments, and introduce effective methodology of image classification via experiments. For example, it will tell that a given image belongs to scenery pictures, peoples, animals, and vehicles image with confidence rate from 0.0 to 1.0. The system uses MPEG-7 visual descriptors for image features and the classification module employs neural network. The usage of this system is enormous. It can be properly fit into the image classification engine for search system. We discuss related works for the image genre classification in section 2. In section 3, we propose our Neural Network based image genre classification system. The simulation results are presented in section 4. We conclude in section 5.

2 Related Works 2.1 MPEG-7 Descriptors and Image Classification Systems MPEG-7 is a recent emerging standard used in image classification systems. It is not a standard dealing with the actual encoding and decoding of video and audio, but it is a standard for describing media content. It uses a XML to store metadata and solves the problem of lacking standard to describe visual image content. The aim, scope, and details of MPEG-7 standard are nicely overviewed by Sikora of Technical University Berlin in his paper [1]. There are a series of researches that use various MPEG-7 descriptors. Ro et al. [2] shows a study of texture based image description and retrieval method using an adapted version of homogeneous texture descriptor of MPEG-7. Other studies of image classification use descriptors like a contour-based shape descriptor [3], a histogram descriptor [4], and a combination of color structure and homogeneous

Personalized Content Presentation for Virtual Gallery

1047

descriptors [5]. As a part of the EU aceMedia project research, Spyrou et al. propose three image classification techniques based on fusing various low-level MPEG-7 visual descriptors [6]. Since the direct inclusion of descriptors would be inappropriate and incompatible, fusion is required to bridge the semantic gap between the target semantic classes and the low-level visual descriptors. There is a CBIRS that combines neural network and MPEG-7 standard: researchers of Helsinki University of Technology developed a neural, self-organizing system to retrieve images based on their content, the PicSOM (the Picture + self-organizing map, SOM) [7]. The technique is based on pictorial examples and relevance feedback (RF). The PicSOM system is implemented by using tree structured SOM. The MPEG7 content descriptor is provided for the system. In the paper, they compare the PicSOM indexing technique with a reference system based on vector quantization (VQ). Their results show the MPEG-7 content descriptor can be used in the PicSOM system despite the fact that Euclidean distance calculation is not optimal for all of them. Kim et al. [8, 9] proposed a neural network based classification module using MPEG-7. In this model, inputs for the neural network are fed from the feature values of MPEG-7 descriptors that are extracted from images. Since the various descriptors can represent the specific features of a given image, the proper evaluation process should be required to choose the best one for the given image classification. In their adult image classification, it is found that color related descriptors, such as Color Layout and Color Structure are very effective in performance. In fact, classifying adult image is closely related with skin color detection. 2.2 Image Classification Image classification is a core process of digital image analysis. It is used in many areas, such as remote sensing and image retrieval. Remote sensing is an acquisition of meaningful information from an object by a recording device, which is not in physical or intimate contact with the object. For example, image classification is applied to a data interpretation process of remotely acquired digital image by a Geographic Information System (GIS). Image retrieval is another research area using image classification. A user requests an image by query. It returns an image (or a set of ordered images) from its image database by matching features of a query image, such as color histogram and textual measures, with those of database images. Image classification is also used to create image databases and adding images into for the image retrieval system. The system extracts semantic descriptions from images and putting them into semantically meaningful categories. The automatic classification of WWW images as photographs or graphics including cartoons is a well-known contents image classification. Examples are the WebSeek search engine [10] and the systems described in [11]. Unfortunately, their features take advantage of some characteristics of web images that do not exist in cartoon images, notably the aspect ratio and the word occurrence in the image URLs. In the same fashion of [10, 11], [12] emphasizes the basic characteristics of cartoons and implemented nine color descriptors in order to distinguish photos and graphics on a database of 1200 samples.

1048

W. Kim et al.

The photo/graphics classifier of [12] had been previously implemented as part of the Acoi system [13]. A decision rule classifier [14] has been trained on the features given in [12] on varying quantities of training data. The implementation has a classification score of 0:9 on a data set of 14,040 photos and 9,512 graphics harvested from the WWW. One of the classic classification problems in content-based image retrieval systems is city images vs. landscapes. [15] separates urban images and rural images using a multiscale steerable pyramid to find a dominant orientation in four by four subblocks of image. They classify the image as a city scene if enough subblocks have vertical orientation tendency. A texture orientation used in [15] is one of popular low-level features of images, which is used for pattern retrieval. [16] proposes to use the Gabor wavelet features for texture analysis. In their paper, they provide a comprehensive experimental evaluation and they indicate an analysis using Gabor wavelet features are more accurate in pattern retrieval than analyses using three other multiscale texture features: pyramid-structured wavelet transform (PWT), tree-structured wavelet transform (TWT), and multiresolution simultaneous autoregressive model (MR-SAR) features by comparing them. Another popular feature used to retrieve images from digital image libraries or multimedia databases is a color histogram. It is the efficient and insensitive method, but it has coarse characteristics as well. So images of totally different appearance can have similar histograms. [17] proposes a Histogram refinement technique to compare images using additional constraints. The technique includes of splitting the pixels in a given bucket into several classes, based upon local property. Pixels in the same class can be compared with others in the same bucket.

3 The Proposed Image Genre Classification System 3.1 The Proposed Architecture The sample photos of animal, car, people portrait, and scenery are illustrated in Figure 1. The proposed system classifies these images as one of four categories. The categories are predefined like (a) animal photos, (b) car photos, (c) people portrait photos and (d) scenery photos.

(a) Animal Photos

(b) Car Photos

(c) People portrait photos

(d) Scenery photos

Fig. 1. Example images from the four categories used for simulation


1049

Figure 2 represents the overall architecture of the proposed image gallery classification system. The proposed architecture consists of two modules; feature extraction module and classification module. Features defined in the MPEG-7 descriptors for the given query images are extracted, and then used as inputs for the following classifier module. Training Feature Extraction

Data Normalization

Testing Neural Network Module

Data Normalization

Feature Extraction

Classification Testing DB

Training DB

Animal

Car

People portrait

Scenery

Fig. 2. The overview of the proposed Image Gallery Classification System

3.2 Feature Extraction Module By running the MPEG-7 XM program, features of training images are extracted in XML format. This feature information in XML format is parsed in the next step and is normalized into values between 0 and 1 with respect to values generated by each descriptor. These normalized values are used as inputs for the neural network classifier. An illustration of extracting input values for Edge Histogram is shown below. The original values, 0 1 1 0 0 6 2 2 4 2 5 2 6 4 4 1 1 0 4 0 4 0 4 0 1 4 1 6 7 4 5 0 6 4 4 5 0 1 5 3 2 3 4 6 3 2 4 6 6 5 4 2 5 6 3 5 4 4 4 2 3 3 3 5 4 2 4 2 5 4 2 4 5 4 4 2 4 6 5 2 are converted into normal values, 0.0 0.1 0.1 0.0 0.0 0.6 0.2 0.2 0.4 0.2 0.5 0.2 0.6 0.4 0.4 0.1 0.1 0.0 0.4 0.0 0.4 0.0 0.4 0.0 0.1 0.4 0.1 0.6 0.7 0.4 0.5 0.0 0.6 0.4 0.4 0.5 0.0 0.1 0.5 0.3 0.2 0.3 0.4 0.6 0.3 0.2 0.4 0.6 0.6 0.5 0.4 0.2 0.5 0.6 0.3 0.5 0.4 0.4 0.4 0.2 0.3 0.3 0.3 0.5 0.4 0.2 0.4 0.2 0.5 0.4 0.2 0.4 0.5 0.4 0.4 0.2 0.4 0.6 0.5 0.2 and followed by the class information. The class information, which is attached to the feature value, is the orthogonal vector value. For example, an Animal image is represented as (1 0 0 0), whereas a Car image would be as (0 1 0 0).

1050

W. Kim et al.

3.3 Classification Module Using the data set of the normalized input features and classes of sports, we can model an NNC in the classification module. According to different MPEG-7 descriptors, the number of the input features can be various. Let us denote the input feature vector obtained from the first MPEG-7 descriptor as X D1 = ( xD1,1 , x D1, 2 , " , x D1,i , " , x D1,n ) , here x D1,i is the ith input feature extracted from 1

MPGE-7 descriptor 1 and the subscript n1 is the dimension of the input features from the first MPEG-7 descriptor. With the same way, the input feature vector obtained from the last MPEG-7 descriptor k can be expressed as X Dk = ( x Dk ,1 , x Dk , 2 , " , x Dk ,i , " , xDk ,n ) . Also, the output vector can be expressed as k

Y = ( y1 , y 2 , ", yi , ", y s ) , here yi is the output from the ith output node and the subscript s is the number of classes. By utilizing the hard limit function in the output layer, we can have binary value, 0 or 1, for each output node yi as Equation (1). netinputo ≥ 0 ⎞ ⎟ otherwise ⎟⎠

⎛ 1, yi = f o (netinputo ) = ⎜⎜ ⎝ 0,

(1)

where fo is the hard limit function at the output node and netinput o is the net input of fo. As shown in Equation (2), the net input is can be expressed as the product of the output vector in the hidden layer, denoted as Yh, and the weight vector Wo at the output layer. netinputo = Wo Yh T

(2)

With the same way, the hidden layer output vector, Yh , can also be computed by functioning the product of the input weight vector and the input vector. Thus, the accuracy of the NNC depends on the values of whole weight vectors. To obtain the optimal weight vectors, the NNC is trained using the back-propagation algorithm which is commonly utilized for training neural networks. The training is done after coding each class of sports into s dimension orthogonal vector. For example, since we have four classes, the classes are coded to (1 0 0 0), (0 1 0 0 ), (0 0 1 0) and (0 0 0 1). Once obtaining an optimal weight vector, we evaluate the performance of NNC using the test data which is unseen during training phase.

4 Simulation 4.1 Environments

The simulation uses a total of 800 images for training (200 for each photo class images), and 400 for testing (100 for each photo class images). It employs five descriptors for image description features, Color Layout, Color Structure, Region Shape, Homogeneous Texture, and Edge Histogram. The inputs consist of MPEG-7 normalized descriptor values. Several classification modules were evaluated; five modules


1051

for each descriptor and the others for performance evaluation of selected two descriptors; Edge Histogram and Color Structure. The number of hidden layers and nodes used in the modules as well as the number of iterations are illustrated in each result Table descriptions. The output layer consists of 4 nodes, one for each photo image class. 4.2 Result

The simulation results of five descriptors for the photo image are shown in Table 1. The overall successful classification rate of the five descriptors is 61.25%, which is a disappointed result. On the other hand, the proposed photo image classification system performs better result in Color Structure and Edge Histogram descriptors; 65.00% and 77.50% respectively. Moreover, in Table 2 and Table 3, we simulated the selected two descriptors in detail. Both the Color Structure and Edge Histogram descriptors did not improve much with the various combinations of tests. Table 1. Test Results of Five descriptors after 100,000 iterations with 2 hidden layers with 30 nodes each (%)

Animal

Car

People

Scene

CL EH HT RS CS average CL EH HT RS CS average CL EH HT RS CS average CL EH HT RS CS average

Animal 42.50 70.00 55.00 45.00 60.00 54.50 22.25 1,25 16,25 21.25 12.50 14.70 5.00 6.25 8.75 16.25 17.50 10.75 8.75 27.50 13.75 26.25 13.75 18.00

Car 26.25 3.75 7.50 20.00 13.75 14.25 60.00 85.00 40.00 43.75 68.75 59.50 21.25 0.00 7.50 12.50 10.00 10.25 18.75 8.75 10.00 22.50 17.50 15.50

People 13.75 0.00 15.00 15.00 16.25 12.00 12.50 0.00 13.75 16.25 7.50 10.00 73.75 93.75 71.25 55.00 68.75 72.50 7.50 2.50 1.25 20.00 6.25 7.50

Scene 17.50 26.25 22.50 20.00 10.00 19.25 5.00 13.75 30.00 18.75 11.25 15.75 0.00 0.00 12.50 16.25 3.75 6.50 65.00 61.25 75.00 31.25 62.50 59.00

1052

W. Kim et al.

Table 2. Test Results of Edge Histogram descriptor after 500,000 iterations with 2 hidden layers with 50 nodes and 100 nodes (%)

Animal Car People Scene

nodes 50 100 50 100 50 100 50 100

Animal 67.50 67.50 6.25 6.25 6.25 5.00 25.00 25.00

Car 3.75 3.75 83.75 82.50 0.00 0.00 11.25 10.00

People 0.00 0.00 0.00 0.00 93.75 95.00 1.25 1.25

Scene 28.75 28.75 10.00 11.25 0.00 0.00 62.50 63.75

Table 3. Test Results of Color Structure descriptor after 500,000 iterations with 2 hidden layers with 50 nodes and 100 nodes (%)

Animal Car People Scene

nodes 50 100 50 100 50 100 50 100

Animal 62.50 61.25 10.00 10.00 17.50 12.50 12.50 12.50

Car 13.75 13.75 68.75 65.00 12.50 10.00 16.25 15.00

People 11.25 16.25 8,75 7.50 67.50 68.75 5.00 1.25

Scene 12.50 8.75 12.50 17.50 2.50 8.75 66.25 71.25

Overall, the proposed photo image classification system performs better result in Color Structure and Edge histogram than other three descriptors. The results seem very promising and can be applied to various image processing domains. It can be easily extended to medical image processing, in which identifying a particular image belongs to a certain symptom is very critical. Also it can be implemented as the main part of image search engine or image collection engine. For a large image data base, it is very useful tool for image retrieval system.

5 Conclusion In this paper, we proposed a method to classify pictures for personalized virtual gallery navigation. The proposed system categorizes images into different styles, such as animals, vehicles, peoples and scenery images. The classification module of the system learns corresponding classification task according to feature values, which is extracted from MPEG-7 descriptors. The simulation results show that the proposed system successfully classifies images into correct classes with the rate of over 80% depending in the employed features. Utilizing the proposed method, users of virtual gallery can store their preferences by recording the visiting history with their personal devices. Recorded history can be


1053

evaluated by its duration and, if given, user’s explicit preferences on each pictures. Using this evaluated measure, for the next visit the virtual gallery proposes newly added or not yet visited items to user for their new experience. This approach can be applied not only for the virtual gallery, but into generic picture management by tagging each picture automatically. The MPEG-7 descriptors can be enriched by this classification along with pre-existing features. By extending proposed preference measurement method, multimedia content querying can be enhanced in more user friendly, context describing ways.

References 1. Sikora, T.: The MPEG-7 visual standard for content description – an overview. IEEE Transactions on Circuit and Systems for Video Technology, Vol. 11, No. 6 (2001) 696-702 2. Ro, Y., Kim, M., Kang, H., Manjunath, B., Kim, J.: MPEG-7 homogeneous texture descriptor. ETRI Journal, Vol. 23, No. 2 (2001) 41-51 3. Bober, M.: The MPEG-7 visual shape descriptors. IEEE Transactions on Circuit and Systems for Video Technology, Vol. 11, No. 6 (2001) 716-719 4. Won, C., Park, D., Park, S.: Efficient use of MPEG-7 edge histogram descriptor. ETRI Journal, Vol. 24, No. 1 (2002) 23-30 5. Pakkanen, J., Ilvesmaki, A., Iivarinen, J.: Defect image classification and retrieval with MPEG-7 descriptors. Lecture Notes in Computer Science, Vol. 2749. Springer-Verlag, Berlin Heidelberg, New York (2003) 349-355 6. Spyrou, E., Borgne, H., Mailis, T., Cooke, E., Arvrithis, Y., O’Connor H.: Fusing MPEG7 visual descriptors for image classification. Lecture Notes in Computer Science, Vol. 3697. Springer-Verlag, Berlin Heidelberg, New York (2005) 847-852 7. Laaksonen, J., Koskela, M., Oja, E.: PicSOM – Self-organizing image retrieval with MPEG-7 content descriptor. IEEE Transactions on Neural Networks: Special Issue on Intelligent Multimedia Processing, Vol. 13, No. 4 (2002) 841-853 8. Kim, K., Lee, H., Yoo, S., Baik, S.: Neural Network Based Adult Image Classification. Lecture Notes in Computer Science, Vol. 3696, pp 481-486, Springer-Verlag, Berlin Heidelberg, New York in 2005 9. Kim, W., Lee, H., Park, J., Yoon, K.: Multi Class Adult Image Classification using Neural Networks. Lecture Notes in Computer Science, Vol. 3501, pp 222-226, Springer-Verlag, Berlin Heidelberg, New York in 2005 10. Smith, R., Chang, S-F.: Searching for images and videos on the world wide web, Tech. Rep. 459-96-25, Center for Communications Research, Columbia University, 1996. 11. Rowe, C., Frew, B.: Automatic caption localization for photographs on word wide web pages, Tech. Rep., Department of Computer Science, Naval Postgraduate School, 1997. 12. Athitsos, V., Swain, J., Frankel, C.: Distinguishing photographs and graphics on the world wide web, in IEEE Workshop on Content-Based Access of Image and Video Libraries, Puerto Rico, June 1997. 13. Windhouwer, A., Schmidt, R., Kersten, L.: Acoi: A System for Indexing Multimedia Objects, in International Workshop on Information Integration and Web-based Applications & Services, Yogyakarta, Indonesia, November 1999. 14. Quinlan, R: Programs for Machine Learning, Morgan Kaufmann, 1993.

1054

W. Kim et al.

15. Wang, J., Li, J., Wiederhold, G.: SIMPLIcity: Semantics-sensitive Integrated Matching for Picture LIbraries, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol 23, pp. 947-963, 2001 16. Manjunath, S., Ma, Y.: Texture features for browsing and retrieval of image data, IEEE Transactions on Pattern Analysis and Machine Intelligence,” vol. 18, no. 8, August 1996. 17. Pass, G., Zabih, R.: Histogram refinement for content-based image retrieval, In Proceedings of the 3rd IEEE Workshop on Applications of Computer Vision, Sarasota, Florida, USA, December 1996. 18. Papagiannakis, G., Schertenleib, S., O'Kennedy, B., Arevalo-Poizat, M., MagnenatThalmann, N., Stoddart, A., Thalmann, D.: Mixing Virtual and Real scenes in the site of ancient Pompeii, Computer Animation and Virtual Worlds, p 11-24, Volume 16, Issue 1. February 2005. 19. Kim, H., Joslin, C., Di Giacomo, T., Garchery, S., Magnenat-Thalmann, N.,: Device-based Decision-making for Adaptation of Three-Dimensional Content. The Visual Computer. 22(5) May 2006.

Accessible Information from Radiated Sound of Extracorporeal Shock Wave Lithotriptor for Improving Fragmentation Efficiency Yun Seok Jang1 and Jin Ho Choi2 1

Dept. of Electrical Engineering, Pukyong National University, San 100, Yongdang-Dong, Nam-Gu, Busan, 608-739, Korea [email protected] 2 Dept. of Computer Engineering, Pusan University of Foreign Studies, 55-1, Uam-Dong, Nam-Gu, Busan, 608-738, Korea Abstract. This paper investigates the information related to the radiated sounds during treatment using the extracorporeal shock wave lithotriptor(ESWL). First, we observe the vibration of the objects from the radiated sounds and extract the information about the change of the peak frequency from the relation between the vibration and the radiated sounds. Next, we observe the change of the peak frequency according to the position of the focus and demonstrate that it is more difficult to vibrate objects exactly at the focal point than above or below the focus of the ESWL device. Lastly, we analyze the relation between the cavitation and the radiated sounds and examine the benefiting condition for the object fragmentation. Keywords: ESWL, radiated sound, bending vibration, peak frequency, cavitation bubble.

1 Introduction The effectiveness of piezoelectric extracorporeal shock wave lithotriptor(ESWL) for the therapy of calculus has been well known in the field of urology. The piezoelectric ESWL is machinery that can generate underwater shock wave using piezoelectric elements [1-3]. The piezoelectric ESWL has an effective feature to treat calculus. The calculus after treatment is well-eliminated out of body, because it can break the calculus into fragments. In this paper, we measure and analyze the radiated sounds induced by the piezoelectric ESWL. We observe the vibration of objects, the change of peak frequency in the power spectra of the radiated sounds and the cavitation related to the use of the ESWL. From the results of the observations, it is found that the benefiting information for the therapy of calculus can be extract from the radiated sounds during the breaking process.

2 Information Related to Vibration of Object To begin with, the sounds radiated from objects induced by the piezoelectric ESWL are measured and analyzed in order to observe the proper vibration by underwater Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1055 – 1061, 2006. © Springer-Verlag Berlin Heidelberg 2006

1056

Y.S. Jang and J.H. Choi

shock wave. A bronze bar is employed as a phantom in this experiment. The length and radius of cross section of the metal bar are 6.2 cm and 0.5 cm , respectively. Figure 1 shows the waveform and its power spectrum of the radiated sound measured by a microphone when the bronze bar is struck by the shock wave of the ESWL. The range to observe the power spectrum is restricted to the audio frequency range of 020 kHz , because a skilled operator can distinguish whether the calculus is hit by the shock wave or not.

Fig. 1. The waveform and the power spectrum of the metal bar struck by the shock wave of the ESWL

In the power spectrum of Fig. 1, clear peak is observed at 7.50 kHz . It is found that this peak frequency coincides with the resonance frequency of the bending vibration, which is theoretically given by

fn =

aλ2n vl . 4πl 2

(1)

where λ1 = 4.73 , λ 2 = 7.85 , λ3 = 10.99 , … l is the length, a is the radius of cross section, and vl is the velocity of the longitudinal wave in the bar. The velocity of the longitudinal wave in bronze is 3300 m / s when it is calculated theoretically. The theoretical frequency calculated by above equation is 7.64 kHz in the case of this metal bar. This value almost coincides with the experimental value. Next, we observe the phenomena associated with the fragmentation of the object due to the piezoelectric ESWL and investigate why the radiated sound changes during the breaking process. A piece of chalk is employed as the object in this experiment because the acoustic impedance of chalk is almost equal to that of a urethral calculus. The vibrations are measured by a laser Doppler velocimetry, the results of which are analyzed by a FFT analyzer. Figure 2 (a) shows the waveforms and the spectra for the object vibrations measured by a laser Doppler velocimetry. Each waveform and power spectrum in Fig. 2 is analyzed from the 100th, 300th, 500th, 700th, 900th and 1,500th shock wave shot. It is found that the peak frequency in the spectrum vary with the number of shock wave shot, or the degree of fragmentation of the object. Figure 2 (b) shows that the peak frequency varies to the lower frequency as the number of shots increase.

Accessible Information from Radiated Sound of ESWL

1057

Fig. 2. (a) The power spectra of the vibrations detected when the shock wave is applied to the object, (b) Variations in the peak frequency with the number of shots during the breaking process

3 Information Related to Focal Position on Objects The parameter d r is defined as the relative position between the focal point and the position of the object. The value d r = 0 indicates that the center of the object exactly coincides with the focal point of the ESWL. When the center of the object is laid to the direction of the ESWL from the focal point, d r has a negative value. When it is laid to the opposite direction, d r has a positive value. Figure 3 shows the relative positions of the object and the focal point of the ESWL using the parameter d r .

1058


Fig. 3. The relative positions of the model and the focal point of piezoelectric ESWL

Fig. 4. The power at the peak frequency when the value of

dr

is changed

We observe the variation in the amplitude at the peak frequency by changing the value of d r from -10 mm to +10 mm . The result is shown in Fig. 4. The amplitude at the peak frequency decreases near the value d r = 0 . The vibration mode of the bar considered in order to analyze this phenomenon. When a bar is free to move at its ends, the center of the bar is a node of the mode 2 in the bending vibration. For this


1059

reason, it is difficult to make the center of the bar-shaped object vibrate. From these experimental results the amplitude of the peak frequency at the position d r = 0 is smaller than that at the position d r > 0 or d r < 0 in the focal region. This result demonstrates that it is more difficult to vibrate a bar-shaped object exactly at the focal point than above or below the focal point. Thus, we can estimate that the effective method to break the object is to focus above and below the focal point because it is easier to make the object vibrate exactly at the focal point.

4 Information Related Cavitation Bubble There are many studies about the performance of the ESWL and the influence into human body by the shock wave from the ESWL [1-4]. It is an important issue that cavitation is always considered together with shock wave [5-7]. A medium of the shock wave is related to the cavitation phenomenon. Therefore, we investigate the break efficiency and the power of the shock wave on focal region due to the medium of the shock wave. First, we observe the break efficiency by changing d r as above mentioned. Chalk is employed as the object of this experiment. We compare the number of the shock wave to shot until the time the chalk is broken to two pieces with d r change. Undegassed water and degassed water is used as the medium of the shock wave.

Fig. 5. The number of shots to break due to the positions of the focal point on the object

1060


The result of this experiment is represented as shown in Fig. 5. From this result, we observe that there is a difference of the shot number until the break by nature of the medium. This result demonstrates that the cavitation bubble affects on the break efficiency of the ESWL.

Fig. 6. The comparison of the ESWL power in the focal region due to the medium

The case of degassed water is better medium than that of undegassed water when the ESWL is used on the therapy of calculus. Next, the power of the shock wave is examined in the focal region by changing a kind of the medium for examining closely the influence of the cavitation bubble. The power of the shock wave is measured on the focal region using a special hydrophone. Figure 6 shows the result to observe the power of the shock wave in both media. Clearly, the power in the degassed water is stronger than that in the undegassed water on the focal region. These experimental results related the cavitation demonstrate that the shock wave of the ESWL in degassed water is more powerful.

5 Conclusions This paper report the information related to the radiated sounds from the ESWL. It contains the vibration of the object, the focal position on the object and the relation between the break efficiency and the cavitation. In the experiment related to the vibration, the peak frequency of the bending vibration varies with the number of shock wave shots. Accordingly, it is possible to hear sounds of lower frequency as the object is gradually fragmented.


1061

We can obtain the information related to the position of the focal point in the next experiments. The amplitude of the peak frequency at d r >0 and d r 0 ), and bij' = 0 (when bij ≤ 0 ). '

R = (r1 , r2 , r3 L, rn ) is obtained.

From this, the matching rate vector where rj = r ( k j ) .

4.1.2 Similarity Appraisal The similarity appraisal is the key of detection. In the multi-model matching, the importance of each attribute in the similarity appraisal is different, therefore, various attributes need virtual different weight value. The model attribute similarity can be expressed as: m

Sim (a , b) =

∑( f

a i

⋅ f i b ⋅ wi2 )

(5)

i =1

m

∑( f

a i

⋅ wi ) 2

i =1

m

∑( f

b i

⋅ wi ) 2

i =1

where wi expresses the weight of ith attribute vector,and

m

∑w i =1

i

= 1.

The matching rate has considered external factor of the biology model, but the similarity considered intrinsic connection of the biology model. 4.2 Principle of the Model The model receiver firstly submits the concrete environment data to establishe the environment model; Submits the biological characteristic data and the initial structure data to establish the biological initial geometry model; Submits the biology growth characteristic data and the biology motion state data to establish the living model. Afterwards, under the actuation of time series, the simulation cybertron calculates the change of shape structure, and controls the geometry model to complete the calculation of these changes, thus obtains the three dimensional structure for biological growth process at a certain time. Afterwards, virtual biology detection mechanism is executed, if in the scope of matching degree and similarity degree, the virtual model is produced, and according to the specific request, the next step simulation is carried on. Otherwise, return to tell it.

5 Experiment and Result Analysis In order to confirm the validity of the model, we introduce the model to the biology class of senior high school in a section of ’human and biosphere ’. We construct the process of a seed germinates, in which the seed is affected together by the model of soil, moisture, illumination, and earthworm. The overall system is realized through PC programming utilizing VC++6.0 and OpenGL.

Virtual Biology Modeling Method Based on Validity Confirmation Mechanism

1125

Fig. 3. A branching model

Fig. 4. Protract the final state of seed germination

Fig. 5. The entire state change driven by one kind of model (e.g. light model)

Table 1 describes the required model parameters and their value in the entire simulation process. Table 1. Parameter Water (w) Soil (S) Biology(B) Light (L) Geometry(G) K

Value 3 2 4 3 28 20

Note Water model of the system Soil model of the system Animal model of the system Light model of the system Geometry model of the sysytem Parameter

1126

Y. Liang et al.

In Table 2, NDM indicates the constructed virtual living model without using the multi-model matching detection method. In Table 3, DM indicates to use this method. WS indicates that the initial condition only applies the light model and the geometry model in this plan. According to the model parameter given in Table 1, WOS,WLOS and WSOLG can in turn indicate which concrete condition they use,respectively. The protracking time is the sum of establishment initial condition and finally obtaining the model result. Average protracking time is the mean of protracking model time under the same initial condition. Fidelity is used to scale protracking similarity. If there are bracket and plus sign behind, they indicate that the protrack accords with the actual need. Table 2. NDM (Testing sequence) Condition Average control time Fidelity

6/32 WS 30s 0.92(+)

8/32 WOS 68s 0.81(+)

8/32 WLOS 119s 0.5

10/32 WSOLG 125s 0.37

Table 3. DM (Testing sequence) Condition Average control time Fidelity

6/32

8/32

WS WOS 38s 66s 0.951(+) 0.914(+)

8/32 WLOS 90s 0.921(+)

10/32 WSOLG 112s 0.918(+)

Through test and contrast for many times, we know that, if there is no detection mechanism, the more we use model in protract, the longer the protrack time is. With this mechanism, we can avoid forgotting the individual condition in the experiment, or avoid producing the model in the situation that the condition hypothesis and the nature rule do not match, which may reduce the meaningless time consumption caused by the wrong model based further research, and enhance the efficiency of the model in the teaching application.

6 Application Significance in Teaching On the one hand, this kind of model enables student to understand scientific knowledge, and change student's cognition level to abstraction. By abstracting the substantive characteristics of prototype and simplifying complex prototype object, they can construct the model which reflects the prototype essence relationship. On the other hand, the model constructing process is also a process of science exploration. In this process, the students need to determine the research object, establish parameter,

Virtual Biology Modeling Method Based on Validity Confirmation Mechanism

1127

choice the research method, and finally validate whether the experiment result is consistent with the actual,which fully display the student's independent learning capability and innovation ability.

7 Summary The paper proposes a new method of virtual biology model construction based on valid confirmation mechanism. According to the model system structural, we design the model cybertron, the model detector and the multi-model of geometry, environment and biology. It verifies the rationality and authenticity of biological model through the condition enactment combined with the natural biological logic relations. And its celerity and accuracy have been confirmed in the biological experiment class. However, along with the scientific development, as well as more and more people pay attention to the model match, our research needs to ceaselessly deepen, which is the work we will have to do from now on.

References 1. Laurent MIGNONNEAU and Chrism SOMMERER, Designing Interfaces for Interactive Artwo~s[EB OL] 2. Lindenmayer, A Developmental systems without Cellular Interactions. their languages and Grammars, [J]InJournal of theoretical Biology, (1971)30: 455—484 3. deReffye P,Edelin C, Francon Jetal. Plant models faithful to botanical structure and development[J]. Computer Graphics, 1988, 22(4):151-158. 4. Bouchon J, de Reffye P, Barthelemy D. Model is ationet Simulation del' Architecture des Vegetaux [M]. Paris, INRA, 1997.23 441-487 5. Godin C, Carglio Y. A multiscale model of plant topological structures [J]. theor Bio, 1998, 84(3): 1-46. 6. Zhao xing, de Reffye Philippe, xiong fanlun. virtual plant growth double criterion automaton model [J], 2001, 24(6): 608-615. 7. Mandelbrot.B.B. The fractal geometry of nature [M], Free man,San Fransisco, 1982. 64(2):6-34 8. Mˇech, R., and Prusinkiewicz, P. Visual models of plants interacting with theirenvironment. Computer Graphics (SIGGRAPH ’96 Conference Proceedings) (August 1996), 397–410. 9. Prusinkiewicz, P., James, M., and Mˇech, R. Synthetic topiary. Computer Graphics (SIGGRAPH ’94 Conference Proceedings) 38 (July 1994), 351–358.

／．

：

Virtual Reconstruction of Two Types of Traffic Accident by the Tire Marks Xiaoyun Zhang, Xianlong Jin, and Jie Shen Shanghai Jiaotong University {Jxlong, Shenjieihn, General_zhang}@sjtu.edu.cn

Abstract. With the development of simulation technology, the numerical methods can help to reconstruct the traffic accidents. Based on the dynamic models of collisions, a Momentum/Energy method to predict the pre-impact motion was introduced through the optimization of the rest positions of the vehicles. With the help of Pc-Crash, the method of traffic accident reconstruction can be achieved, and several simulation runs will be performed, to find a solution, where post impact trajectories and rest positions correspond with the real accidents. In the paper, two kinds of collision accidents, the car crash to the wall and the car crash to another car, are analyzed based on the method. The simulation results show that applying the tire marks in the accident scene and the Momentum/Energy method can provide the scientific and numerical references for the analysis and judgment of the traffic accidents. Keywords: Vehicle; Traffic accident; Virtual reconstruction; Skid marks.

1 Introduction With economy developing in China, more and more family own their private cars as the necessary living tools. Compared with the years of 2001 and 2002, the number of traffic accidents, deaths, injury and direct economic loss increased 2.14%, 3.26%, 2.85% and 7.66%. Compared with the years of 2002 and 2003, these numbers decreased 13.7%, 4.6%, 12.1% and increased 1.4%, respectively, in 2004, they increased 14.9%, 4.9%, 8.6% and 17.6%. According to the statistical data, the drivers breaking the traffic rulers are the main reasons in these traffic accidents. The reasons include over speed, wrong way, driving without licence, drunk driving and tired driving, etc. More and more traffic accidents increase the pressure of the relative department. On the other hand, with the developing and opening, stricter requirements for the appraisal of traffic accidents are needed: international conventions; more rapidly responding; standard operation routines. One complete traffic accident disposal includes three parts: survey at the accident scene, accident analysis and responsibility ascertaining. The main tasks of the accident scene survey are: measuring, drawing the accident scene draft and taking photos at the accident scene, etc. Accident analysis will take the survey as the basis and analyze the accident reasons. Based on the accident analysis and according to the relative codes, the accident responsibility can be decided at last. Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1128 – 1135, 2006. © Springer-Verlag Berlin Heidelberg 2006

Virtual Reconstruction of Two Types of Traffic Accident by the Tire Marks

1129

The skid marks are the main research objects in accident reconstruction. In the paper, the dynamic models were studied. And the method to predict the pre-impact motion of the vehicle is introduced. Then two typical accidents are reconstructed by the methods.

2 The Optimization Method Before the optimization process can be done, a quality function has to be found. And before defining a quality function for the simulation results, the parameters, which can be used to describe the real accident, have to be defined. [1-3] 2.1 The Collision Model The impact center is determined by a fundamental principle, the equation of linear and angular impulse-momentum. An x-y vehicle coordinate system is selected such that the origin is located at the center of gravity. The dynamic impact model uses numerical integration of Newton’s second law in the form of differential equations whereas the analytical impact model uses the law of conservation of momentum and is presented by a set of algebraic simultaneous equations without reference to forces. As defined the impact can be divided into two phases: the “compression” phase and the “restitution” phase. At the end of the compression phase the velocities for both vehicles at the “impulse point” are identical in case of a full impact. Due to a certain elasticity of the vehicle structures, the two vehicles will separate again. Linear momentum and angular momentum are conserved, and energy loss is accounted for with a coefficient of restitution. The “coefficient of restitution” is defined as ratio between restitution momentum and compression momentum, and the range of the coefficient lies between 0~0.3. [1] 2.2 Definition of Post Crash Movement Depending on the available data from the accident scene, some of these parameters may be uncertain. In many accident situations the approximate location where the vehicles collided can be found due to the tire or scratch marks, the shatter fields or the dust marks. If the vehicles have not been moved from their rest positions when the scene investigations were made, these positions and the car directions are well defined. If skid marks were found on the accident scene, intermediate vehicle positions can be defined on the post impact trajectories. Therefore, from the trajectories, the impact position, rest positions and intermediate positions can be used to define the accident situation. [2] 2.3 Optimization Strategies As for the optimization strategies, several different algorithms like the Coordinate Approach by Gauss-Seidel, the Simplex method, the Gradient and Newton approach, Monte Carlo and Evolutionary methods deal with the problem of multidimensional

1130

X. Zhang, X. Jin, and J. Shen

optimization. Not all these methods are applicable, because some of these methods need derivatives of the target function to the input parameters. Thinking of the factors of robustness, numerical stability and high progress rates, the Evolutionary method, that is, the genetic algorithm is chosen. [2, 4]

3 Application Cases Using the optimization methods and the Pc-Crash software, two kinds of traffic accident, car crash to the wall and one car crash to another car, are reconstructed with the information of the tire marks in the scene. 3.1 The Case of One Car Crash to the Wall The accident happened in Hainan Province in 2001 and the photo of the accident scene as shown in Fig. 1. The weather of the day was sunny and the road surface was blacktop.

Fig. 1. The photo of the accident scene

In the accident, the high-speed test of the vehicle in the design process was just finished, and the car was driving to the park. The driver operated error lead the car crash to the wall. Because the seat belt was used, the driver was not injured seriously. With the help of photogrammetry programs, the three dimensional model of the accident scene (as shown in Fig. 2) can be obtained from the photographs. Then the

Fig. 2. 3D model of the accident scene


1131

three-dimensional model of the scene can be used as an input of the software Pc-Crash, and the parameters that defining post-impact movement of the accident are available in the model data. In the accident, the degree of the angle between the tire mark and the wall is 17 and the position of the car at the state of stop can be decided. After trajectories optimization for this accident, the pre-impact velocity can be calculated to be 50km/h, and the simulation error is 1.0%. As shown in Fig. 3, this result is quite coincident with the accident scene.

Fig. 3. Reconstruction of the accident

The researcher also can view at the position of the driver by the Open GL technique, just as shown in Fig. 4.

Fig. 4. At the view of the driver

(1) v = 50 km/h

(2) v = 35 km/h

(3) v = 27 km/h

(4) v = 0 km/h

Fig. 5. The course of the car crash to the wall

1132


The movement process of the car crash to the wall was shown in Fig. 5, and the velocity of the car at the different stage was also shown in the figure. 3.2 The Case of the Car Crash to Another Car One vehicle-to-vehicle collision traffic accident that happened in Shanghai in 2004 was reconstructed on computer. The weather of the night is rainy and the road surface is blacktop. The photo of the accident scene was show as the Fig. 6. Before defining a quality function for the simulation results, the parameters that can be used to describe the real accidents have to be defined. In the accident situations the approximate location where the vehicles collided can be found due to the tire or scratch marks, the shatter fields or the dust marks. On the other hand, the overlap of the vehicles during the impact can be determined using the remaining deformation of the vehicles after the impact.

(a) The photo of the vehicle (No.1)

(b) The photo of the vehicle (No.2)

Fig. 6. Photos of the accident scene

In the accident, the stop position should be gotten from the drawing of the accident scene, and from the photos of the accident vehicles, it is known that the left side tires of the vehicle No.1 all exploded due to the collision, so it can be concluded that the tractor of the vehicle No.2 slide and collided continuously with the left side of the carriage of the vehicle No.1, and causing the tractor and the trailer of the vehicle No.2 to form a

Fig. 7. The CAD model of the accident scene


1133

larger angle at the stop position. The vehicle No.1 turned right after the collision and knocked down the tree in the roadside finally. In the accident, the driver of the vehicle No.2 died, and the driver of the vehicle No.1 was injured seriously. The CAD model of the accident scene is shown in the Fig. 7. The information of the two vehicles is show in the table 1. Table 1. The information of the two vehicles

Vehicle Overall length/mm Overall width/mm Overall height/mm Wheelbase/mm Total mass/mm

NO.1 16500 2480 3020 13410 41000

No.2 tractor 5750 2490 2780 3410 6500

No.2 trailer 7196 2480 1500 5550 23520

According to the theory of vehicle dynamic models and the method of trajectories optimization, the results of the accident reconstruction were shown as the Fig. 8. On the other hand, the velocity of the vehicles at the moment of contact can be calculated in the simulation result.

(a) Position of the first crash

(b)Position of the second crash

(c) Position of the third crash

(d) The forth crash to the tree

Fig. 8. The result of the accident reconstruction

In the figure 8(a), crash happened between the tractor of the vehicle No. 1 and the tractor of the vehicle No.2, at the moment of contact, the velocity of the tractor of vehicle No.1 is 44.0km/h, and the velocity of the tractor of vehicle No.2 is 45.0km/h. In the figure 8(b), crash happened between the trailer of the No.1 vehicle and the tractor of

1134


the No.2 vehicle, the velocity of the trailer of vehicle No.1 is 41.1km/h, and the velocity of the tractor of vehicle No.2 is 36.6km/h. In the figure 8(c), crash happened between the trailer of the No.1 vehicle and the trailer of the No.2 vehicle, the velocity of the trailer of vehicle No.1 is 32.0km/h, and the velocity of the tractor of vehicle No.2 is 9.2km/h. In the figure 8(d), crash happened between the tractor of the No.1 vehicle and one tree, the velocity of the tractor of vehicle No.1 is 18.8km/h. The optimization error of the simulation result is 9.0%, usually optimization error below than 10% is allowed. In this case, the reason of a big optimization error is that the rest positions of the two vehicles are known, but only litter tire marks were kept well. 3.3 The Range of Application Based on the information of the skid marks, collision accidents analysis can be finished quickly, however these methods have strict requirements on the accidents scene. Now most advanced vehicles have equipped with ABS, so the skid marks are much fainter or non-existent. On the other hand, in collision accident the driver maybe damage the accident scene because of unconsciousness or being nervous. For example, the driver may move the vehicle to the roadside, or moves on slowly to stop, or back slowly to stop, without following “The law of road traffic security, People’s Republic of China”. All these will badly damage skid marks and other key factors in accidents, then the spot survey and justice appraisal will be influenced. Moreover, the weather (for example rain) and other vehicles (pedestrians), even conscious action will damage the spot. So accident analysis with skid marks has its limitations. To reconstruct the accident without skid marks, the new methods: vehicle body deformation, event data recorders etc, and new devices will be applied in accidents to resolve the problem mentioned above. [5-7]

4 Conclusions This paper studies the dynamic models of collision accidents, based on the study, a method to predict the pre-impact motion of the vehicle is introduced through the optimization of the post impact trajectories and rest positions correspond with the real accident. Then one typical collision accident case was simulated based on the method, it is proved to be effective on analysis of this kind of accidents, and so can provide a scientific foundation for accident judgments.

References 1. Steffan H, Moser A. The Trailer Simulation Model of PC-CRASH. SAE paper 980372, USA 2. Moser A, Steffan H. Automatic Optimization of Pre-Impact Parameters Using Post Impact Trajectories and Rest Positions. SAE paper 980373, USA 3. Steffan H, Moser A. The Collision and Trajectory Models of PC-CRASH. SAE paper 960886, USA


1135

4. William E C, Moser A. Reconstruction of Twenty Staged Collisions with PC-Crash's Optimizer. SAE paper 2001-01-0507, USA 5. Zhang X Y, Jin X L, Sun Y, et al. Vehicle Crashworthiness Simulation Based on Virtual Design of Auto-body, Journal of Dong Hua University, 2004, 21(2): 63~68 6. Zhang X Y, Jin X L, Qi W G, et al. Virtual Reconstruction of Vehicle Crash Accident Based on Elastic-plastic Deformation of Auto-body, Key Engineering Materials, 2004, 274-276: 1017~1022 7. Zhang X Y, Jin X L, Qi W G, et al. Crashworthiness Simulation and Structure Optimization Design of Auto-body Based on Finite Element Method, Journal of Systems Engineering and Electronics, 2004,15(4): 112~116

A Seamless Visualizaton Model of the Global Terrain Based on the QTM Xuesheng Zhao1,2, Jianjun Bai1, Jun Chen2, and Zhilin Li3 1

Department of Surveying and Land Science China University of Mining and Technology (Beijing), Beijing, China, 100083 [email protected] 2 National Geometrics Center of China, No.1 Baishengcun, Zizhuyuan, Beijing, China, 10004 [email protected] 3 Department of Land Surveying and Geo-Informatics The Hong Kong Polytechnic University, Kowloon, Hong Kong Tel.: (+852) 2766 5960; Fax: (+852) 2330 2994 [email protected]

Abstract. A seamless & adaptive visualization model of global DEM based on the QTM (Quaternary Triangular Mesh) is made to approximate to the Earth terrain in this paper. The approaches start with an Octahedron QTM partition based on the spherical surface. In order to improve the operation efficiency, the global DEM array points are organized as a hierarchy of Diamonds and the corresponding algorithms on indexing mechanism, coding scheme, and neighbor finding based on linear quadtree are given in details. Then, a dynamic operation method of the multi-resolution Diamond datasets in visualization is approached. Furthermore, an adaptive simplification rule of hierarchical triangles in Diamond data blocks is presented, in which an idea of the Binary Triangle Tree is introduced to form a continuous DEM mesh on the edges between different resolutions. In the end, the experiment and analysis are done with the GTOPO30 dada. The results is smoothly and receivable. Keywords: Visualization model, DEM, QTM, Diamond, Adjacent index.

1 Introduction Real-time and realistic visualization model of the Earth terrain is the significant component in Digital Earth and has been one of the research hotspots in geography and spatial information science for many years. The regular or irregular grid Digital Elevation Models (DEM) based on the idea of map projections are effective traditional methods to modeling the terrain on the local spherical surfaces as a flat, and many corresponding algorithms are presented over the last decade [Lindstrom et al. 1996; 2002; Duchaineau et al. 1997; Hoppe 1996]. However, the spherical surface is not as topologically equivalent as planar Euclidean space in geometry. Although the traditional DEM grid may support an individual project or small-scale terrain visualization modeling, they do have some significant drawbacks for modeling the large-scale or the whole Earth surface, such as unacceptable geometric distortions, Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1136 – 1145, 2006. © Springer-Verlag Berlin Heidelberg 2006

A Seamless Visualizaton Model of the Global Terrain Based on the QTM

1137

data broken or overlap, space inefficiency, and difficulty with data sharing between different map projections [Gold and Mustafavi 2000; Kolar 2004]. To overcome these deficiencies, a number of approaches to subdividing the surface of the earth in a continuous and hierarchical way for the global terrain visualization have been suggested in recent years, such as latitude & longitude grids [Gerstner 2003; NIMA 2003; Bjroke et al. 2004] and the adaptive subdivision grids [Lukatela 2000; Kolar 2004], etc. These global discrete grids tessellate the Earth’s surface directly with non-overlapping (or broken) cells. They can be simulated the earth terrain when they are recursive partitioned to a certain degree and can resolve those problems caused by map projections [Dutton 1999]. However, latitude/longitude grids do not have equal area cells, which is important for many applications and a lot of redundancy data is occurred near the pole areas [Sahr and White 1998]. The shape and area of those adaptive subdivision grids are irregular and ununiformity in a same subdivision. Both of them are difficult to manage large volume of global DEM data and to manipulate multi-resolution data visualization efficiently. It is desirable to have grids consisting of highly regular regions with evenly distributed elevation points. This paper aims to present a QTM (Quaternary Triangular Mesh)-based method for the construction of a visualization model of the earth terrain seamlessly and adaptively. The reason is that the QTM grid structure is uniformity, hierarchical, and numerically stable everywhere on the spherical surface. Its hierarchical grid structure can be used to efficiently manage multi-resolution global data, and it allows spatial phenomena to be studied at different levels of detail in a consistent fashion across extensive regions of the sphere [Sahr et al. 2003; Chen et al. 2003]. Another reason is that the QTM grids would not only allow the same structure to be used over a wide range of spatial resolutions and efficiently load only needed segments [Faust et al. 2000], but also allow the presentation of data at multiple levels and offer several major advantages, such as being unique and domain-independent, appropriately indexed or linearized grids express spherical surface location in a single string, preserving geometrical integrity both locally and globally, and making resolution explicit in the length of the string [Goodchild & Yang 1992], etc. Following this introduction is a modeling principle of the global DEM grids based on QTM is then presented in briefly. This is followed by the presentation of a hierarchical data organizing and adjacent indexing algorithm of the Diamond-based structure in details. Then, the dynamic paging method of Diamond blocks is given. An adaptive simplification rule of hierarchical triangles in Diamond data blocks is presented by recursive edge bisection. Following that, GTOPO30 data (which are from Courtesy of US Geological Survey) of the whole Earth are used to test these methods and algorithms presented in this paper. Finally, the conclusions and recommendation for future work will be presented.

2 Modeling Principle of the Global DEM Grids Based on QTM The QTM structure used in Fekete [1990], Dutton [1989; 1999], and Goodchild and Yang [1992], are all based on inscribed octahedron. The reason is that its vertices occupy cardinal points and its edges assume cardinal directions, following the

1138

X. Zhao et al.

equator, the prime meridian, and the 90th, 180th and 270th meridians, making it simple to determine which facet a point on the planet occupied [Dutton 1989]. When a triangle is subdivided, the latitudes/longitudes pairs of any two of its three vertices are averaged to yield edge midpoint locations. Clearly after each level of subdivision, the triangles become smaller, and at the 21st level of subdivision, their size is approximately 1m, going down to 1 cm at 28th level. Fig.1 illustrates subdivisions at levels 1, 2 and 3.

(a) level 1

(b) level 2

(c) level 3

Fig. 1. QTM partition of spherical facet based on octahedron [Dutton 1989]

This partition scheme preserves the properties of uniformity, hierarchical, and numerically stable everywhere on the spherical surface. The vertices, which make up QTM, are called QTM vertices whose position can be identified by latitude/longitude coordinate. Thus, the global terrain can be represented by the array of elevation values of the vertices.

3 Hierarchical Organizing Method Based on the “Diamond ”s It is well known that the QTM grids are hierarchical triangles and do not have uniform orientation (some triangles point up while others point down). Many algorithms defined on triangle grids are more complex because of taking into account triangle orientation. White [2000] notes that pairs of adjacent triangle faces may be combined to form a Diamond, and this Diamond may be recursively sub-divided in a fashion analogous to the square quadtree subdivision. 3.1 Indexing and Coding of Diamond Based on Linear Quadtree According to this idea, the whole earth surface can be represented as a quadtree that the root has four children node (four base-Diamonds as shown in Fig.2), and the interior node has four children node (four child-Diamonds). The global DEM data is organized based on levels of Diamond as basic storage unit, which correspond to different resolution traditional map. In the file system, they are the binary terrain data files in which the height values of the QTM vertices are stored in Binary Large Object (BLOB).


1139

Fig. 2. The DEM data organizing based on the levels of Diamonds

The surface of the earth can be found an expression in a linear quadtree and only the locations of leaves need to be registered in the storage process. The leaves can be labeled by its Diamond code and indexed according to the Z space-filling curve (shown as Fig.3). Each Diamond be assigned a Morton code MQ=q0q1q2….qk, in which q0 is the quadcode of the base Diamond assigned a numerical label 0,1,2 or 3 according its location on the surface of the earth. It is illustrated as: q0 = 0 (90° > λ ≥ 0°), q0 = 1 (180° > λ ≥ 90°), q0 = 2 (270° > λ ≥ 180°), and q0 = 3 (360° > λ ≥ 270°). Each Diamond will be substituted by four smaller Diamonds when performs subdivision and can be labeled respectively through adding an additional digit 0, 1, 2 and 3 according to their location (left, down, up and right) in the parent Diamond.

Fig. 3. The Morton code and Z space filling curve of Diamonds

3.2 The Neighbor Finding of the Diamond It is easy to determine the Morton codes of either the sons or the parent of a Diamonds according to the properties of the Morton codes. The sons of a Diamond are determined by appending the digits 0,1,2 and 3 at the end of the Morton codes, and the address of the parent of a Diamond is obtained by simply discarding the trailing quaternary digit. In order to determine adjacent neighbors of the Diamond, firstly the conversion process between the Morton codes and row/column number of

1140

X. Zhao et al.

the Diamond must be done by formula (1), which is introduced in details in paper [Lee and Samet 2000]. The correlation between them is shown as following as Fig. 4:

M Q = 2Ib + J b

(1)

Fig. 4. The correlation between quaternary Morton code and row/column number

There are three cases in the neighbor-finding of the Diamonds: the Diamond located in the inside of the base Diamond (quadrant), the Diamond located in the boundary of the base Diamond, and the Diamond located in the corner of the base Diamond. As the Morton codes of adjacent Diamond inside the same base Diamond can be calculated easily. To the briefly, only the row/column numbers of adjacent Diamonds in the different base Diamond is presented as table 1. Similarity, the row/column numbers of neighbor Diamonds in Right boundary, South corner and East corner can be figured out. Moreover, the Morton codes of these neighbors can be calculated by formula (1). Table 1. Row/column numbers of adjacent Diamonds in the different base Diamond

Diamond location

Left(up) boundary Left(down) boundary North corner East corner

Row/column Row/column numbers of numbers Edge-neighbor Diamond (i, 0 ) (0, j)

(I, 0) (I, J)

(I, I-i) (I-j, J) (I, 0), (I, 0), (0, 0)

Row/column numbers of Vertex-neighbor Diamond

(I, I-i-1), (I, I-i+1) (I-j-1, J), (I-j+1, J) (I, 1), (I-1, 0), (I, 0) (1 0), (0, 1)


1141

3.3 The Dynamic Operation Algorithm of Diamond Blocks As the amount of dataset is too much to be stored in main memory at one time, in the visualized operations of terrain data, only several Diamonds in a given level of data are added in according to proportion between the screen space distance and its corresponding object space distance. Usually the screen resolution is a constant. With the changing of the viewpoint and angle of view, the object space distance corresponding to the adjacent pixel in the screen is changed. The object space distance D corresponding to the adjacent pixels can be calculated by: D =

2 × d × tg l × λ

α 2

(2)

In which λ is number of the pixels per distance unit in the projection plane; d is the distance between the view and the object plane; α is angle of view; and l is the length of the projection plane. For all Diamonds located in the same level, only the Diamonds inside the view-frame are selected to add, and the Diamonds located outside are not considered for the efficient rendering. Diamonds are chosen as the active areas based on considerations for memory size of available PC and desired range of view. This amount of terrain data is in the main memory at any given time and available for operation. When the operation algorithm is initialized, the viewpoint is centered on the 4 Diamonds. A bounding box is established around the screen center. When the user reaches the bounding box in any direction, memory space is freed in the direction opposite of travel, Diamonds terrain is added in the direction of travel (shown as Fig. 5).

Fig. 5. The dynamic adding or removing of Diamonds

4 An Adapting Simplification Method of DEM Data Block In order to improve the efficiency in visualization, the Diamonds data blocks located in the memory need to be simplified to multi-resolution triangle grids base on the

1142

X. Zhao et al.

terrain propertied. In this model, it cannot preserve the continuity of the terrain surface along the edges between the different subdivision levels (shown as Fig.6). Traditionally, the continuity of the terrain can be kept by tessellating the triangles in the higher level.

Fig. 6. Hierarchical QTM and the crack caused by different subdivisions

However, the traditional method exists some deficiencies. Firstly, the subdivisions need to be controlled by people and the operation can be done only when the difference between the adjacent levels are no larger than one. Secondly, the regularity of tessellation grids can not be preserved, in which some grids are QTM and others are irregular. In order to overcome this problem, an adaptive idea of Binary Triangle Tree [Lindstrom & Pascucci 2002] is introduced. Firstly, it is supposed that the edge that wants to be subdivided is called base edge in one triangle, the two triangle that share the same base edge is made up a pair of triangles (shown as triangle a and b in Fig.7a). The simplification principle of the adapting grid is as following (shown as Fig.7):

(a)

(b)

Fig. 7. Bisecting partition of triangles

Step 1, the triangle is part of a Diamond---the triangle and its Base Neighbor is split simultaneously, such as triangle a and b in Fig.7a. Step 2, the triangle is not part of a Diamond--- Firstly, its Base Neighbor is split and then do step 1. For example in Fig.7b, if triangle c is to be split, its base neighbor d must be split into d1, d2 and then the triangle c and d2 are split simultaneously. Step 3, the process is to be done recursively until its error exceeds the tolerance value.


1143

By recursively subdivision based on Binary Triangle Tree, an adapting, continuous, and hierarchical triangular mesh is formed in the data blocks and their vertices coincide with the vertices of QTM completely. The problem of discontinuity in different subdivision levels is overcome and the number of triangles of paged Diamond in visualization is reduced greatly.

5 Experiments and Analysis The experiment results of the hierarchical algorithm of the global terrain visualization are analyzed in this section. In order to be organized into Diamond based file, the GTOPO30 dataset (from Courtesy of US Geological Survey) should be translated into a QTM data structure before visualized operation. In this section, it

(a) τ =1; N=70912

(b) τ =2; N=33786 Fig. 8. The visualization of global DEM at four different error tolerances (left: triangulation grid map, right: shaded map)

1144

X. Zhao et al.

is interpolated onto 4 Diamonds, and then 16 Diamonds, 64 Diamonds and 256 Diamonds dataset respectively; each includes 2049*2049 grid vertices. Each elevation value in the Diamond file represents the height value in meters above sea level and is stored using 2 bytes as a signed integer. Ocean areas are marked by nodata values. Compare with the GTOPO30 Data, the volume of the QTM based DEM data is reduced about a half of its initial. It can be illuminated by formula (3) (Q is the volume of the QTM based DEM data, while G is the volume of the GTOPO30 DEM data):

Q G

=

4n 2 − 4(2n − 1) (n − 1) 2 1 = 2 ≈ (2n − 1) × (n − 1) × 4 2n − 3n + 1 2

(3)

The number of the triangles on the Earth is N =33445532 before simplification and it decreases greatly by using different error tolerances, such as τ =1, 2 respectively. The correspondent visualized results are shown as Fig.8 (All ocean area are not display). Number of triangle is decreased greatly (left figures of Fig.8a and 8b) with their error tolerances increase, but the visualized results among them have no obvious change (right figures of Fig.8a and 8b). That is to say, in order to improve visualized efficiency, a simplified model can be applied by selecting a suitable error tolerance τ.

6 Conclusions and Future Work A global multi-resolution Digital Elevation Model and a feasible solution for its visualization and management remains a challenging vision. In this paper, a seamless & adaptive visualization model of the global terrain based on Modified-QTM is approached. Not only the deficiencies of data broken or overlap caused by traditional DEM are overcome, but the distortion in area and/or shape and a lot of redundancy data occurred in the latitude/longitude grids are avoided as well. One contribution in this paper is developing a quadtree structure of Diamonds to organize the global DEM data on QTM vertices, which can be storied and indexed using linear quadtree. The Diamond is as topologically equivalent to the square, which has some excellent properties, such as radial symmetry, translation congruence, and uniform orientation, and the corresponding algorithms on indexing mechanism, coding scheme, and neighbor finding are simple and effective. In visualization operation, when the viewpoint move at a change distance to the Earth surface, the neighboring Diamonds can be indexed and paged in or out dynamically. Another contribution in this paper is presenting an adaptive simplification model of hierarchical triangles in DEM data blocks based on Binary Triangle Tree. This adaptive triangle mesh is regular and uniform. The problem of discontinuity in different subdivision levels is overcome and the number of triangles of the diamond added in visualization is reduced greatly. The experiment illustrates that the results is smoothly and receivable. Acknowledgement. The works described in this paper was substantially supported by an award from the Natural Science Foundation of China (under grant No.40471108) and by a project from the Hong Kong Polytechnic University (Project No. G-U045).


1145

References 1. Bjørke J., Grytten J., Morten H. and Stein N., Examination of a Constant-Area Quadrilateral Grid in Representation of Global Digital Elevation Models. International journal of geographic information science, 18 (7): 653-664, 2004. 2. Chen J., Zhao X.S., and Li Z.L., An Algorithm for the Generation of Voronoi Diagram on the Sphere Based on QTM. Photogrammetric Engineering & Remote Sensing. 69(1):7990, 2003. 3. Clarke K., Dana P. and Hastings J., A New World Geographic Reference System, Cartography and Geographic Information Science, 29(4): 355-362, 2002. 4. Duchaineau M., Wolinsky M., Sigeti D., Miller M., Aldrich C., and Weinstein M., ROAMing Terrain: Real—Time Optimally Adapting Meshes, in Proc. Visualization '97. Oct., pp.81-88, 1997. 5. Dutton G., Modeling Locational Uncertainty via Hierarchical Tessellation. In Goodchild M.F. and Gopal S., editors, Accuracy of Spatial Databases, Taylor & Francis. pp125-140, 1989. 6. Dutton G., A hierarchical Coordinate System for Geoprocessing and Cartography, Lecture Notes in Earth Sciences, Springer-Verlag, 230pp, 1999. 7. Faust N., Ribarsky W., and Jian g T.Y., Real-Time Global Data Model for the Digital Earth. http://www.ncgia.ucsb.edu/globalgrids/papers/faust.pdf., 2000. 8. Fekete G., Rendering and Managing Spherical Data With Sphere Quadtree. Proceedings of Visualization '90. IEEE Computer Society, Los Alamitos, CA. 176-186, 1990. 9. Gerstner T., Multi-resolution Visualization and Compression of Global Topographic Data. Geoinformatica. 7(1): 7-32, 2003. 10. Gold C.and Mustafavi A., Towards the Global GIS. ISPRS Journal of Photogrammetry & Remote Sensing, 55(3): 150-163, 2000. 11. Goodchild M.F., and Yang, S.R., A Hierarchical Data Structure for Global Geographic Information Systems. Computer Vision and Geographic Image Processing, 54(1):31-44, 1992. 12. Hoppe H., Progressive Meshes. SIGRAPH 96 Conference Proceedings, pp99-108, 1996. 13. Kolar J., Representation of The Geographic Terrain Surface Using Global Indexing. Proceeding of 12th International Conference on Geoinformatics. Sweden, 321-328, 2004. 14. Lee M. and Samet H., Navigating through Triangle Meshes Implemented as Linear Quadtree, ACM transactions on Graphics, 19(2): 79-121, 2000. 15. Lindstrom P., Koller D, Faust and Gregory., A Real-Time Continuous Level of Detail Rendering of Height Fields. SIGGRAPH’96 Conference Proceedings, pp109-118, 1996. 16. Lindstrom P. and Pascucci V., Terrain Simplification Simplified: A General Framework for View-Dependent Out-of-Core Visualization. IEEE Transactions on Visualization and Computer Graphics, July-September 8(3):239-254, 2002. 17. Lukatela H., A Seamless Global Terrain Model in the Hipparchus System, Http://www.geodyssey.com/global/papers, 2000. 18. NIMA, Digital Terrain Elevation Data, http://www.niama.mil/, 2003. 19. Sahr K. and White D., Discrete Global Grid Systems. Computing Science And Statistics, Interface Foundation of North America. Inc., 30pp, 1998. 20. Sahr K., White D. and Kimerling A., Geodesic Discrete Global Grid Systems, Cartography & Geographical Information Science, 30(2): 121-134, 2003. 21. White D., Global Grids From Recursive Diamond Subdivisions of The Surface of an Octahedron or Icosahedron. Environmental Monitoring and Assessment, 4(1):93-103, 2000.

Photographs Alignment and High Dynamic Range Image Composition Based on Varying Exposure Levels Shungang Hua and Lidan Wang Key Laboratory for Precision and Non-traditional Machining Technology of Ministry of Education, Dalian University of Technology, Dalian 116024, P.R. China [email protected], [email protected]

Abstract. An approach is proposed in this paper for aligning the photographs, which are taken with varying amounts of exposure towards the same scene, and fusing into a high dynamic range image. Firstly, the median threshold images of the photograph-sequence are computed, and their image pyramids are constructed using multi-scale techniques. Comparing the image pyramids levelby-level completes shift-alignments of photographs. Based on the shiftalignments, rotation-alignments of photographs can be achieved. According to least-squares principle, the response function curve of the imaging process can be recovered; the mapping relationship between pixel values and exposures is acquired with the sample pixels from the aligned photograph-sequence. A satisfactory high dynamic range image can be constructed from the photographs. The experimental results show that the proposed approach is fast and efficacious. Keywords: image alignment, high dynamic range image, image pyramid, response curve, irradiance.

1 Introduction In contrast to common digital image, high dynamic range image (HDRI) [1], whose pixel values are in proportion to the radiance values in the scene, can represent the varying dynamic range of the scene. Because HDRI is usually stored with mantissa and exponent, the confine of gray-levels is broken. The details in dark or bright regions of the scene can be displayed clearly. Therefore HDRI can be applicable to areas such as image-based modeling, rendering, and lighting, physically-based rendering, virtual reality, augmented and mixed reality, motion blur, and so on [2][3][4][5]. Although a few special digital cameras can be used to capture HDRI directly, it is expensive and not prevalent. Most researchers and photographers still employ a standard digital camera and composite HDRI from differently exposed photographs of the same scene. One of the limitations of this method is that the camera should be still among photographs. If there are some slight shifts or rotations among images, the constructed HDRI will be blurry. However camera’s dithering is inevitable during shoot. For this, Greg Ward [6] presented a method of photographs registration for compositing HDRI. But the method was only for shift alignment, and didn’t take rotation into account. Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1146 – 1155, 2006. © Springer-Verlag Berlin Heidelberg 2006

Photographs Alignment and High Dynamic Range Image Composition

1147

In this paper, an approach is presented for shift and rotation alignment among photographs, then the HDRI is constructed from varying exposed photographs of the same scene.

2 Image Alignment In order to obtain a HDRI, we should take multiple photographs towards the same scene with varying exposures. We ought to take photographs on a fixed position ideally, but it is difficult to keep the camera still strictly, even using tripod mounting. Since the dithering errors will have a great impact on recovering the response curve of the imaging system, we must register them at first. Our strategy of image alignment is that choosing one image from the image sequence as a reference (generally, we select the mid-exposed one for better effect), then registering the others with it respectively and computing offsets among them. 2.1 Binarization and Noise-Exclusion A direct method of image alignment is comparing the outlines of the objects in the images. However, conventional edge detection is sensitive to exposure time. When the exposure time of photographs is different markedly, outlines of corresponding objects will appear differences obviously (see Fig. 1). So it isn’t applicable to alignment between varying exposure images. If we choose the median value of the image as a threshold, we can partition the pixels into two equal populations, one brighter and one darker than the threshold, and create the binary image (threshold image). Theoretically, the same scene will produce the same binary image at any exposure level by the median value. Hence, for gray-scale images, we can compute histograms and partition the pixels by the median value M. The threshold images can be created using below equation:

⎧0 , V < M . V1 = ⎨ ⎩1 , V ≥ M

(1)

where V1 is pixel value after image binarization, V is pixel value in original gray-scale image, M is the median value of gray-scale image. For the color images, we should transform them into gray-scale images before binarization process. There are several methods for gray-scale transform. Fig. 1 shows the results of edge detection and median threshold binarization for two differently exposed images, which have been transformed from color into gray-scale firstly. Two median threshold images are consistent with each other as shown in Fig. 1(c) and (d), but the results by edge detection have great distinction as shown in Fig. 1(e) and (f). This example shows the advantage of median threshold binarization for differently exposed images.

1148

S. Hua and L. Wang

(a) Gray-scale image with exposure time 1/4s

(b) Gray-scale image with exposure time 1s

(c) Median threshold image of (a)

(d) Median threshold image of (b)

(e) Edge image of (a)

(f) Edge image of (b)

Fig. 1. Results of median threshold binarization and edge detection

Alignment between images is sensitive to the noises, especially near the median M, which will destabilize difference computation. For instance, the pixels, which should be a little darker than (brighter than or equal to) the median pixels, because of noise, appear to be a litter brighter than or equal to (darker than) the median, then we will assign value 1 (0) to it by mistake. For the pixels far away from the median, the influence of noises is relatively less. We should throw these noises away before comparing computation. In order to eliminate the influence of noises near the median, we can assign value 0 to the pixels, which are within ± 4 of the median value, and keep other pixels original values:


⎧0 , V2 = ⎨ ⎩V1 ,

V −M ≤4 . V −M >4

1149

(2)

where V2 is pixel value in noise-exclusion binary image; V1 is pixel value in original binary image by equation (1); V, M are the same as in equation (1). After noiseexclusion, we can take the differences of these two binary images with an exclusiveor (XOR) operator to get shift and rotation offsets between images. 2.2 Shift Alignment There are not only shift offsets but also rotation offsets between images. We compute all the shift offsets firstly, and then carry out the rotation alignment base on the shift offsets. After binarization and noise-exclusion, we can facilely find the aligning location corresponding to the minimum XOR difference by moving an image and comparing it with the reference image continually. However, for high-resolution images, because all pixels must be computed every time, moving and comparing directly will expend much time. Therefore this method is not applicable to multiple high-resolution images, especially in real-time system. To accelerate comparing (computing) process, we take the following strategies. 1) We construct the image pyramids [7] for binary images by multi-scale technique. For each lower level in an image pyramid, we take the previous binary image and filter it down by a factor of two in each dimension. The number of the pyramid levels depends on the less one between the height and the width of the original image, and besides, the lowest level must be resolvable. In this paper, we shrink the less dimension to 17~32 pixels. A five-level pyramid, which has been shrunk 4 times, is shown in Fig. 2.

Fig. 2. Image pyramid

In order to get the overall shift offset for alignment, we begin with the lowest level (binary) image pair. Move a unaligned binary image within a range of ±1 pixel in each dimension (contains 9 locations: Up, down, left, right, left-up, left-down, rightup, right-down and center), and then compare it with the reference binary image by

1150

S. Hua and L. Wang

XOR operator. If the corresponding pixel values are not equal to each other, the result of XOR is 1, else is 0. We can get the shift offset corresponding to the minimum XOR difference, which is the sum of value 1. The location with the minimum difference is the optimum aligning position of this level. At the next level, multiply the previous offset by 2, and compute the minimum difference offset within a ±1 pixel range of this double previous offset. Repeat this operation until to the highest level (original image). Thus we can get the final shift offset. 2) We divide the pixels into groups, which have 16 or 32 pixels, and apply bitwise operations on 16-bit or 32-bit words for moving and XOR operation. It can speed up the computation, not only because 16 or 32 pixels can be disposed in one process, but also the speed of bitwise operation is far faster than byte-wise operation. 2.3 Rotation Alignment Based on the shift offsets, we get a shift-aligned image sequence with the overlapped parts. For Rotation alignment, we take a circle in image whose center locates at the center of the image and diameter is the less one of the height and the width. We compare the pixels on the circumferences between images to search the optimum matching location within a range (the range of ±5° in this paper). Rotate the circumferential all pixels of the unaligned image pixel by pixel, and compare it with the reference image by XOR operator. Count the result of XOR to find out the minimum difference corresponding to the optimum aligning location. And then relative to the reference image, rotate the image in terms of the rotation offset. After rotation alignment, we cut images to keep the overlapped parts. Because of coupling between shifts and rotation, if the rotation angle is too large, the result of shift alignment may be inaccurate. In order to obtain an accurate result, we should repeat above operation. Namely, shift alignment → rotation alignment → shift alignment → rotation alignment →…, until the rotation offset is 0. During this process, we should not really cut out the no-corresponding pixels after shift alignment. Instead, we just extract the overlapped pixels for the next rotation alignment, lest the images should be cut excessively. We just cut the white margin after rotation alignment until the last rotation offset is 0.

3 HDRI Composition HDRI Composition is to recover and record the true lighting information of the scene. For this, we ought to recover the response curve of the imaging system from the aligned images with different exposures, and get the mapping relation between exposure and pixel value, then fuse the images into a HDRI of the scene. 3.1 Recovering the Response Curve The response curve of the imaging system represents the relation between the exposure H and the pixel value V (As to color image, red, green, blue channels should be deal with independently). Generally, the irradiance E accepted by camera is proportional to the radiance I in the scene. In regular exposure latitude (exposure time


1151

Δt 1/1000s), the exposure H is equal to the product of the irradiance E and the exposure time Δt (i.e.H = EΔt)).). In most cases, the absolute radiance I is not necessary. The relative radiance (or irradiance E) is enough to represent the lighting information of a scene. Hence, we have to work out the relation between pixel value V and exposure H or irradiance E. in other words, we should recover the response curve. 1) Recovery principle According to literature [1], we can recover the response curve of the imaging system. We sample some corresponding points from the differently exposed images. As to these points, we assume that the pixel value V is a function of the exposure H (Capital letter V represents discrete variable, and small letter v represents continuous variable in this paper). So we can write an equation as below:

Vij = f ( H ij ) .

＝

(3)

＝

where f is a nonlinear function; i 1~N indicates the index of sample pixel location (N is the number of sample pixels in each image); j 1~M indicates the index of image (M is the number of images); Vij represents the pixel value on location i in image j; Hij represents the exposure on location i in image j. Generally, the pixel, whose exposure H is larger, has bigger pixel value V, thus we make the reasonable assumption that the function f is monotonically increasing. As shown in Fig. 3, ordinate V represents pixel value, and abscissa lnH is natural logarithm of exposure. There are M = 5 differently exposed images, from which we sample N= 4 groups of the pixels. Because we don’t know the irradiance Ei, the horizontal position of the response curve can’t be defined. Thereby Ei is assumed to be 1 unit temporarily. Different symbols in Fig. 3 mean different groups, and each group of the sample pixels has a sample curve. What we have to do is to move these curves horizontally until they line up to a smooth monotonic curve. But the horizontal position of the response curve is unknown, and need to be defined artificially. The response curve shows the nonlinear characteristic in dark and bright regions. 2) Implementation Usually, the pixel values in common digital image are finite integer from 0 to 255, so we care the mapping relation between 256 pixel values and exposures rather than the expression of function f. Because f is monotonic, we take inverse function and natural logarithm for equation (3), then: F (Vij ) = ln Eij + ln Δtj

where, F (Vij ) = ln f

−1

，

(4)

.

(Vij ) and i, j are the same as in equation (3).

In order to let the sample pixels satisfy equation (4) in a least squares sense, we transform equation (4) to objective function as follows: N M i=1 j=1

2

[F (Vij ) − ln Ei − ln Δtj ] = o

(5) .

1152

S. Hua and L. Wang

Fig. 3. Process of recovering the response curve

In order to reduce the influence of the nonlinear characteristic, we can bring a weighting factor w(V ) into equation (5). w(V ) has minimum value when V= 0 and 255, and increases gradually when V approaches 128. For smoothing the curve, smoothness term F ′′(v ) ≈ F (v − 1) − 2 F (v) + F (v + 1) is introduced too, so we can get following equation: N

M

254

∑∑{w(Vij )[F (Vij ) − ln Ei − ln Δt j ]}2 + η ∑[w(V ) F ′′(V )]2 = ο . i =1 j =1

(6)

V =1

where η is smoothness factor, which is assigned experimentally. The weighting factor in smoothness term is also for reducing the effect of the nonlinear characteristic. In above equation,

Δt j and w(V ) are known, while ln Ei and F (Vij )

are what we want to get. We have N×M equations about the N groups of the pixels. Besides we add F (128 ) = 0 to confirm the horizontal position of the curve. Thus the N×M+1 equations compose the solving system for the N values of ln E i and 256 samples of F (V ) . In order to ensure that the solving system is overdetermined, we should let N × M + 1 ≥ 256 + N , i.e. the pixel groups N ≥ 255 (M − 1) . Equation(6) can be robustly solved using the singular value decomposition (SVD) method, so the response curve can be recovered. 3.2 Constructing HDRI Based on the recovered response curve, we can derive a HDRI from the image sequence. ln Ei can be obtained directly by equation (4), but we compute ln Ei using all the images to reduce the errors as below equation:

Photographs Alignment and High Dynamic Range Image Composition M

ln Ei =

∑ w(V

ij

1153

)[ F (Vij ) − ln Δt j ]

j =1

M

.

∑ w(Vij )

(7)

j =1

where i is the index of each pixel location in image; j is the index of image. The Ei calculated by equation (7) is proportional to the radiance of the scene, and is just what we want. For color image, we will deal with red, green, blue channels respectively.

4 Experiment of Alignment and HDRI Composition Using the approach proposed in this paper, we have aligned an image sequence with varying exposure levels, recovered the response curve of the camera, and then constructed the HDRI of the scene. We have developed an alignment and HDRI composition system using VC++ 6.0 programming environment. The HDRI is saved as Radiance’s RGBE format [8]. Fig. 4 is an image sequence with 6 photographs (resolution: 650×520) taken from Kodak Z7590 digital camera, whose shutter speeds are 1/16, 1/8, 1/4, 1/2, 1 and 2s. During shooting, camera has some shift and rotation offsets. We recover response curve by 150 groups of the sample pixels from the image sequence. The random sample pixels are chosen automatically.

Fig. 4. Digital Photographs with different exposures

1154

S. Hua and L. Wang

(a) Constructed HRDI with an unaligned sequence

(c) Constructed HRDI with a shift-androtation-aligned sequence

(b) Constructed HRDI with a shift-aligned sequence

(d) Amplified details

Fig. 5. High dynamic range image composition

Fig. 5(a) is a constructed HDRI with original unaligned sequence, and the HDRI is blurry in the whole image. Fig. 5(b) is with a shift-aligned sequence. It is clear in the center of the image, but blurry near the edges. Fig. 5(c) is with a shift- and rotationaligned sequence, and the constructed HDRI is clear in the whole image. Fig. 5(d) is amplified details of the panes in Fig. 5(b) and 5(c). This image sequence is aligned two times. Its maximum shift offset is 30 pixels in horizontal direction and 5 pixels in vertical direction. Its maximum rotation offset is 3.2°. As shown in Fig. 5(c), the brightest relative radiance in the HDRI is (97.9635, 32.8506, 24.6320), and the darkest one is (0.0261, 0.0272, 0.0281). Dynamic range is beyond 3.7×103. The details in dark regions (under the table) are clearer than the first four images in the original sequence, and the details in bright region (computer screen) are clearer than the last three images.

5 Conclusion In contrast to edge detection, median threshold binarization is more suitable for aligning the differently exposed images of the same scene. In this paper, we construct image pyramids for the binary images, and get the shift offsets and rotate offsets


1155

among the images. The response curve of the camera is recovered from aligned images by least square method. The irradiance (relative radiance) of every point in the scene can be calculated, and the HDRI is constructed. The experimental results show that the proposed approach is effectual and the constructed HDRI is satisfactory.

References 1. Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. Proc. of ACM SIGGRAPH ’97. ACM Press, New York (1997) 369-378 2. Debevec, P.E., McMillan, L.: Image-Based Modeling, Rendering, and Lighting. IEEE Computer Graphics and Applications 22 (2002) 24-25 3. Debevec, P.E.: HDRI and Image-Based Lighting. Proc. SIGGRAPH (2003), Course 19,San Diego,July (2003). 4. Debevec P.E.: Image-Based Lighting. IEEE Computer Graphics and Applications 22 (2002) 26-34 5. Agusanto, K., Li, L., Zhu, C.G., Sing, N.W.: Photorealistic rendering for augmented reality using environment illumination. Proceedings of the Second IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR ’03). Tokyo, Japon, October (2003) 208-216 6. Ward, G.: Fast, Robust Image Registration for Compositing High Dynamic Range Photographs from Handheld Exposures. Journal of Graphics Tools 8 (2003) 17-30 7. Li, J.Y.: Centroid-based approach to multiscale representation of binary images. Journal of Zhejiang University (Sciences Edition) 30 (2003) 284-288 8. Ward, G.: High Dynamic Range Image Encodings. 9. http://www.anyhere.com/gward/hdrenc/hdr_ encodings.html (2002)

Elastic Algorithm: A New Path Planning Algorithm About Auto-navigation in 3D Virtual Scene Yong Chen, Ke-qin Wu, Dong Wang, and Ge Chen College of Information Science and Engineering ,Ocean University of China, Qingdao 266071,China [email protected]

Abstract. Elastic Algorithm proposed in this paper is a new and fast path planning algorithm to solve the problem of auto-navigation in 3D virtual scene. First, with a grid model based representation of virtual environments, different obstacles are assigned different color in the projection map of scene and their encasing boxes are stored to provide local path. Then, Elastic Algorithm is designed to optimize the basic path. It can be applied to complex condition and moving object, and has high stability and efficiency in solving the path planning problem. Keywords: Path Planning; Grid model; collision-free; Navigation.

1 Introduction Navigation in virtual environment plays an important role in Virtual Reality and Computer Animation. In virtual 3D scene, users can observe and investigate 3D objects from different angles of view by means of navigation to get more useful information [1]-[2]. There are two chief ways of navigation for user to explore the virtual scene. The first is interactive navigation. This kind of method is very agile, users can totally control the navigation position and view direction as their wish, by means of operating the mouse or the keyboard, but the users got lost easily when they enter an unfamiliar virtual scene. Another way is selected navigation. Users can visit the virtual scene according to the prearranged path. This prearranged path is always recorded by the producer before use. This method is easily realized but produces inconvenience. Especially in the condition of complex scene, if the aim of the users cannot be predicted, the task is hard to be accomplished. This paper addresses automatic path finding problem. According to the user’s destination, an optimized path is produced by our algorithm.

2 Previous Works Path Planning is also called Obstacle Avoidance or Collision-free Path Planning, it aims at producing a Collision-free path from a start position to a target position in the environment which contains obstacles, according to a certain criterion (e. g. the Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1156 – 1165, 2006. © Springer-Verlag Berlin Heidelberg 2006

Elastic Algorithm: A New Path Planning Algorithm About Auto-navigation

1157

shortest distance)[3]-[4]. The current auto-navigation path planning algorithm in 3D virtual scene is mainly developed from Robot Path Planning Algorithms and Path Finding Methods in Geography Information System (GIS). Path Planning is a very important task in the Robotics. Since the invention of the first manipulator arm in American in the late 1940’s, the related research and application developed rapidly in occident and Japan. Accordingly, the research on Robot Path Planning and related algorithms sprung up. A lot of effective algorithms have been introduced to solve the Path Planning problem, such as geometry method, space decomposition method, potential field method, mathematical analysis method, etc. In the area of Robot, we hope the Robot can detect the unknown or partly unknown surroundings with all kinds of sensors, and then use the acquired information to make decision of marching forward. So the Path Planning problems we concerned are always based on the unknown or partly unknown surroundings [5]-[11]. Path Planning is also a research hotspot in GIS. Researchers often describe the Shortest Path problem as the Meshwork problem of graph theory. This method can solve the problem such as traveling on cities and highways, but it complicates the problem when dealing with 3D virtual scene containing large non-obstacle area. Comparing with Robotics and GIS, the initial condition in solving Path Planning of Virtual Reality, Computer Animation, Game and Driver Exercitation is much simpler. Before the 3D scene is rendered, information of all objects has to be known, such as the size and position of the static objects, the original position, speed, outside force of the moving objects. So, when complete environment information is known, the main task of Path Planning in 3D scene is to find the optimized path from the start position to the target position.

3 Information Representation of 3D Scene Before the calculation of collision-free path in 3D scene, we must represent all information of the environment, such as obstacle, collision-free area, etc. There are 4 basic methods to represent virtual scene [11]: (1) Representing obstacle by polygon; (2) Representation of collision-free area; (3) Uniform grid method; (4) Route map method. Method (2) and (3) are not applicable to complex 3D environment. And method (1) is time-consuming, so encasing box is often used to simplify complex 3D obstacle, resulting in imprecise collision detection. In our algorithm, we make full use of adjacency relation among objects, thus the projection map of 3D scene, similar to method (3), is selected to represent environment information. 3.1 Represent of Environment Information Based on Projection Map When the 3D scene is modeled, the Layout Plan of collectivity building is often used to ensure the precision of modeling. The obstacles projection map of 3D scene is produced based on it. We project the obstacles of 3D scene which cannot be passed through to a raster plane, resulting in 2D projection map in uniform grid format. Fig. 1 shows a sample of 3D scene. According to it, Fig. 2 is produced to present the

1158

Y. Chen et al.

discrete obstacle information with size 1024 × 800. For static environment, the rasterization only needs to be performed once.

Fig. 1. The sample 3D scene

Fig. 2. The obstacle’s projection map of 3D scene

3.2 Representation of Obstacles In order to distinguish Obstacles in 3D scene, we fill the Obstacles with different color in the obstacles projection map as shown in Fig.3. This projection map will be represented by a matrix called the Scene Projection Map Matrix (SPMM).

Fig. 3. Representing Obstacles with different color


1159

Definition 1: Obstacle Point and Non-Obstacle point In the SPMM, when Map (ui, vi) = 0, there is no obstacle on the site (ui, vi) in the 3D scene, so this site is called Non-Obstacle Point; otherwise the site is called Obstacle Point, the value of Map (ui, vi) corresponds to a certain obstacle in the 3D scene.

⎧0　 Non - Obstacle Point mapi (u, v) = ⎨ ⎩C　 Obstacle Point

4 The Elastic Algorithm for Path Planning In this section, we give an overview of our approach and then introduce our algorithm in detail. 4.1 Algorithm Overview When walking in a unfamiliar environment of the real world, the walker will chooses his direction toward his destination, as shown in Fig.4.a, S->T; if he meets an obstacle, he may try to round it first, and then newly follows his direction toward his destination, as shown in Fig.4.a, S->A->B->C->D->T, point S represents the start point and point T represents the target. More likely, the walker will choose the shorter route which is approximately tangent to the obstacle, as shown in Fig.4.b, S->B->C->T. This idea quite likes using elastic to connect the start point and the target point. When there is an obstacle in its way, the elastic will be pulled, along the boundary of the obstacle, since it cannot pass through directly the obstacle. It is the elasticity of the elastic that make the length of the path to be the shortest. Based on this idea, we designed this elastic algorithm to solve the automatic path finding problem.

a

b

Fig. 4. Main algorithmic idea inspired by elastic

4.2 Algorithm Description In this elastic algorithm, we should pretreat the obstacles in the SPMM and get the route surrounding them, then perform two stages disposal: first, create the basic path and record it as a linked list build with a series of linked nodes (e.g. S->A->B->C->D>T); second, use the elastic algorithm to get an optimized navigation path.

1160

Y. Chen et al.

4.2.1 Algorithm Introduction Definition 2: Surrounding Route of Obstacle and Boundary Point A Surrounding Route of Obstacle (SRO) is composed of its Boundary Points and these Boundary Points are stored in its corresponding circular linked list; Boundary Points are such nodes that have and at least have one neighboring node which represents the obstacle, as the points composing the green line surrounding the quadrangular obstacle in Fig.5.

Fig. 5. Surrounding Route of Obstacle

SRO can be produced by scanning the SPMM, after all the Boundary Points have been founded, we can store them in a circular linked list of the corresponding obstacle. In this way, if the path meets one obstacle, its color in projection map will show us which obstacle it is, the SRO will be cut into two parts, the shorter one will be selected as the local basic path. 4.2.2 Creating Basic Path The main task of this stage is to create a basic path in the SPMM to prepare for the optimization stage. Begin Step1: read S,T; S->PL; // S is start point; // T is target point; // PL is Path Link List and empty at first; Step2: current point= next point on line S-T, temp= Map (current point); If current point ≠ T then Goto Step 3; Else Goto Step 5; Step3: If temp=0 then Goto Step 2; Step4: If temp=C then Find two Boundary Point; // Like A and D in Fig.4.a Find Local Path according to C; // Like ABCD in Fig.4.a //Note that the Local Path is composed of many points more //than 4 points, A, B, C and D Local Path->PL; Step5: T->PL; //Add T into PL End. Algorithm 1. Creating Basic Path


1161

Fig. 6. Elastic Algorithm demonstration

Begin Step1: the first node in PL->front, the second node in PL-> mid; If mid = Destination Then Goto Step 6; // Algorithm end; Step2: the next nodes of PL->temp; If temp = Destination Then Goto Step 6; Else Temp->back; Step3: If mid is in the line of front to back Then { Delete mid from PL; Back->mid; Goto Step 2; } Step4: If (mid is not in the line of front to back) and (no obstacle is in the line of front to back) Then { Delete mid from PL; Back->mid; Goto Step 2; } Step5: If (mid is not in the line of front to back) and (there are obstacles in the line of front to back) Then { Add the new obstacles’ local path into PL between front and mid; The first node of local path->mid; Goto Step 2; } Step 6: End. Algorithm 2. Elastic Algorithm for optimizing basic path

1162

Y. Chen et al.

Basic method: draw a line between the start point A and the target point B, check the value of the point on the line one by one. If the point being checked is a Obstacle Point, find two Boundary Points A and D, then read corresponding SRO to find the local basic path, add the local path A->B->C->D into the path linked list; if the point is NonObstacle Point, just check the next point; if the point is target point, end the algorithm. 4.2.3 Elastic Algorithm for Optimizing Basic Path The Basic Path can meet the demand of collision-free, but it is not the optimal path we want. And some parts of the Path Linked List are composed of Local Path which is created by too many points, so we have to predigest it. The Elastic Algorithm is designed for this purpose. Basic Method: after obtaining the Basic Path Linked Line PL, as SABCDT in Fig.4.a, we perform elastic algorithm: drag SA along A->B until it meet B or another obstacle’s point F. if it meet B, remove A from PL; if F, remove A and add F into the path linked list PL between S and B, take F as S. Repeat the above step until reach T. 4.2.4 Applying Elastic Algorithm Inside Tier Building In this Elastic Algorithm, we project the 3D obstacles to 2D plane, so when the start point and the destination are on different floors of a building, the Elastic Algorithm should be adjusted. To find a path between two positions on different floors,we perform the algorithm in different floors respectively: first, marks two ends of the stairway on the two floors as two midway node, then, find the path between start point and one midway node in the same floor and the path between target point and another midway node respectively, and connect the two path with the mid path between the two midway nodes to build a complete path from the start to the destination (see Fig.7).

Fig. 7. Elastic Algorithm for Tier Building

5 Algorithm Analysis 5.1 Flexibility Excellent flexibility is achieved because the projection map of scene is adopted in the elastic algorithm.


1163

5.1.1 Complex Obstacle Avoidance The obstacle avoidance method based on encasing box of 3D object, proposed by [12], is more applicable to convex obstacles, but relatively incapable of processing complex obstacles, such as the parterres shown in Fig.8.

Fig. 8. Finding path through parterres

The encasing box based method may fail to find a path through complex obstacles sometimes or has to divide them into several pieces. But the Elastic Algorithm can deal with any concave obstacles easily. As shown in Fig. 8, a collision-free path form S to T is quickly found by the algorithm. Theoretically, any environment, even the one as complex as labyrinth, can be processed by this algorithm. 5.1.2 Real-Time Obstacle Avoidance for Moving Objects Some applications such as computer game and virtual training require an ability of real-time obstacle avoidance. In these applications, moving objects are controlled by mouse or keyboard. Genetic methods often have difficulty in real-time obstacle avoidance [13, 14]. But by comparing the projection position of the object with the projection map of the scene, the Elastic Algorithm solves this problem easily. 5.2 Efficiency Because the algorithm is based on the Scene Projection Map Matrix, the adjacency relationship between pixels can be utilized conveniently. Therefore, the local collision-free paths are quickly found by reading the precomputed encasing boxes of obstacles. Two main factors affect the efficiency of finding a collision-free path: the distance between the start node S and target node T, and the number of obstacles in a virtual scene. On the one hand, the efficiency of finding the basic path from S to T is directly affected by the number of nodes in the path. On the other hand, the efficiency of optimizing the basic path by the Elastic Algorithm is directly affected by the number of obstacles in the path. Therefore, in general, for the scene with a lot of little obstacles, this algorithm is more efficient than others.

6 Test and Result We apply this algorithm to a virtual walkthrough environment. All objects in the environment are 3D objects which were modeled by 3D Max software and stored in

1164

Y. Chen et al.

user-defined format. A scene of the environment is shown in Fig.9. In the test, the algorithm can quickly find the optimal path form any start position to target position. Fig. 10 gives 4 examples to show its ability to deal with different complex cases.

Fig. 9. An example of 3D virtual walkthrough environment

(a)

(c)

(b)

(d)

Fig. 10. Four examples of path finding in the 3D virtual walkthrough environment


1165

7 Conclusion This paper proposes a new algorithm based on Scene Projection Map Matrix, called Elastic Algorithm, to address the problem of automatic path planning and obstacle avoidance in 3D virtual navigation. First, local collision-free paths are detected based on the encasing boxes of obstacles which are precomputed with well known environment information. Then the Elastic Algorithm was employed to optimize them. The excellent efficiency and flexibility of the algorithm was analyzed, and an experiment in applying this algorithm to 3D virtual scene verified its effectiveness and practicability.

References 1. P Jansen-Osmann. Using Desktop Virtual Environments to Investigate the Role of Landmarks. Computers in Human Behavior, 2002.18: 427~436 2. Zhigeng Pan, A.D.Cheok, Hongwei Yang, Jiejie Zhu, Jiaoying Shi, Virtual reality and mixed reality for Virtual learning environments, Computers & Graphics. 2006.30(1): 20~28 3. C. Latombe, Robot Motion Planning. Norwell: Kluwer, 1991. 143~176 4. Mingming Zhang, Zhigeng Pan, Pingan Wang, Time-critical rendering algorithm with incorporation of LoD,visibility culling and object impostor, Journal of Visualization and Computer Animation, 2003.14(4): 211~223 5. J.T.Schwartz, M.Shair, A Survey of Motion Planning and Related Geometric Algorithms. Artificial Intelligence, 1998, 37(1): 55~67 6. R.A.Brooks, T. Lazano-Perez, A Subdivision Algorithm in Configuration Space for Find Path with Rotation. Proc. Of the 8th Int. Conf on Artificial Intelligence. Karlsruhe, FRG. 1983: 366~380 7. Y.K.Hwang, N. Ahuja, A Potential Field Approach to Path Planning. IEEE Trans. Robot. & Autom, 1992,8(1): 23~32 8. Z. Ma, Z. Yuan. Real-time Navigation and Obstacle Avoidance Based on Grid Method for Fast mobile Robot. Robot, 1996, 18(6): 344~348(in Chinese) 9. Z. Yuan, M. Gao. A New Method of Real-time Navigation and Obstacle Avoidance for Mobile Robot in Dynamic Environment． Robot, 2000, 22(2): 81~88(in Chinese) 10. Yunfang Ge, Zhigeng Pan, Weiwei Xu, Vision-based Path Planning in Intelligent Virtual Environment, 6th International Conference on Computer Graphics and Artificial Intelligence, Limoges, FRANCE, 2003: 25~34 11. Weiwei Xu, Zhigeng Pan, Yunfang Ge, Novel Method for Curved Path Human Walking Animation, in Proceedings of Second International Conference on Images and Graphics, Hefei, China, 2002: 993~999 12. A. Du, G. Zhai. Design of Moving Path Based on Simplified Terrain. Computer Engineering and Application, 2005.18: 45~48.(in Chinese) 13. Q. Li, L. Lin, G. Yan. A Evolutionary Algorithm for Path Planning of Mobile Robot. Proceedings of the 3rd world congress on intelligent control and automation. Hefei, China, 2000, 28(2): 1206~1209(in Chinese) 14. G. Chen, L. Shen. Genetic Path Planning Algorithm for Complex Environment Path Planning. Robot, 2001, 23(1): 40~45.(in Chinese)

Mental Representation of Spatial Structure in a Virtual Room-Sized Space Zhiqiang Luo and Henry Been-Lirn Duh Centre for Human Factors & Ergonomics, School of Mechanical & Aerospace Engineering, Nanyang Technological University, Singapore [email protected], [email protected]

Abstract. To keep spatial awareness in virtual multilevel building, navigator should understand the spatial structure of space. Mental representation of spatial structure in a virtual cylindrical room was investigated in present experiments. Participants first observed two static views of space in which the two viewpoints were on the same vertical plane but with different heights, and then either judged the relative direction of object in the first experiment or egocentrically pointed out the direction of object in the second experiment. The results suggested that the orientation-dependent representation of spatial structure was created even though participants could learn the space from two different viewpoints lying on the vertical dimension of space. Keywords: virtual reality, spatial cognition, navigation.

1 Introduction Understanding the structure of space is a natural activity when people explore a threedimensional (3D) space. Navigation is a necessary component of task performed in simulated buildings. Previous studies demonstrated that people poorly performed navigation task in virtual buildings, especially when the vertical traveling was involved [1]. Soeda et al [2] suggested that, after traveling the stairwell, people could keep the orientation by environmental information. One intuitive way to realize this method is to give visual access to different floors so that individual can know spatial features on the target floor before vertical traveling [3]. Thus, mental representation created by observing the spatial layout on both the upper floor and the target floor is assumed to better help individual to keep high spatial awareness in building. Mental representations of room-sized space are orientation dependent when the space is learned from single viewpoint on both the horizontal and vertical dimensions of the room-sized space [4, 5]. After subjects experienced one of five perspectives lying on the vertical dimension of virtual space, Luo [4] found that subjects who observed 45-degree view performed best in judging the relative direction among objects, whereas participants observing 0-degree perspective could best point out the direction of object with respect to the egocentric reference. In Shelton and McNamara’s study [5], subjects who stood on the ground observed the spatial layout from one position in the room. Performance in judging the relative direction was more Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1166 – 1174, 2006. © Springer-Verlag Berlin Heidelberg 2006

Mental Representation of Spatial Structure in a Virtual Room-Sized Space

1167

accurate for imagined heading parallel to study view than for imagined headings parallel to novel views. The effect of viewpoint elevation was systematically investigated in designing the aviation display. Wickens and colleagues [6, 7, 8, 9, 10] suggested that the display showing the egocentric view better supports the local guidance in which the egocentric reference is required, and the display showing the exocentric view, especially when the degree of elevated viewpoint was 450 , enhances the understanding of structure of space in which the exocentric reference takes main function. Further, they investigated the function of split-screen display that showed both egocentric and exocentric views in the same display and found this display provided the best support for the continuous task of local awareness and guidance and poor support for some of global situation awareness (SA) tasks [8]. The goal of present study was to investigate the effect of dual views on the mental representation of the spatial structure in a virtual room-sized space. Two views, 0-degree (egocentric) view and 45-degree (exocentric) view, were considered in present experiments. The egocentric view was presented from the perspective of the individual standing on the floor of room, and the exocentric view was presented from the perspective of the individual viewing the spatial structure from a higher position (for example the upper floor). Previous studies found that mental representation to spatial layout in room was orientation-dependent even though people could experience three static views on the floor [5]. The design of dual views in aviation display also suggested that the combination of egocentric and exocentric views could not improve the understanding of global structure of space [8]. Thus, it was hypothesized that the mental representation created by observing two views, the egocentric and exocentric views, was orientation dependent. Two tasks, the judgment of relative direction (JRD, “Imagine you are at the position of A object and facing B object, Point C object” in Experiment 1) and egocentric pointing task (“Imagine you are facing object A, Point B object” in Experiment 2), were adopted to test the representation of space in mind. JRD task was primarily sensitive to object-to-object spatial relations represented in memory, whereas the egocentric pointing task was sensitive to self-to-object spatial relations in memory [11, 12].

2 Experiment 1 There were 24 student, 12 males and 12 females, from Nanyang Technological University participating the experiment. They received 10 dollar for participantion. The virtual scene was constructed in EON (Eon Reality Company, 2004), and showed on the i-glass HMD (i-O Display Systems, LLC) with the 26.50 diagonal field of view. The room was cylindrical shape and measured 8 m diameter and 6 m height in virtual space. There were 7 objects located on the floor. Three of objects had heights (2 meters) greater than observer’s eye height (1.7m) on the floor. When taking the tests, participants would point out the target direction in front of a board via a laser pointer.

1168

Z. Luo and H. Been-Lirn Duh

2.1 Experiment Design Participants experienced both 0-degree and 45-degree views. The distance between the viewpoint and the center of layout was 4m in virtual space. Two orders of viewing were designed: one was beginning at 00 and then at 450 while another one was first beginning at 450 and then at 00 . Participants spent the same time on observing the spatial layout in each view. The direction between any two objects was defined, where 00 direction was defined from Stoneware to Lamp, and the degree of direction increased counterclockwise (see Fig. 1). The 00 direction paralleled with the orientation of subject in the egocentric view. Further, the room was constructed with the cylindrical shape in order to minimize the extra disturbance on representation caused by linear walls [5]. JRD was adapted to test the mental representation in memory. Specifically, participants were required to point out the direction of the top of target. When performing the task, participants stood at the mark in front of a board, and used the laser pointer to point out the direction required in the task. The experimenter recorded the response time and the coordinates of both the laser point on the board and participant’s eyes. When these two coordinates were known, the horizontal and vertical angles from eyes to the red point on the board could be calculated. The principle independent variable was the imagined heading and the order to viewing. Six unequally imagined headings were used, ranged from 720 to 3250 . The dependent variables were the angular error and the pointing latency. The angular error was measured as the angular difference between the actual direction of target and the pointing direction. The pointing latency was measured as the time from the presentation of question to the end of response. 2.2

Procedure

There were three phases in experiment. First, participants took the following memory test. They were required to scan 9 objects printed on a paper for 30 seconds and then generate these objects at the corresponding positions on a blank paper with same size. These 9 objects were not used in the virtual room. Participants who correctly recalled more than 5 objects were allowed to proceed to the next phase. All participants could recall more than 6 objects in this test. Second, participants observed the 7 experimental objects printed on a paper. After they were familiar with the objects, they wore HMD and were instructed to watch an empty virtual room viewed from the assigned viewpoint. The room size was the same with the experimental room containing the objects. Then the experimenter displayed the experimental virtual room containing 7 objects viewed from the same viewpoint. Participants spent 2 minutes learning the location and size of each object. During this time the experimenter helped them to recognize the virtual objects if necessary. After experiencing the first view, participants would watch the previous empty room from the second viewpoint. They also spent 2 minutes observing the experimental spatial layout from the latter viewpoint. Last, participants began to take spatial test in front of the testing board.


(a)

1169

(b)

Fig. 1. (a) The spatial layout of room and the arrow with degree presenting the direction; (b) viewpoint locations, the dashed line with two arrows presenting the two orders of viewing

2.3 Result and Discussion All dependent measures were analyzed using a split-plot factorial ANOVA with terms for order of viewing, gender, imagined heading. Order of viewing and gender were between-participants; imagined heading was within-participant. An α level of .05 was used. 50

40

20

Order of viewing 10

Vertical angular error (deg)

Horizontal angular error (deg)

40 30

30

20

Order of viewing 10

0 - 45

0 - 45

0

45 - 0 72

90

150

210

270

325

0

45 - 0 72

90

150

210

270

Imagined heading (deg)


(a)

(b)

325

Fig. 2. (a) Angular error in judgments of relative horizontal direction and (b) angular error in judgments of relative vertical direction in JRD task. Error bars represent ±1 standard error of the mean (SEM).

To the performance on judging the horizontal direction, the interaction between the imagined heading and order of viewing was significant, F(5, 110) = 5.602, p = .002

1170


(see Fig. 2(a)). Further, the significant effect of imagined heading was not found in the order of viewing from 00 to 450 , while in the order of viewing from 450 to 00 , the effect of imagined heading was significant, F(5, 55) = 5.08, p = .016. The data suggested that the mental representation presented the orientation-dependent property if the egocentric view was learned lately, but this property was weak in memory if the exocentric view was learned lately. The larger viewing angle in the exocentric view seemed to be an important determinant of orientation-free performance in JRD test [12]. The reason could be that subjects better remembered the global property, especially the depth, of array distributed on the floor if they observed the exocentric view lately. The global environmental information was assumed to facilitate the generation of judging horizontal direction [6]. 30

Latency (s)

20

Order of viewing

10

0 - 45

0

45 - 0 72

90

150

210

270

325


Fig. 3. Latency to perform JRD tasks in JRD task. Error bars represent ±1 standard error of the mean (SEM).

The effect of imagined heading on the judgment of vertical direction was significant, F(5, 110) = 64.348, p < .001, see Fig.2(b). Further the pos hoc analysis found the performances for each heading of 90 0 and 3250 was substantially worse than performances for the other four imagined headings ( 720 , 1500 , 210 0 , 270 0 ), with p < .001 in each level. The data indicated that judging vertical direction might require different knowledge from that needed in judging horizontal direction. For example, the height of object should be considered in judging vertical direction but not in judging horizontal direction. Height of object was observed more efficiently in the egocentric view than the exocentric view. No effect of order of viewing suggested two groups acquired the same representation for the vertical direction. Fig. 3 plotted the pointing latency to perform the task under two orders of viewing. The effect of imagined heading was significant, F(5, 110) = 4.585, p = .004. But in each order of viewing, the function of imagined heading might be different. In the order from 00 to 450 , the time for the heading 3250 was significantly faster than time spent for headings 1500 , 210 0 and 270 0 . In the order from 450 to 00 , the time spent for heading 270 0 was longer than time for headings 720 , 1500 , and 3250 . The time difference in the condition of order of viewing from 00 to 450 seemed to stem from


1171

increased difficulty in mentally translating to the required position that was far from viewpoints. The time difference in the condition of order of viewing from 450 to 00 showed a different pattern. The increased mental rotation departing from the frontback ( 00 - 1800 ) dimension delayed the reaction time in task.

3 Experiment 2 There were 20 student, 10 males and 10 females from Nanyang Technological University, participating the experiment for the pay 10 dollar per one hour. The materials and procedures were similar to those used in the first experiment. But participants would perform the egocentric pointing task. In addition, participants were required to imagine standing at the position where they observed the spatial layout from 0-degree perspective in the virtual room. Participants were also required to point out the direction of the top of target. The dependent measures were the same as those in first experiment But the imagined heading was now defined between the participant and object faced in virtual room. Only five imagined headings which lied around the 00 direction were considered in this task. The effect of imagined heading on the vertical angle is significant, F(5, 90) = 79.504, p < .001 (see Fig. 4(a)). Performance on the imagined heading of 00 was substantially better than performance on the other headings, F(1, 98) = 15.610, p < .001. It was noteworthy that the performance on the imagined headings ( 250 and 3400 ) departed farthest from the 00 direction was worse than performance on other three imagined heading, F(1, 98) = 122.288, p < .001. The data suggested that the likelihood of slippage in vertical information increased with increasing mental process. It also indicated that subjects could better represent the spatial vertical information around the 00 direction. 60

30

40

20

Latency (s)

Vertical angular error (deg)

50

30

Order of viewing

20

10

Order of viewing

10

0 - 45

0

45 - 0 0

10

25

340


(a)

350

0 - 45

45 - 0

0 0

10

25

340

350


(b)

Fig. 4. (a)Angular error in judgments of vertical direction and (b) latency in egocentric pointing task. Error bars represent ±1 standard error of the mean (SEM).

1172


The effect of imagined heading on the response latency was significant, F(5, 90) = 6.932, p < .001 (see Fig. 4(b)). The interaction between the imagined heading and order of viewing also showed significant effect on the response latency, F(5, 90) = 4.247, p = .009. Further, there was no difference between any imagined heading in the order of viewing from 00 to 450 ; while in the order of viewing from 450 to 00 , the effect of imagined heading was significant, F(4, 49) = 4.33, p = .005. To some extent, the data indicated the dynamic property of mental representations created by observing two views, which would be discussed in detail in General Discussion.

4 General Discussion The principle goal of experiments was to determine the role of dual views on the mental representation for the spatial structure of virtual room. The two experiments tested two different spatial relations, object-to-object and self-to-object relations, in memory. The results mainly showed the orientation-dependent performance which supported the hypothesis of present study. We explained the performance in the context of spatial references created by both observer and objects in environment. In the context of virtual room-sized space, the 0-degree and 45-degree views were different with respect to the perception and task supported. The 45-degree view provided more global information of environment and visual cues creating the depth dimensionality of layout, whereas the egocentric view gave more visual cues describing the characteristics of objects, especially the height information [14]. Second, the exocentric reference was the natural choice to code spatial relationship among objects in the 45-degree view, which was appropriated for the exocentric task involving geographical directional judgments [13]. Egocentric reference was usually adapted in the egocentric view to code spatial relations, which supported the egocentric task involving the orientation [7]. The intention to combine these two views was that subjects could integrate the different spatial information provided in each view in order to effectively enhance the understanding of structure in space. The experimental data in present study revealed that this objective was partially achieved by this design. When subjects were performing JRD task, they needed to construct exocentric retrieval coordinates over the reference object to retrieve position of object [11]. The efficiency of constructed coordinates was determined by both the perceived properties of structure and the egocentric view. Mental transformations were involved to mediate the conflict between the constructed reference for retrieval and the coding reference. Further, errors (in terms of pointing accuracy and latency) were produced in the process of mental transformation. When judging the spatial vertical direction, subjects required to retrieve the relative distance between the tested location and target’s position, and the relative height of target. Much mental process should be involved to interpret the depth and height in the context of exocentric reference. One possible way was to recall the representation created in exocentric view to determine the depth and pick up the height information mainly from the representation got in the egocentric view. The whole performance on judging vertical direction indicated that participants could better represent the spatial vertical information in the direction they ever orientated than the novel directions they never faced.


1173

Regarding the judgment of spatial horizontal direction in JRD task and reaction time in egocentric pointing task, both the imagined heading and order of viewing could influence the performance. Specifically, the effect of imagined heading on representation was not observed when the exocentric view was observed as the second view. From the point of view of memory update, the data suggested that the spatial knowledge learned by observing each view was integrated with each other. One possible explanation of the integration of spatial knowledge was that the representation in the first point of view might be normalized to coincide with the new representation by the way that the old representation was interpreted or inferred by the updated reference system defined in the second point of view. The performance suggested that the fidelity of scene in the egocentric view played strong role on orientation-specific representation, which brought bias to the reference to code spatial relations in memory.

5 Conclusion In summary, individuals can not construct view-invariant representation of virtual room space after observing from two viewpoints that lied in different heights of space. However, there were a few evidences that the experience of dual views can partially improve the understanding of spatial layout. If the egocentric view is learned first and then the exocentric view is observed, the representation of relative horizontal direction shows the orientation-free property.

References 1. Richardson, A. E., Montello, D. R., Hegarty, M.: Spatial knowledge acquisition from maps and from navigation in real and virtual environments. Mem. & Cog. 27 (1999) 741-750. 2. Soeda, M., Kushiyama, N., Ohno, R.: Wayfinding in cases with vertical Motion. In: Proceedings of MERA97: Int. Conf. on Environment-Behavior Studies (1997). 3. Passini, R.: Wayfinding in architecture. New York: Van Nostrand Reinhold (1984). 4. Luo, Z.: Effect of perspective elevation and environment features on representation of a virtual room space. Virtual Reality (submitted, 2006). 5. Shelton, A. L., McNamara, T. P.: Systems of spatial reference in human memory. Cog. Psy., 43 (2001) 274-430. 6. Hickox, J.C., Wickens C.D.: Effects of elevation angle display, complexity, and feature type on relating out-of-cockpit field of view to an electronic cartographic map. J. Exp. Psy.: Applied 5 (1999) 284-301. 7. Wickens, C. D., Hollands, J. G.: Engineering psychology and human performance (3rd ed). Englewood Cliffs, NJ: Prentice Hall (2000). 8. Wickens, C. D., Olmos, O., Chudy, A., Davenport, C.: Aviation display support for situation awareness (No. ARL-97-10/LOGICON-97-2). Savoy, IL: University of Illinois, Aviation research Lab. (1997). 9. Wickens, C. D., Prevett, T. T.: Exploring the dimensions of egocentricity in aircraft navigation play. J. Exp. Psy.: Applied 1 (1995) 110-135.

1174


10. Wickens, C. D., Vincow, M., Yeh, M.: Design application of visual spatial thinking: The importance of frame of reference. In: Shah, P., Miyake, A. (eds.): The Cambridge handbook of visuospatial thinking. New York, USA: Cambridge University Press (2005) 383-425. 11. Easton, R. D., Sholl, M. J.: Object-array structure, frames of reference, and retrieval of spatial knowledge. J. Exp. Psy.: Learning, Memory, and Cognition 21 (1995) 483-500. 12. Sholl, M. J., Nolin, T. L.: Orientation specificity in representations of place. J. Exp. Psy.: Learning, Memory, and Cognition 23 (1997) 1494-1507. 13. Howard, I.: Spatial vision within egocentric and exocentric frames of reference. In Ellis, S., Kaiser, M., Grunwald, A. (eds.): Pictorial communication in virtual and real environments. Bristol, PA: Taylor & Francis (1991) 338-358. 14. Morar, S. S., Macredie, R. D., Cribbin, T.: An investigation of visual cues used to create and support frames of reference and visual search tasks in desktop virtual environment. Virtual Reality 6 (2002) 140-150.

A Novel Planar Walking Algorithm for Virtual Walking Machine Jangwoo Park1, Jungwon Yoon2, Yo-An Lim1, and Jeha Ryu1 1 Human-Machine-Computer Interface Lab., Dept. of Mechatronics, Gwangju Institute of Science and Technology, Gwangju, Korea {jw7912, yo-anl, ryu}@gist.ac.kr 2 School of Mechanical & Aerospace Engineering, Gyeongsang National University, Jinju, Korea [email protected]

Abstract. This paper proposes a novel planar walking algorithm for the Virtual Walking Machine (VWM), which is a dual foot platform type locomotion interface. Unlike existing algorithms used for VWM, the proposed algorithm utilizes a constraint relationship between two platforms as well as position control. Experiments on the VWM show that the proposed planar walking algorithm can generate straight and turning motions which are smoother and collision-free. Keywords: Locomotion Interface, Walking Algorithm, Navigation.

1 Introduction Several kinds of locomotion interfaces for walking in a virtual environment have been developed. Among them, the types of treadmill and foot platforms are typical configurations considering human walking patterns. Although these types of locomotion interfaces can easily generate straight walking motion, it is difficult for them to make natural turning motions mainly because of their mechanical restrictions [1]. For example, the treadmills that have 1-DOF (degree of freedom) belt motion can not make the natural turning motion. The Sarcos Treadport does not have any turntable that rotates the whole system, so it uses the gesture-based rate control [1]. ATLAS measures lateral position of a leg to determine a turning angle of the turntable and makes physical turning motion by the turntable [2]. However, the user cannot get feelings of natural turning due to the limitation that it has only 1-DOF belt motion. In order to solve the problem, 2-DOF treadmills were developed [8, 9]. However, they have problems in implementing the turning motion such as generation of lateral force and mechanical complexity. Similarly, many devices with foot platforms were also developed. In the case of the Gait Master [3], they make X, Y, Z stages and put plates for yaw motion on the stages. While this mechanism allows more walking degrees of freedom then treadmills does, its walking motion is limited and is not smooth. Lately, a new type of locomotion interface called 'CirculaFloor' was developed. It uses four mobile robots in order to make omni-directional walking, but it cannot generate uneven terrain and the available walking speed is slow [4]. In view of mechanical Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1175 – 1185, 2006. © Springer-Verlag Berlin Heidelberg 2006

1176

J. Park et al.

characteristics, the VWM [5] is a foot platform type locomotion interface. The previous walking algorithms for VWM let a user for straight walking only. This paper proposes a novel planar walking algorithm which allows straight walking and turning motions simultaneously. First, we will explain the existing straight walking algorithm. Then, we will modify and extend it to develop the planar walking algorithm.

2 Position-Based Straight Walking Algorithm Human gait cycle consists of a stance phase, when a foot contacts the ground, and a swing phase, when a foot is in the air. When both feet are in the stance phase, it is called double stance phase. The existing straight walking algorithm for the VWM [5] was developed considering the double stance phase, which increased the natural walking feeling on the VWM. During the double stance phase, both feet contact ground (platforms), so there should be no relative motion between them. This is called double stance condition.

yL O

y R y RC

y

Fig. 1. Foot Platform Coordinates for Straight Walking

In Fig. 1, yR and yL are the current center positions of each foot platform on straight walking line, the superscript C is command for next control step, y RC is right platform’s command position. The existing straight walking algorithm on the VWM is based on the command velocity and can be summarized as follows: For Swing Phase, y iC = y itrack For Stance Phase, (i) For Single Stance Phase,

sw y iC = −vavg ,i

(1)

(ii) For Double Stance Phase, sw sw IF ( y R + y L ) / 2 > 0 , y iC = − K (vavg , R + v avg , L ) / 2 sw sw IF ( y R + y L ) / 2 < 0 , y iC = K (v avg , R + v avg , L ) / 2

( i = R, L )

Where, y is the command velocity of a platform, y is the velocity of the tracker, sw K is the velocity gain, and v avg is the average velocity during the previous swing C

track

phase, defined as

v

sw avg

1 = Tsw

t +Tsw

∫ y

sw

dt

(2)

t

Where y sw is the velocity of the platform during the swing phase, and Tsw is the time of swing phase. Through this velocity-based algorithm, the distance moved during the

A Novel Planar Walking Algorithm for Virtual Walking Machine

1177

previous swing phase is compensated in the current stance phase. During the double stance phase, this algorithm makes the velocity of both platforms equal resulting in zero relative motion. But this algorithm has some problems; During every single stance phase, the velocity of the platform in the stance phase is always constant. However, the velocity of the platform in the stance phase change abruptly when the phase changes, from single stance phase to double stance phase or vice versa, or when the average of the position of two platforms meets the center of VWM. This makes the user unstable while walking on VWM. To solve this problem, we propose a position-based walking algorithm as follows; For Swing Phase,

yiC = yitrack

For Stance Phase,

yiC = yi − K y y avg

(3) ( i = R, L )

Where yC is command position of a platform, yavg is the average position of two platforms, calculated as yavg=(yR+yL)/2, and Ky is the position gain. In this algorithm, the platform in the swing phase follows the tracker position like the previous one. However, in the stance phase, the proposed algorithm uses the average position yavg. If yavg is zero, we can assume that the center of user's body is on the center of the device. If yavg is not zero, the platform of the stance phase moves towards the direction to make yavg zero, i.e., the proposed algorithm compensates user's position by making yavg zero, which ensures that the velocity of the platform in the stance phase changes smoothly because yavg does not change abruptly. Note that the proposed algorithm satisfies the double stance condition because the changes of the positions of both platforms are equal in the double stance phase. Fig. 2 compares the command velocity profiles of two algorithms (right platform). Fig. 2 (a) shows that the command changes abruptly when the phase changes, while Fig. 2 (b) shows the command velocity changes smoothly during the stance phase.

(a) velocity-based algorithm

(b) position-based algorithm

Fig. 2. Command Velocity Profile of a Platform

1178

J. Park et al.

Fig. 3 shows the experimental results of the platform. The dotted line is the command, and solid line is the actual velocity of the platform. It can be seen that the position-based algorithm follows the command velocity better then the velocity-based algorithm. Note that the proposed algorithm can be extended easily to more general planar walking algorithm, which will be explained in the next section.

(a) velocity-based algorithm

(b) position-based algorithm

Fig. 3. Velocity Profile of a Platform

3 Planar Curved Path Walking Algorithm Planar walking means any kind of walking on a plane, such as straight walking, turning, side step etc. However, the most frequently occurring walking patterns are straight walking and turning [6]. To let a user walk on the plane as naturally as possible by using a locomotion interface, a reliable walking algorithm is important. φL

y

( xL , yL )

O

φR

x

( xR , yR )

Fig. 4. Foot Platform Coordinate for Planar Walking

Fig. 4 shows the posture of two foot platforms in the walking plane x-y. The origin O is the center of the locomotion interface. Basically, the proposed position-based straight walking algorithm in the previous section can be independently applied to each variable x, y, φ in order to make the planar walking scenario as follows:

A Novel Planar Walking Algorithm for Virtual Walking Machine For Swing Phase, X iC = X itrack and

1179

φiC = φitrack

For Stance Phase, X iC = X i − K X X avg and φiC = φ i − K φ φ avg ( i = R, L )

(4)

Where X is the position vector (x,y), Xavg=(XR+XL)/2, φavg=( φR+ φL)/2 , KX is a position gain matrix, and Kφ is orientation gain. If applied to each variable x, y, φ independently, the algorithm in Eq.(4) faces two important problems: (i) collision between the two platforms (ii) the algorithm does not satisfy the double stance condition, that is, there occurs a relative motion between the two platforms during the double stance phase. We need, therefore, the additional constraint conditions for solving these problems. Many biomechanics researchers studied human turning motion with constant curvature, and reported that path curvature is an important factor of human walking [6, 7]. The concept of path curvature can be used to solve the above two problems. 3.1 Radius(ρ) and Angle(θ) of Path Curvature In order to use the concept of path curvature, we need to estimate the radius of path curvature. Fig. 5 shows how to estimate the radius when the right foot is in the swing phase. There are three curved paths; the right line is for the right platform, the left line is for the left platform, and the center line is the user’s walking path. If the position of the right foot in swing phase is ( x Rtrack , y Rtrack ), then we can find a circle which passes through initial reference point (L,0) ( J point in Fig.5) and ( x Rtrack , y Rtrack ) (Q point in Fig.5), and the center of the circle is on the x-axis. The point (OC, 0) is the center of the circle, and L is dependent on the distance between both feet when the user stands upright. Then, the radius of path curvature satisfies the following equation;

ρ = OC =

( x Rtrack ) 2 + ( y Rtrack ) 2 − L2 2( x Rtrack − L)

Fig. 5. Path Curvature Estimation

(5)

1180

J. Park et al.

Generally, the curvature radius ρ is always positive. However, we use the sign of the radius to distinguish the direction of turning. If the path is curved towards the right, then the sign is positive, otherwise, the sign is negative. In the case that the left foot is in swing phase, we can use Eq. (5), after substituting L for -L. We define another important variable θ, the angle of path curvature, as the angle between x axis and the line passing through the center of a circle OC, and the position of the platform (x, y).

θ R = sin −1 (

yR

ρ−L

−1 ) , θ L = sin (

yL

ρ+L

)

( −90 < θ ≤ 90)

(6)

Because of mechanical characteristics of VWM, θ can vary between -90 and 90. By using radius ρ and angle θ, a planar walking algorithm can be developed. 3.2 Planar Curved Path Walking Algorithm

Basic concept of the planar curved path walking algorithm is that the swing phase platform follows the tracker position and the stance phase platform compensates user posture, which is the same as the straight walking algorithm. However, additional constraint condition that two platforms should have the same center of curvature must be satisfied in the planar curved path walking algorithm i.e.: For Swing Phase,

ρ ( xitrack , yitrack ) and θ iC = θ itrack ( xitrack , yitrack )

For Stance Phase,

θ iC = θ i − Kθ θ avg

(7) ( i = R, L )

Fig. 6. Avoidance of Collision

Where ρ( xitrack , yitrack ) is the curvature radius and θ track( xitrack , yitrack ) is the curvature

angle of tracker, and θavg=(θR+θL)/2 is the average angle. The curved paths of both platforms are changed by the estimated curvature radius in real-time. However, the


1181

curvature radius does not change during the double stance phase. Using curvature radius and curvature angle, we can get the actual platform postures (x, y, φ) as

x R = ρ − ( ρ − L) cos(θ R ) , y R = ( ρ − L ) sin(θ R ) , φ R = −θ R x L = ρ − ( ρ + L) cos(θ L ) , y L = ( ρ + L ) sin(θ L ) , φ L = −θ L

(8)

Note only two variables ρ and θ are needed to represent three variables x, y, φ. The above planar curved path walking algorithm can avoid collision between two platforms and can satisfy the double stance condition. Fig. 6 shows the case of the right turn. Pmax is the point on the right platform which has maximum distance from the center of curvature OC. Pmin is the point on the left platform which has minimum distance from the center of curvature OC. The collision avoidance condition for two platforms can be satisfied if

OC Pmin > OC Pmax ⇔ OC Pmin − OC Pmax > 0

(9)

The geometric factors in Eq. (9) are the shape of platform which can be assumed to be a rectangular plate for simplicity, the distance between centers of two the platforms, 2L, determined by anthropometric parameters, and the curvature radius ρ. Given the shape of platform and L, we should find available radius of curvature which satisfies the collision avoidance condition. y

−L

L p min

x

OC

p max

Fig. 7. Condition of Collision Avoidance

Fig. 7 shows that Oc Pmin − Oc Pmax is getting smaller as the curvature center Oc move towards the origin. This means that we should find the minimum of Oc satisfying the condition, and we should use ρ which is not less than the minimum. After these considerations, we have found that the minimum of ρ is 1m for the VWM [5]. Note that the double stance condition can be verified easily. During the double stance phase, there is no change of the curvature radius, and the curvature angle changes for both the platforms are the same. ρ = const. and θ = θ (10) R

L

So there is no relative motion between two platforms. For no relative motion between two platforms, all coordinates of the right platform with respect to the left platform frame should be constants.

1182

J. Park et al.

x R / L = 0 and y R / L = 0 and φR / L = 0

(11)

4 Navigation in Virtual Environment We interface VWM with a virtual environment for applying the proposed planar walking algorithm for navigation in VE. VWM supplies user’s posture information to VE. Using this information, VE can change posture of view. A large screen is installed in front of VWM for navigation experiment. During navigation, a user feels that scene is changing with his movement. OpenGL® is used for implementing the virtual environment. It uses modelview transformation method for coordinate transformation. The most important problem of navigation in VE is to move the ground of VE according to user’s movement. To solve the problem, we make the ground movement in VE equal to the stance phase platform movement which user considers as the ground. As discussed earlier, the two important factors of stance phase platform's movement are curvature radius and curvature angle. With these two variables, we can generate the ground movement in VE.

Fig. 8. Movement of the Ground in VE

Fig. 8 shows the movement of ground in VE according to the movement of the stance phase platform. At the initial time, the origin of ground in VE, OVE, and the center of VWM, O, are same. If a user walks along a constant path curvature radius ρ, movement of curvature angle θlocal is as follows:

θ local = θ − θ initial

(12)

Where θinitial is the initial curvature angle in the stance phase (Fig. 8 shows the case that θinitial is equal to 0). With θlocal and ρ, the ground movement can be represented as

xlocal = ρ (1 − cosθ local ) , ylocal = ρ sin θ local , φlocal = −θ local

(13)

However, above equations can be applied for one step only. For a user to walk many steps, we need additional algorithm to take care of accumulation of steps. For this, we define the walking reference points xo, yo, φo.

A Novel Planar Walking Algorithm for Virtual Walking Machine y

y O

O

θlocal x

x

φO = φglobal

OVE

1183

φ global

( xO , yO ) = ( x global , y global )

( xlocal , ylocal )

(a)

OVE ( x global , y global )

(b)

Fig. 9. Transformation Algorithm for Accumulation of Steps

Fig. 9 (a) shows the moment when the phase changes. The walking reference points are updated by the posture of OVE. Fig. 9 (b) shows how to change the ground origin OVE after updating. The vector (xo, yo) will be changed in the view of the global coordinates as θlocal changes. (xglobal, yglobal, φ global) is the actual posture of the ground in VE. By using the walking reference points xo, yo, φo, we can implement the effect of step accumulation. The additional algorithm is as IF Phase_change ==FALSE,

⎡ x global ⎤ ⎡ xlocal ⎤ ⎡ cos φlocal ⎢y ⎥=⎢ ⎥+⎢ ⎣ global ⎦ ⎣ ylocal ⎦ ⎣− sin φlocal φ global = φo + φlocal IF Phase_change == TRUE, xo = x global , yo = y global ,

sin φlocal ⎤ ⎡ xo ⎤ cos φlocal ⎥⎦ ⎢⎣ yo ⎥⎦

(14)

φo = φ global , θ initial = θ

where xglobal, yglobal, φglobal are actual posture of OVE, which are transformed by xo, yo, φo, and xo, yo, φo, θ initial are updated when phase changes.

5 Experimental Results The proposed planar curved path walking algorithm can implement straight walking and turning simultaneously. We performed simple navigation experiments to verify the effectiveness of the proposed planar walking algorithm. To test the performance of straight walking and turning at the same time, we developed virtual environment for experiment. The available minimum radius of curvature in the VWM[5] is about 1m because of many mechanical constraints. So, we made three types of path curvature such as 2m, 1m, 0.5m in VE. We selected three subjects, and trained them for two hours. After training, we let them walk all the paths at once per each subject. Walking trajectory can be computed by using xglobal, yglobal, φglobal. The results of the experiments are shown in Fig. 10. The dotted line is the desired path in VE, and solid line is the actual walking trajectory of each subject. As shown, subjects followed the 2m and 1m curvature paths well (Fig. 10 (a)), but could not follow 0.5 m curvature

1184

J. Park et al.

path well (Fig. 10 (b)). Considering that the limitation of available radius of curvature is 1m, we can think that path which has smaller curvature than the available minimum radius of curvature is difficult for user to follow. In conclusion, both straight walking and smoother large curvature turning without any collision are possible with the proposed planar curved path walking algorithm.

(a) 2.0m

(b) 0.5m Fig. 10. Experimental Result

6 Discussions and Conclusion As shown in the Fig.10, the proposed planar curved path walking algorithm can be used to implement the straight walking and turning simultaneously. Actually, the proposed algorithm can be applied not only to VWM, but also to any locomotion interface that can be classified into dual foot platforms. In the proposed algorithm, we assume that the factors to determine the curvature of walking path is the foot position in the swing phase. However, turning is very complex action, and there are many other factors determining walking path curvature, besides swing phase foot position. There is no report which defines the functional relationship between walking path curvature and kinematical parameters exactly. So, for more accurate human-like natural walking, we will have to study walking motion in the view of the biomechanics, to find functional relationship about turning motion.

Acknowledgments Research reported here was supported by a grant (No. R01-2002-000-00254-0) from the Basic Research Program of the Korea Science & Engineering Foundation and by the Korea Research Foundation Grant funded by the Korean Government (MOEHRD) (KRF-2005-005-J09902).


1185

References 1. J. M. Hollerbach, "Locomotion interface,"in: Handbook of Virtual Environments Technology, K.M. stanney, ed., Lawrence Erlbaum Associates, Inc. 2002, pp. 239-254. 2. H. Noma, T. Miyasato, "A New Approach for Canceling Turning Motion in the Locomotion Interface, ATLAS," ASME DSC - Vol.67, 1999, pp. 405 – 407. 3. H. Iwata, H. Yano, F. Nakaizumi, "Gait Master: a versatile locomotion interface for uneven virtual terrain,"Virtual Reality, 2001. Proceedings. IEEE 13-17 March 2001, pp. 131 - 137. 4. H. Iwata, H. Yano, F. Nakaizumi, H. Noma, "CirculaFloor," Computer Graphics and Applications, IEEE Volume 25,Issue 1,Jan.-Feb. 2005, pp. 64 – 67. 5. Jungwon Yoon, Jangwoo Park and Jeha Ryu, "Walking Control of a Dual-Planar Parallel Robot for Omni-directional Locomotion Interface" IROS 2005. IEEE/RSJ International Conference on 2-6 Aug. 2005, pp. 1151 - 1156. 6. Micheal S. Orendurff, Ava D. Segal, Jocelyn S. Berge, Kevin C. Flick, David Spanier, Glenn K. Klute, "The kinematics and kinetics of turning: limb asymmetries associated with walking a circular path", Gait & Posture 23, pp. 106 - 111. 7. Gregoire Courtine and Marco Schieppati, "Turning of a Basic Coordination Pattern Constructs Straight-Ahead and Curved Walking in Humans", Journal of Neurophysiology Vol 91 April 2004, pp. 1524 - 1535. 8. R. Darken, W. Cockayne, D. Carmein. "The Omni-Directional Treadmill: A Locomotion Device for Virtual Words," Proc. of User Interface Software Technology (UIST 97), pp.213-222. 9. H. Iwata, "The Torus Treadmill:Realizing Locomotion in VEs," IEEE Computer Graphics and Applications, Vol. 19, No. 6 1999, pp. 30-35.

An Interactive System of Virtual Face Pose Estimation Zheng-yi Wu1, Jian Shi1, Xiaoming Tong2, and Zheng Tan1,2 1

Department of Information and Control Engineering, Changshu Institute of Technology, Jiangsu 215500 2 Institute of Information Engineering, Xi’an Jiaotong University, Xi’an 710049 {wwzzyy, shijian}@cslg.edu.cn

Abstract. On the basis of an established individualized face 3D model, this paper implements a new facial pose detection based on face model plane. The new method, by utilizing the feature point correspondence between model plane and model image, can compute the three parameters of face pose angle and effect model display, representing a simple interactive system of virtual face pose estimation. Experiments show that proposed algorithm, simple and highly automated, can accurately and effectively detect facial pose information and display pose changes of virtual models. Keywords: face pose estimation; face 3D model; homography.

1 Introduction Virtual reality has been received growing attention in the field of film, game and cartoon design as well as intelligent human-computer interaction, model-based video coding, and intelligent monitoring. As the face carries the most important information about the subject, it is necessary to individualize facial models and describe the changes of facial pose and expression. Virtual reality, human-computer interaction, multi-pose estimation, and expression recognition are all based on facial pose information. Given the homography between coplanar facial features and video frame pixels, this paper develops a virtual reality system to estimate facial pose in video sequence based on the individuation of facial models and face detection algorithm. As 2-D images cannot fully show the position of facial feature points, we need prior knowledge to estimate the facial pose. Most algorithms use image features (such as grayscale, color and image gradient) to detect facial pose, but that will necessitate a large number of training samples to establish the correlation between facial pose and image features[1]. Recently, thanks to the fast development of 3-D modeling technology, the acquisition of individualized 3-D face has been made much easier, and the estimation algorithm based on such models [2,3,11] has attracted wide attention. By establishing accurate geometrical information, model-based methods lead to even more precise and robust results, demonstrating their potential for wide application. Based on the homography of feature points obtained from models and video sequence, our pose estimation algorithm computes the three parameters for facial pose angle. Experiments have shown the validity of the method. The virtual reality system Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1186 – 1192, 2006. © Springer-Verlag Berlin Heidelberg 2006

An Interactive System of Virtual Face Pose Estimation

1187

can real-time detect the position of feature points by following the algorithm, thus realizing facial pose interaction.

2 Basic Algorithm We assume that a 2D point is denoted by

m = (u , v) T , and a 3D point by

M = ( X ,Y , Z )T . The corresponding augmented vectors are T T = (u , v,1) , respectively. The relationship between a 3D M = ( X , Y , Z ,1) and m and its image projection m is given by point M ~ ~ = A[R t ]M (1) sm where s is an arbitrary scale factor, the extrinsic parameters [ R t ] is the rotation and translation which relates the world coordinate system to the camera coordinate system [4], and A is called the camera intrinsic matrix. Without loss of generality, we assume the model plane is on Z = 0 of the world

coordinate system. We denote the ith column of the rotation matrix R by According to Eq.(1), we have

⎛u ⎞ ⎜ ⎟ s⎜ v ⎟ = A[r1 r2 ⎜1⎟ ⎝ ⎠

r3

⎛X⎞ ⎜ ⎟ ⎜Y ⎟ t ]⎜ ⎟ = A[r1 r2 0 ⎜ ⎟ ⎜1⎟ ⎝ ⎠

⎛X ⎞ ⎜ ⎟ t ]⎜ Y ⎟ ⎜1⎟ ⎝ ⎠

ri .

(2)

Therefore, the relation between the model point M and its image m are constructed by a homography matrix H :

~ ~ = HM sm with H = A[r1

r2

t]

(3)

As is shown that the 3×3 matrix H is also determined by a scale factor. Given an image of the model plane, a homography matrix can be estimated. According to Eq.（ 3） , we have

⎛ h11 ⎜ H = ⎜ h21 ⎜h ⎝ 31 [r1

h12 h22 h32 r2

h13 ⎞ ⎟ h23 ⎟ = λA[r1 r2 1 ⎟⎠ t] =

1

λ

A −1 H

t]

(4)

(5)

1188

Z.-y. Wu et al.

where the intrinsic matrix A could be determined previously according to the camera standardized algorithm. Using the knowledge that r1 and r2 are orthonormal, we can calculate r1 ， r2 and t ， as well as

r3 = r1 × r2 .

By the above presentation, the pose estimation can be ascribed to the calculation of homography matrix H , which can be obtained by Eq.（ 3） . The coplanar facial features illustrated in Fig. 1 are selected for the computation. Fig. 1. Facial features

3 System Structure As shown in Fig. 2， the system consists of 5 modules, i.e., modeling, video sampling, feature orientation, pose estimation and model display. The 3-D face modeling is carried out off-line, while the others are real-time.

Video Sampling

Feature orientation

Pose estimation

Model display

Modeling

Fig. 2. System structure

3.1 3-D Face Modeling

Firstly, we obtain face model based on single obverse face image [5], which uses coplanar facial features of the image to adjust the positions of a general 3-D grid face model. Then we establish the individual face model according to the statistical information of face features. Finally, we store the coordinates of the feature points on the plane as the model information.

(a)2-D image

(b)3-D grid

(c)texture

Fig. 3. 3-D face model

(d)3-D model


1189

3.2 Facial Features in Video Sequence

The feature orientation algorithm for initial frame differs from that of continuous frames. For initial frame, we get possible skin area by target extraction and color detection, and obtain candidate area through trough detection. Then the precise feature position is derived by feature verifying technology. While for continuous frames, we predict the feature area according to the relativity between frames and determine the feature position of current frame according to the previous frame. As video sample system is usually fixed on desktop, we can remove the background by target extraction. Though the detection steps increase, this measure will effectively remove the noise in images with complex background, which will improve the running speed. As skin color is distributed centrally in color space (Cr, Cb), and is not much affected by changed lighting condition, the color area thus derived is made more robust. After removing noise, we get the candidate face area set [5]. As arms and some other areas will cause interference, we need to test the candidate area of face by detecting the two troughs of the face area grayscale --- the two troughs form two eyes, lending to the construction of a tentative face. This tentative face can be verified by being projected on to the original face space [6].

Fig. 4. Face detection

When we get the position of eyes, the information of eyes’ edge is used to determine the position of canthus. First, we obtain such information through detection in the eyes’ region. Then the noise in eyes’ region is morphologically eliminated, and the precise position of canthus is obtained by canthus detection in the binary map of eyes’ region. In this paper, only the outside canthus is used for pose estimation.

(a)eyes image

(b)edge image

(c)edge region

Fig. 5. Canthus detection

(d) canthus extraction

1190

Z.-y. Wu et al.

The mouth region contains more red component and less blue component than other facial regions. Hence, the detection of mouth region is fulfilled by enhancing the lip color chromatism [7]. The chrominance component Cr is enhanced to (Cr)2. By adding the reverse sum of the brightness component Y, the mouth area is obviously enhanced as shown in Fig. 6(b). Then we subtract the enhanced blue components (Cb)2, and the mouth region is finally obtained.

(a)face image

(b)enhanced image

(c)mouth region

Fig. 6. Mouth detection

After the detection of the mouth region, the corners of the mouth are obtained in the same way as the canthus. For continuous frames, face, eyes and mouth regions are derived by area tracking technology and then the feature points can be determined. 3.3 Derivation of Facial Pose

When we get the positions of feature points, the pose parameters can be derived by computing the homography matrix according to the position of face feature points and face models, just as mentioned in Section 2. 3.4 Model Display

The object can be rotated and translated by adjusting the camera position in OpenGL. After we get the parameters of facial pose, we can display the model’s pose changes ,which should match the poses in the video frame. The algorithm diagram is shown in Fig. 7.

Fig. 7. Estimation flowchart


1191

4 Experiments and Results The virtual reality system is run on PC, which adopt a common CCD camera and the video data form is YCrCb, which can directly used in image process. Fig.8 shows parts of the research results. The video frame lies on the top left corner, below which is shown the value of the three rotation angels of the facial pose, and our model results are displayed on the right side. The results demonstrate that our algorithm can estimate facial pose in video sequence effectively. According to the experiment analysis, the pose estimation errors mainly come from the model-inherent errors and feature position errors. The error is conspicuous when the rotation angel is so large that the feature points are covered. Therefore, the method can only guarantee the estimation accuracy in the case of the facial features being uncovered. However, this problem could be tackled by considering more facial feature points and selecting the most appropriate facial feature points.

Fig. 8. Pose estimation results

5 Conclusions In this paper a facial pose estimation algorithm based on video sequence is proposed, and a real-time virtual facial pose interface system is developed. Research result shows that this method can estimate video facial pose effectively. Our further research will focus on the selection of facial features so as to eliminate the influence of expression changes and make pose information more robust. On the other hand, the improvement of feature estimation algorithm could enhance the model's sensitivity to facial expression change, thus making help to reconstruct the expression of a virtual face.

References [1] Q. Chen, H. Wu, T. Fukumoto, M. Yachida, 3D head pose estimation without feature tracking, In: Proceedings of Third IEEE International Conference on Automatic Face and Gesture Recognition, 88-93,1998. [2] P. Yao, G. Evans, A. Calway, Using affine correspondence to estimate 3-D facial pose, In: Proceedings of IEEE Int. Conf. on Image Processing, 3: 919-922, 2001.

1192

Z.-y. Wu et al.

[3] I. Shimizu, Z. Zhang, S. Akamatsu, K. Deguchi, Head pose determination from one image using a generic model, In: Proceedings of IEEE International on Image and Graphics, 290 – 293, 2004. [4] Z. Zhang. A Flexible New Technique for Camera Calibration， IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 11,1330-1334, Nov, 2000. [5] T. Zheng, C.X. Guo, Y.C. Xing, 3-D face modeling and emotion computing based on single obverse face image, The 1st conference on emotion computing and intelligent interface, China,2003. [6] Y.C. Xing, C.X. Guo, T. Zheng, Accurate face estimation based on trough in complex background, The 4th conference on biologic recognition, China, 2003. [7] Rein-Lien Hsu, Abdel-Mottaleb M., Jain A.K., Face Detection in color images, IEEE Trans. On Pattern Analysis and Machine Intelligence, 2002, 24(5): 696~706. [8] T. Darrell, G. Gordon, M. Harville, J. Woodfill, Integrated person tracking using stereo, color, and pattern detection, In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 601 – 608, 1998. [9] R. Rae, H.J. Ritter, Recognition of human head orientation based on artificial neural networks, IEEE Transactions on Neural Networks. 9:257-265, 1998. [10] R. Newman, Y. Matsumoto, S. Rougeaux, A. Zelinsky, Real-time stereo tracking for head pose and gaze estimation, In: Proceedings of Fourth IEEE International Conference on Automatic Face and Gesture Recognition, 122-128, 2000. [11] M. Dimitrijevic, S. Ilic and P. Fua, Accurate Face Models from Uncalibrated and Ill-Lit Video Sequences, Conference on Computer Vision and Pattern Recognition, Washington, DC, June 2004

An Efficient Manipulation of Game Contents on Heterogeneous Platforms Using MR Interface Gi Sook Jung, Seung-Dae Jeong, Kyung Ho Jang, and Soon Ki Jung Virtual Reality Laboratory, Department of Computer Engineering, Kyungpook National University, 1370 Sankyukdong, Bukgu, Daegu, 702-701, Korea {gsjung, sdjeong}@vr.knu.ac.kr, {khjang, skjung}@knu.ac.kr

Abstract. In this paper, we propose a prototype system, which deals with dynamic and huge graphic data on heterogeneous platforms, thus enabling clients to manage their shared virtual world in their hardware/software environment. Through the proposed system, users can experience the same virtual world in different interface metaphors: the keyboard of PC, a virtual control using a handheld MR (Mixed Reality), and the key button/stylus of PDA. Since we consider various service styles to various clients, first of all, we need to adjust the system’s service level to low-end devices having lower performance. Therefore, we processed two kinds of considerations, data simplification, and the virtual control interface using handheld MR method to overcome limitations related to data transmission and data input. Keywords: handheld augmented reality, handheld mixed reality, graphic streaming, virtual control, heterogeneous platform.

1 Introduction Efficient manipulation of multimodal massive data, which plays an important role in the fields such as entertainment, medical applications, education, and scientific applications, is an emerging interdisciplinary research field. Through PC-based 3D network games or networked virtual reality, we can experience a remote manipulation of a huge body of dynamic data. Since both the user and the system must effectively deal with that kind of data set, we need to develop a versatile system that considers efficient data simplification and that provides effective interface for heterogeneous platforms including low-end devices. For this reason, we have developed a prototype system built on existing 3D network game, and then examined efficient handling of a massive body of graphic data and the usability of optional MR (Mixed Reality) interface. From these experimental results, we propose not only efficient data simplification methods including hierarchical geometry hashing, visibility culling, and NPR (Non-Photorealistic Rendering), but also an effective MR interface as a client’s ability. Fig. 1 shows tablet PC and PDA users playing Quake II[10] demo game in a new rendering style, with handheld MR and key button/stylus interface, respectively. Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1193 – 1203, 2006. © Springer-Verlag Berlin Heidelberg 2006

1194

G.S. Jung et al.

Fig. 1. Heterogeneous clients with different interfaces

In this paper, we describe the unique properties of our system focused on efficient graphic data simplification and intuitive virtual game control, after outlining related work and system overview. Finally, we analyze the experimental results, and then conclude by mentioning future work.

2 Related Work Our related work can be classified as 3D graphic streaming, handheld AR interface, and virtual game control. 2.1 3D Graphic Streaming Several methods, especially related to remotely navigating large virtual world, have been proposed in the form of streaming 3D scene on the server by E. Teler and D. Lischinski[14], M. Hosseini and D. Georganas[5], and G.S. Jung, et al.[6,7]. E. Teler and D. Lischinski[14] presented a 3D scene streaming approach for remote walkthroughs, which allows a user on a client machine to interactively navigate a remote 3D scene. M. Hosseini and D. Georganas[5] used the MPEG-4 standard for streaming 3D worlds and the corresponding animation over the web while users navigate and manipulate the content on server. Whereas both systems dealt with static and predefined virtual world on server, G.S. Jung, et al.[6,7] managed a dynamic game data, which could be streamed after graphic data simplification and simple compression, and which considers heterogeneous clients’ various computing power and network bandwidth.

An Efficient Manipulation of Game Contents on Heterogeneous Platforms

1195

2.2 Handheld AR Interface Recently, a number of handheld AR (Augmented Reality) systems using marker tracking have been developed. These systems use various interface styles to overcome input limitation of handheld devices, as well as to provide a new sense of game playing. We can discern the research trend through several prototype games based on visual code system[11,12,13], AR Tennis[4], the first stand-alone Handheld AR[16], and Invisible Train[15]. M. Rohs[12] used camera-equipped mobile devices and marker-based interaction for gesture-based control of games that incorporate physical and virtual aspects. They developed several prototype games using visual code system[11], which generates pose-related information such as distance, tilt, rotation, and world coordinate system with respect to the marker, and adapts marker-based interaction techniques with well-defined interaction primitives[13]. A. Henrysson[4] introduced AR Tennis as a prototype system to support face to face collaborative AR gaming. They adapted the ARToolKit library[1] to the Symbian mobile phone operating system and then developed a sample collaborative AR game, AR Tennis. They considered the mobile phone as a tangible input device and exploited its motion as a primary interaction method, unlike traditional AR interfaces in which the display and input devices are separated. D. Wagner and D. Schmalstieg[16] proposed what they call the first stand-alone AR system for a PDA with a commercial camera. The application provides users with a three-dimensional augmented view of the environment, at a good accuracy of overlay registration by using ARToolkit[1], the tracking code of which is ported to the PDA. In addition, they introduced an optional client-server architecture that is based on wireless networking and is able to dynamically and transparently offload the tracking task in order to provide better performance in select areas. D. Wagner, et al.[15] presented a system architecture for interactive, infrastructureindependent multi-user AR applications running on off-the-shelf handheld devices. They implemented a four-user interactive PDA game, Invisible Train, as an evaluation setup to encourage playful engagement of participants in a cooperative task. Originally, the term AR was used in the context of a RV (Reality-Virtuality) continuum which also includes AV (Augmented Virtuality) [9]. In the RV continuum, real environment consists of only real objects, and virtual environment consists of only virtual objects. Generic MR environment is defined as one in which real world and virtual world objects are presented together within a single display, that is, anywhere between real and virtual environments which are the extremes of the RV continuum. Even though our system referenced existing AR techniques, it is more AV-like and can be fully adapted to the RV continuum. As a result, we use the term MR. 2.3 Virtual Game Control Studies of virtual game control by [3][8]. R. Headon and R. Curwen[3] have described a sensing system for recognizing and characterizing human movements, which can be applied to an ubiquitous game environment. They considered the control of computer games through players interacting with the physical environment in a natural and appropriate manner. This virtual game control results in bidirectional influences between virtual game world and real game world. They demonstrated a

1196

G.S. Jung et al.

system in which actions in the game are mapped to similar actions in the real world, with a 3D first-person game, QuakeTM. D.H. Lee and S.K. Jung [8] presented a method to integrate a more flexible user interface into X3D using computer vision-based techniques. The proposed method allows navigation of a virtual environment on the WWW using only a USB camera and client program, without the need for a mouse or keyboard interface, specific equipment, or environmental setup.

3 System Overview The current study have advanced our previous 3D streaming engine, which can extend existing PC-based network games onto heterogeneous mobile platforms without modifying the original software[6,7], adding an intuitively playable MR interface. In order to play a network game, all kinds of client events including keyboard, mouse, stylus, and key pad button, can be transmitted to the server after translating those events into the game events on the server. Additionally, for a multimodal game interface, we use the camera’s posture with respect to the base marker. This kind of MR interface provides a tangible metaphor to manipulate intuitively the game contents. 3.1 Conceptual System Diagram Our system has been developed on WLAN; a conceptual system diagram is shown in Fig. 2. The system consists of multiple servers on a PC and clients on a PC or handheld device. We can execute a game on the PC server in the form of MR interface as well as the original game interface. The execution process is as follows. First, a client connects with the invoke manager on the PC server that can play a 3D network game. Second, the invoking manager executes a game server to establish a one-to-one connection between the client and the game server. Lastly, the first connection between the invoking manager and the client is closed. The next connections between another clients and game clients follow the same process.

PC

3D Network Game

Invoking Manager

Game Server (Server)

Game Client (Server)

Game Client (Server)

MR Environment

PC Client

Tablet PC Client

PDA Client

( Optional )

Fig. 2. Various clients on heterogeneous platforms


1197

3.2 Client-Server Structure Through the client-server connection between a client and PC server, our system performs two tasks: graphics streaming from server to client, which is processed by event transmission from client to server, and virtual game control that maps camera pose on client into game events on server(Fig. 3).

Server

Client Camera

Video

Tracker

Event

Pose

Game Application

Event

Event Handler

Event

Event Handler

Event

Scene Graph

Scene Graph

Scene Graph

Geometry

Virtual Controller

Graphics Command

S.G. Generator

Geometry

Video

Renderer Video

Display Fig. 3. Client-server structure

For graphics streaming, as described in our previous papers[6,7], we developed a middle-ware streaming engine that make it possible to play existing OpenGL-based 3D network games on PDA. The server side engine processes OpenGL command stream capture, scene graph reconstruction, data simplification, compression, and transmission. To reduce the amount of transmission, we use a client-server scene graph, hierarchical geometry hashing, visibility culling, and a simple NPR (NonPhotorealistic Rendering) technique. The client-side engine simply renders the scene graph after updating it with received geometry and view information. In order to control a game application on a server in a tangible and intuitive method, we chose an MR interface composed of camera, tracker, and virtual controller on a client. After a tracker detects markers from camera images to calculate camera pose, the virtual controller generates pose-related information, for example, distance, tilt, roll, speed, world coordinates with respect to a marker, and maps those values to specific game events capable of playing the game naturally.

4 Effective Graphic Data Simplification In our client-server connections, the clients not only can be implemented on heterogeneous platforms including PC and PDA, but also can be fully extended to other upcoming mobile platforms. To have heterogeneous clients choose the rendering style as their processing power and network bandwidth, we proposed a data simplification technique for dynamic and huge graphics data on server.

1198

G.S. Jung et al.

4.1 Dynamic Massive Graphic Data Unlike other usual graphic systems, our system has some important differences in terms of generating graphic data on a server. The first thing is that our system uses un-expectable geometry information dynamically and irregularly captured by GLIntercept[2], while other systems usually deal with the preprocessed and static graphic data. The second thing is that we can know not the whole world, composed of primitives, but only a small part at rendering point of time. Acquainted with specific application, for example Quake II[10], we can know and preprocess the whole rendering information, and may not reorganize total geometry structure every frame. However, we assume that the application-specific rendering information is not known in advance, because we want our method to be extended more generalized prototype system dealing with dynamically inserted and deleted massive graphic data. Therefore, we implemented our system in the form of reconstructing and updating a scene graph every frame from command stream about graphics library. 4.2 Data Simplification Methods To simplify dynamic and massive data on a server, which cannot even be transmitted to the clients, we use three techniques: hierarchical geometry hashing, visibility culling, and NPR[6,7]. We designed a hash function for Vertex, Primitive, and Object, which are the nodes of the scene graph, to decrease geometry redundancies generated in intra-frame and inter-frames during the animation. With the results of hierarchical hashing, we can dynamically reconstruct client-server scene graph in inserting, deleting, and updating for captured geometry nodes. In addition, we can use the hash keys to deliver the change of the scene to the client. The reconstructed scene includes both visible and invisible Objects. In order to transmit only visible Objects, it is important to find visible ones through view frustum culling and occlusion culling. In our system, we use the glViewFrustum( ), one of OpenGL functions as the view frustum culling, and the Z-Depth test of graphic card as the occlusion culling, to find visible Objects and Primitives in the current frame. Occlusion culling using Z-Depth test is processed as follows. First, the system gives a number to each Object. Second, the system renders the scene graph after translating the given number into RGB value. Finally, we extract visible Object IDs, Primitive IDs, and Vertex IDs from the rendered results of frame buffer. We use simple NPR instead of texture mapping, because the performance on various platforms is limited to low-end devices and the transmission amount should be reduced for real time rendering. The server calculates the average of texture color, which is mapped to each primitive, to send it to the client. The client renders Objects, which are filled with the received color, in line drawing style.

5 Virtual Game Control Using MR Interface We use the term “virtual game control” in terms of using sensing information achieved from a camera sensor, instead of a general keyboard or mouse input, in playing games.


1199

The ideal virtual game control system is, from our point of view, one in which various sensing information from the real world is suitably mapped into virtual events in the virtual world using a similar metaphor. Although the similarity of interface metaphors is one of the most important elements that effects the usability, intuitiveness, and eventually immersion to a game, we usually compromise between the cost and efficiency. Related to our virtual game control system, therefore, we have found common ground from the previous two researches[3][11,12,13]. R. Headon and R. Curwen[3] defined (Condition, Action, Key event) mapping table to implement virtual QuakeTM control system, in which Condition is determined by floor tiles used in determining direction and EMG sensor worn on the player’s forearm to detect fist-making action that simulates shooting. While they use special equipment such as a floor and an EMG sensors to design a Condition, we use only a camera sensor to define it from handheld device’s pose with respect to a marker. When determining our Condition, we referenced the orientation parameters used in M. Rohs[11,12,13]’s visual code system for mobile phone and its applications. However, the way of calculating our orientation parameters is totally different from his system. Our orientation parameters, their generation algorithm, and Condition mapping table are shown in Fig. 4, Fig. 6, and Table. 1. In order to more clearly show the concept of real and virtual worlds, we changed the name of mapping table items from (Condition, Action, Key event) into (RealCondition, GameAction, KeyEvent). The only difference between the two mapping tables is in the Condition/RealCondition depending on what we sense is whether human movements in ubiquitous computing environment or handheld device’s pose in MR environment. The scene of playing our handheld MRQuake is shown in Fig. 5.

Distance

Roll

Tilt

Fig. 4. Orientation parameters

Fig. 5. Handheld MRQuake on tablet PC

Marker on/off

1200

G.S. Jung et al. Table 1. Virtual control mapping table for Handheld MRQuake Real Condition 200 < Tilt < 270 90 < Tilt < 160 30 < Roll < 90 270 < Roll < 330 Marker on/off time > 200 ms 160 < Tilt < 200 and ( 0 < Roll < 30 or 330 < Roll < 360 ) Scale factor to marker size > 1.3

Game Action Forward Backward Turn left Turn right Jump

Key Event Up Down Left Right Space

Stay

NOT

Shoot

Control_L

ControlThread( ) { loop { Marker recognition processed by ARToolkit Marker pose calculated by ARToolkit Tilt and Roll from pose matrix Scale factor from pose matrix switch ( RealCondition ) { Post KeyEvent as GameAction } }until( end ) } Fig. 6. The algorithm of virtual game control

6 Experimental Results Using Quake II[10] demo program, we tested our prototype system, and analyzed it in graphic data simplification and usability evaluation. 6.1 Graphic Data Simplification For the first 115 frames of Quake II demo, we examined two kinds of data simplification techniques, hierarchical geometry hashing by special hash keys and occlusion culling using Z-Depth test; the results are shown in Fig. 7 and Fig. 8. The reason why we should transmit the small part of the whole world after simplifying geometry data is that current low-end devices generally use software rendering technique and don’t have memory enough to save and process the whole world. Whenever capturing the basic geometry nodes, Vertices, Primitives, and Objects, as defined in our previous studies[6,7], the proposed system always queries each node’s existence to Scene Graph with its hash key. Fig. 7 shows that most of geometry nodes, whether they are intra-frame or inter-frames, are redundant because of their


1201

spatio-temporal coherence. The higher the rate of captured nodes and hit numbers to Scene Graph is, the higher the redundancy is. Therefore, vertex hashing is one of the most critical consideration to reduce redundancy and transmission to the client. Even if the hit rate of Primitives and Objects is lower than Vertices, we must consider their hashing because they have huge vertices as their child nodes. Vertex se 15000 cti 10000 er 5000 V 0

1 16 31 46 61 76 91 Frame

Primitive se 2000 tivi 1000 m ir P 0

1

16 31 46 61 76 91 Frame

Object tsc 60 40 je 20 b O 0

1

16 31 46 61 76 91 Frame

Fig. 7. Hierarchical geometry hashing. dashed line: the number of captured nodes, dotted line: the hit number resulted from the queries to Scene Graph.

Occlusion culling using Z-Depth test is processed for current frame’s nodes of the Scene Graph, which means occlusion culling is processed after geometry hashing. From the results in Fig. 8, we can know a transmission to a client can be limited to the visible child nodes of the Scene Graph. For example, when the system determines the nodes to be transmitted to the client, the search range and transmission data can be decreased efficiently. We hierarchically search and transmit the visible nodes with their related index information. Vertex se 6000 ict 4000 re 2000 V 0

1

16 31 46 61 76 91 Frame

Primitive esv 2000 iti 1000 m ir P 0

1

16 31 46 61 76 91 Frame

Object esv 30 it 20 ce j 10 b 0 O

1

16 31 46 61 76 91 Frame

Fig. 8. Occlusion culling using Z-Depth test. dashed line: current frame’s nodes in Scene Graph, dotted line: current visible nodes of Scene Graph.

At this point, we don’t deal with the size and efficiency of index structure in order to focus on only geometry itself. However, in transmission, balancing index structure and geometry data, the network bandwidth, client’s capacity and processing power are all related to the efficiency of graphic streaming system. 6.2 Usability Evaluation After developing our prototype system, we evaluated the usability of the three interfaces to PC, tablet PC, and PDA: keyboard, virtual control using MR, and key button, in terms of mobility, ease of input, novelty, excitement, and time to become skilled. From six volunteers consisting of five males experienced in several PC based network games and one female unfamiliar with games, we achieved the results shown in Table 2, in which the lowest and highest figures are 0 and 10 respectively, and the time unit is a minute.

1202

G.S. Jung et al. Table 2. Usability Evaluation of three different platforms

Platform Interface Rendering style Mobility Ease of input Novelty Excitement Time to become skilled

PC Keyboard Simple NPR 0.0 9.3 3.3 4.2 3.6

Tablet PC Virtual control using MR Simple NPR 5.3 5.7 8.8 6.3 13.7

PDA key button Simple NPR 9.5 6.7 6.0 5.7 6.5

Since the mobility is limited to the effective area of keyboard input, marker recognition, or wireless LAN, we can take it for granted that the order is PC, tablet PC, and PDA. On the contrary, the easiest input interface is a PC keyboard because users are fully experienced with keyboard input, and also exposed to various PC games. For novelty meaning the sense of new experience, the virtual control of tablet PC results in the highest average that may be cased by curiosity with a different unique interface from their usual one when they play a game. In the case of excitement, users evaluate a virtual control of tablet PC is the most exciting thing, but the order partially resulted from novelty as well as the fact that it is a more tangible interface. As for the final item, the time needed to become skilled, users spent the longest time on learning how to control the game under the virtual control interface. Virtual control interface using MR ranked lower in mobility, ease of input, time to become skilled, whereas higher in novelty and excitement. Interpreting these results, the first reason is that users were not familiar to the virtual control interface; the second is that the interface design was not optimized to the game yet, that is, the real actions of users are not appropriately mapped to the physics in the game, and the third is that they respond to the game with lower sensitivity caused by network delay and marker tracking. Despite the lower ranks of virtual control using MR in mobility, easiness of input, and time to become skilled, we still have a prospect for that interface especially in the situation where users have some difficulties to freely use a keyboard, key button, and stylus. In addition, for users who want more exciting entertainment related to real physical actions, it could provide a good interface under the guarantee of registration accuracy, delay limit, and processing power. Therefore, in the case of more powerful handheld devices than now, we can fully expand it to effective interface of a new game.

7 Conclusions and Future Research In this paper, we proposed a test-bed prototype system to deal with dynamic and huge graphic data on heterogeneous platforms. The system presented a possibility or a paradigm to enable various clients to manage their shared virtual world in their hardware/software environment. Recently, we have considered an environment in which a PC user playing a game in photorealistic rendering style and with multiple-key combinations, a tablet PC user on handheld MR interface with limited mobility, and a


1203

PDA user with a simple NPR interface in mobile environment are sharing their game world by more natural interaction metaphor. Although we have implemented the cooperated prototype system as well as each client-server connections, the system still has some limitations in scalability, usability, and performance. Therefore, in the near future, we will improve the proposed system into more effective system, with a natural interaction metaphor and using the visibility of the game configuration. Acknowledgment. This work was supported by the Brain Korea 21 Project in 2006.

References 1. ARToolKit : http://www.hitl.washington.edu/artoolkit 2. GLIntercept : http://glintercept.nutty.org 3. Headon, R., Curwen, R.: Movement Awareness for Ubiquitous Game Control. Personal and Ubiquitous Computing 2002. Vol.6. No.5-6. Springer-Verlag, London (2002) 407-415 4. Henrysson, A., Billinghurst, M., Ollila. M.: Face to Face Collaborative AR on Mobile Phones. Proc. of ISMAR 2005 (2005) 5. Hosseini, N., Georganas, D.: MPEG-4 BIFS Streaming of Large Virtual Environments and Their Animation on The Web. Proc. of Web3D (2002) 19–25 6. Jung, G.S., Jung, S.K.: A Streaming Engine for PC-based 3D Network Games onto Heterogeneous Mobile Platforms. Edutainment2006. LNCS. Vol. 3942. Springer-Verlag, Berlin Heidelberg (2006) 797-800 7. Jung, G.S., Kwon, S.I., Park, M.W., Won, K.H., Jung, S.K.: A New Dynamic ClientServer Scene Graph for Mobile NPRQuake on Heterogeneous Frameworks. Proc. of CASA2005. (2005) 37-42 8. Lee, D.H., Jung, S.K.: Computer Vision-Assisted Interaction in X3D Virtual Environment on WWW. LNCS. Vol. 2713. Springer-Verlag, Berlin Heidelberg. (2003) 332-340 9. Milgram, P., Takemura, H., Utsumi. A., Kishino. F.: Augmented Reality: A class of displys on the reality-virtuality continuum. SPIE: Telemanipulator and Telepresence Technologies, (1994) 10. Quake II. Id Software : http://www.idsoftware.com 11. Rohs, M.: Real-World Interaction with Camera Phones. 2nd International Symposium on Ubiquitous Computing Systems (UCS 2004). LNCS. Vol. 3598, Springer. (2005) 74–89 12. Rohs, M.: Marker-Based Embodied Interaction for Handheld Augmented Reality Games. Proc. of PerGames at PERVASIVE 2006 (2006) 13. Rohs, M., Zweifel, P.: A Conceptual Framework for Camera Phone-based Interaction Techniques. PERVASIVE 2005. LNCS. Vol. 3468. Springer. (2005) 171–189 14. Teler, E., Lischinski, D.: Streaming of Complex 3D Scenes for Remote Walkthroughs. Proc. of EUROGRAPHICS 2001 ( 2001 ) 17-25 15. Wagner D., Pintaric T., Ledermann F., Schmalstieg D.: Towards Massively Multi-User Augmented Reality on Handheld Devices. Proc. of PERVASIVE 2005 16. Wagner, D., Schmalstieg, D.: First Steps towards Handheld Augmented Reality. Proc. of ISWC2003. White Plains. IEEE Computer Society, NY USA. (2003) 127–137

Manipulation of Field of View for Hand-Held Virtual Reality Jane Hwang1, Jaehoon Jung1, and Gerard J. Kim2,* 1

Dept. of Computer Science and Engineering POSTECH Pohang, Korea [email protected] 2 Dept. of Computer Science and Engineering Korea University Seoul, Korea [email protected]

Abstract. Today, hand-held computing and media devices are commonly used in our everyday lives. This paper assesses the viability of hand-held devices as effective platforms for “virtual reality.” Intuitively, the narrow field of view of hand-held devices is a natural candidate factor against achieving an effective immersion. In this paper, we show two ways of manipulating the visual field of view (perceived or real), in hopes of overcoming this factor. Our study has revealed that when a motion-based interaction was used, the FOV perceived by the user (and presence) for the small hand-held device was significantly greater than the actual. The other method is to implement dynamic rendering in which the FOV is adjusted depending on the viewing position and distance. Although not formally tested, the second method is expected to bring about higher focused attention (and thus immersion) and association of the visual feedback with one’s proprioception. The paper demonstrates the distinct possibility of realizing reasonable virtual reality even with devices with a small visual FOV and limited processing power through multimodal compensation. Keywords: Computer Interface, Human Factors, Virtual Reality, Hand-held.

1 Introduction The performance and functionalities of hand-held computing and media devices have advanced dramatically in recent times. Hand-held devices are those computer embedded systems that are small and light enough to be held in one hand such as personal digital assistants (PDA), cell phones, ultra mobile computers, and portable game consoles. Today’s hand-held computers are equipped with processors approaching 1GHz performance with graphics accelerating chips, sound/signal processing modules and a sleuth of sensors (e.g. camera, gyros, light sensors) and multimodal displays (e.g. vibrators, auto-stereoscopic display). Nevertheless, one can be skeptical whether such devices can be used for “virtual reality,” e.g. to the extent *



Manipulation of Field of View for Hand-Held Virtual Reality

1205

of eliciting immersive feelings (not just for 3D contents viewing). One obvious reason is the very narrow field and small absolute size of the visual display in handheld devices. It is not obvious whether interactivity alone can sufficiently enhance the virtual experience to achieve reasonable “VR.” In this paper, we propose two ways of manipulating the visual field of view (perceived or real) in hopes of overcoming this factor, and show the feasibility of the hand-held VR. We conducted an experiment comparing the perceived field of view (FOV), the level of immersion and presence among the users’ of various VR platforms including the hand-held device. That is, the VR platforms were varied in their sizes of physical/software FOV and in styles of interaction. In this comparative study, we considered the use of a motion based interaction as the factor for the style of interaction. Motion based interaction (e.g. gesture, direct interaction) is already considered a desirable style of interaction for virtual reality systems [1]. This is because it involves many parts of our body (if not the whole) and leverages on one’s sense of proprioception, improving the overall user felt presence and immersion (and even task performance) [9]. In the case of hand-held devices, the motion based interaction also becomes coupled with the visual display/head (a situation unique to the hand-held device) because the sensors and the displays are all physically integrated (and moving) together. Currently, interaction in the handheld devices is still mostly button and finger-based and naturally, one way is to enrich the user experience is to provide the body-based and motion based. The paper hopes to demonstrate the distinct possibility of realizing reasonable virtual reality even with devices with a small visual FOV and limited processing power through multimodal compensation. The other method is to implement dynamic rendering in which the FOV is adjusted depending on the viewing position, and distance. For instance, the user can move the device in and out to widen or reduce the FOV (i.e. window metaphor). Although not formally tested, the second method is expected to bring about higher focused attention (and thus immersion) and association of the visual feedback with one’s proprioception. For both methods, sensors (but inexpensive) are required to approximately detect user motion and track one’s head (or view) position and direction.

2 Motion-Based Interaction and Perceived FOV The main purpose of this study was to assess the feasibility of virtual reality with relatively small screens (as in the hand-held devices) by taking advantage of other system factors, such as multimodal interaction, especially motion based interaction. The basic approach was to have the subjects navigate through a given virtual environment (Experiment I) and search and select objects (Experiment II) using VR platforms differing in their screen sizes and software (geometric) FOVs (SFOV). The software FOV (SFOV) differs from the actual physical FOV (PFOV) in that it refers to the angle encompassing a given scene in its original scale.

1206

J. Hwang, J. Jung, and G.J. Kim

2.1 Experiment Design and Procedures There were five conditions for the two experiments as shown in Table 1. The primary condition represents the motion based hand-held platform, and as comparison groups there were four others. The button-based hand-held platform represents the current form of the hand-held devices. To see the effects of “hand grasping” (whether the mere hand grasping contributes to a possible immersion or sustained attention), a small (same as the hand-helds) screen platform condition was also added. The desktop monitor and large PDP (Plasma Display Panel) display based platforms represent the nominal VR platforms with larger display sizes (larger PFOV and SFOV). The motion-based interface was implemented by attaching a USB camera to the hand-held computer (VGN-U71P from SONY) and using the optical flow information to estimate the amount and direction of the movement in mainly three directions (right/left, up/down, forward/backward). For details, see [4]. Table 1. Six test conditions in the experiment. Asterisks denote average values. Platform Characteristics Test Groups Hand-held / Motion based Hand-held / Button based Small Screen / Keyboard & Mouse Desktop Monitor / Keyboard & Mouse Large PDP / Keyboard & Mouse

Screen Size: width x height 10cm x 7.5cm 10cm x 7.5cm 10cm x 7.5cm 34cm x 26cm 68cm x 51cm

Viewing Distance / Physical FOV 37.97cm* / 15° 37.97cm* / 15° 37.97cm / 15° 63.44cm / 30° 58.88cm / 60°

Software FOV 30° (200%) 30° (200%) 30° (200%) 45° (150%) 60° (100%)

We performed two experiments. In Experiment I, the users were questioned for presence, system usability, enjoyment and perceived field of view after navigating through the virtual environment in the given platform. In Experiment II, the task (navigating and selecting objects) completion time was measured. A one factor within-subject experimental design was for both experiments. In both cases, the independent variable was the type of the VR platform. The major dependent variables for Experiment I were the level of presence/immersion, various usability, and perceived FOV, and for Experiment II, the task completion time. As the first experiment assessed presence and immersion, the respective subjects had to fill out an immersive tendency questionnaire which was adapted from the work by Witmer and Singer [11] and Slater [8]. As for assessing the perceived FOV, the questionnaire included a question of whether the display of the given VR platform provided sufficient FOV. In another assessment, the user was asked to mark on a


(a)

(b)

(d)

1207

(c)

(e)

Fig. 1. The VR platforms tested: (a) Hand-held / Motion based (b) Hand-held / Button based c) Small Screen / Keyboard & Mouse (d) Desktop Monitor / Keyboard & Mouse (e) Large PDP / Keyboard & Mouse

pictorial snapshot of the virtual environment the extent to which one felt one could see through the display. Fig. 1 shows snapshots from the experiment. Twenty-five subjects participated in each of the experiment. We omit further details of the experiment due to lack of space. 2.2 Results We used the ANOVA to verify the significance of the conditions and the StudentNewman-Keuls test for grouping the test conditions with respect to the statistical results. We give a report to the major findings of our study with regards to the perceived FOV, presence, immersion, usability, task performance, enjoyment and cyber-sickness. Due to the limitation of the space, we will show the benefits to the perceived FOV, presence and immersion. Fig. 2 shows the ANOVA results for the case when subjects marked the extent of their perceived FOV, and it suggests that the perceived FOV is significantly widened compared to the PFOV (F4,96 = 12.72, p < 0.0001 in marking). In addition to the support by the existing literatures [6][8], the perceived FOV, in our study, also had a strong correlation with immersion and presence (Pearson correlation value was 0.301, p = .001 with immersion, correlation value was 0.475, p < .0001 with presence). Presence and immersion significantly improved with the motion based interaction (immersion: F4,96 = 5.38, p = .0005, presence: F4,96 = 17.43, p < .0001) (See Fig. 2). Note that the perceived FOV of the hand-held with the motion-based interface is almost twice that of the actual, while the presence level is comparable to those with the desktop and large projection screens.

1208


Fig. 2. Perceived FOV / Presence vs. various VR platforms

3 Dynamic FOV Adjustment for Hand-Held Display 3.1 Manipulating Software FOV As already described, the narrow FOV and small size of the hand-held display (without any other provision) can cause lowered immersion in the hand-held VR. In addition, we claim that the fixed FOV despite changing viewing distance is also unnatural and can bring about similar effects (See Fig. 3(a)).

Hand-held Display

(a)

Hand-held Display

(b)

Hand-held Display

(c)

Fig. 3. Considering eye-display distance in hand-held VR: (a) Nominal Hand-held VR, (b) Hand-held VR as a magnifying glass, (c) Hand-held VR as a see-through window into VE

Therefore, to alleviate this problem, we suggest two different software FOV manipulation techniques using an approximate measurement of the eye (or head) position relative to the hand-held device. We attached and interfaced the SRF10 ultrasonic range finder to the hand-held device, and it can detect obstacles in the range of 3 cm to 6 m from the hand-held device screen. The first proposed FOV technique is to adjust the visual FOV to mimic the behavior of a magnifying glass (see Fig. 3 (b) and Fig. 4). The FOV becomes narrower as the view distance is reduced. This method is useful for the applications in which the detailed views of the object are important but size perception is not. The second proposed FOV technique is to use the hand-held device in an opposite way, as a see-through window into the virtual environment (see Fig. 3(c) and Fig. 5). As the head gets closer to the screen (or window), there are more parts of the virtual


1209

environment visible, thus the FOV widens (and objects are drawn smaller). As you can see in Fig. 5, the size of the virtual object “perceived to the user” is kept the same regardless to the eye-display distance. This approach is better suited for applications in which size or spatial perception is important such as medical training VR systems.

(a)

(b)

(c)

Fig. 4. Hand-held VR as a magnifying glass; the size of the virtual object looks bigger when the hand-held display is close to the eyes: (a) close (15 cm), (b) medium range (30 cm), (c) far range (60 cm)

(a)

(b)

(c)

Fig. 5. Hand-held VR as a see-through window; the size of the virtual object looks same because we adjusted the software FOV using the eye-display distance: (a) close (15 cm), (b) medium range (30cm), (c) far range (60cm)

3.2 Model Simplification Considering Visual Perception of User Hand-held devices are still not as powerful in terms of their graphic capabilities compared to the desktop environments that most users are accustomed with. View dependent simplification can be applied along with the varying the FOV to enhance the perceived quality of the target model and the system performance at the same time. In the case of a magnifying glass hand-held VR, as the view distance becomes larger and the FOV widens (objects smaller), the mesh can be simplified because the user’s detail perception will be trivial (See Fig. 6). In the case of the see-through hand-held VR, the opposite rule is applied. This technique can provide more visual detail and realism given the same amount of system resource. Such a system optimization is important for a reduced platform such as hand-held devices.

1210


(a)

(b)

(c)

Fig. 6. Simplifying the mesh with the changing eye-display distance. The case of hand-held VR as a magnifying glass: (a) close (15 cm), (b) medium range (30cm), (c) far range (60cm).

4 Discussion and Conclusion This paper presented the several ways to manipulate the perceived and software FOV of the hand-held display in hopes to realize the virtual reality providing sufficient realism, immersion and presence on a reduced and small platform like hand-held devices. The FOV was chosen for manipulation because it was expected that the screen size was the most negative factor toward realizing the true meaning of VR on hand-held devices. The results have shown that the motion-based interaction, a unique characteristic of hand-held devices, can help improve the perceived FOV and presence/immersion up to a level comparable to the nominal VR platforms. We believe the widely perceived FOV is due to the similarity of the motion-based interaction to how humans obtain a wide FOV through one’s eye ball movement (saccadic movement) and neck movement. In addition, the software FOV can be adjusted with view dependent mesh simplification to provide more realistic or practical display conditions and improve the perceived quality and focus to the user. Because the device is held in user’s hand, this means that the size, quality and spatial perception is inherently tied to one’s proprioception to be even more effective. In the process of the study, a camera/accelerator based relative tracking technique has been developed to realize the motion based interaction on the hand-held device, and an inexpensive ultrasonic range sensing was developed for an approximate tracking of the user’s head relative to the display screen. With these results alone, it is plausible to claim that hand-held VR has a distinct potential to provide sufficient detail and immersion for high presence (e.g. comparable to nominal VR systems). This can be even furthered with other added elements such as multimodality (e.g. voice, tactile/haptic feedback, stereo display, etc), and environment binding/interaction (e.g. playing motion based golf on a grass field).


1211

We are currently working on adding more multimodal sensing and displays for hand-held VR such as multiple vibro-tactile displays and 3D sound. At the same time, we are continually improving the accuracy of the camera based motion-based interaction and ultrasonic sensor based user tracking. Further validation is needed through exploration of various hand-held VR applications and usability studies as well. Acknowledgement. Parts of this paper was submitted to ACM VRST ’06 and Springer Virtual Reality. This work was supported by the grant from the Virtual Reality and Computer Graphics ITRC program by the Korean Ministry of Information and Communication.

References 1. Bowman, D. A., et al. A Survey of Usability Evaluation in Virtual Environments: Classification and Comparison of Methods. Presence: Teleoperators and Virtual Environments, 11, 4, (2002) 404-424 2. Bowman, D. A. and Hodges, L. F. An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments. In Proceedings of the symposium on Interactive 3D graphics, Rhode Island, United States, (1997) 3. Cohen, J., Duca, N., Luebke, D., Schubert, B. GLOD: A Geometric Level of Detail System at the OpenGL API Level. IEEE Visualization 2003, Seattle, WA (2003). 4. Hwang, J., Jung, J., Kim, G. J. Hand-held Virtual Reality: A Feasibility Study, In Proceedings of ACM Virtual Reality Software and Technology 2006, (2006) (to be appeared). 5. Lee, S., et al. Formation of spatial presence: by form or content?. In Proceedings of 7th Annual International Presence Workshop, Valencia, Spain, (2004) 6. Lin, J. J.-W., et al. Effects of field of view on presence, enjoyment, memory, and simulator sickness in a virtual environment. In Proceedings of IEEE Intl. Conf. on Virtual Reality (VRST '06), (2002) 7. Maringelli, F., et al. The Influence of Body Movement on Subjective Presence in Virtual Environments. Human Factors, 40, 3, (1998) 469-477 8. Polys, N. F., et al. Effects of information layout, screen size, and field of view on user performance in information-rich virtual environments. In Proceedings of ACM Virtual Reality Software and Technology (VRST '05), Monterey, California, USA., (2005) 9. Slater, M. A Note on Presence Terminology. Presence-Connect. 3, Jan. (2003) 10. Slater, M. Measuring Presence: A Response to the Witmer and Singer Presence Questionnaire. Presence: Teleoperators and Virtual Environments, 8, 5, (1999) 560-565 11. Witmer, B. G. and Singer, M. J. Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence: Teleoperators and Virtual Environments, 7, 3, (1998) 225-240

Hierarchical 3D Data Rendering System Synchronizing with HTML Yousuke Kimura1 , Tomohiro Mashita2 , Atsushi Nakazawa1,2, Takashi Machida1,2 , Kiyoshi Kiyokawa1,2, and Haruo Tamekura1,2 1

Graduate School of Information Science and Technology, Osaka University 2 Cybermedia center, Osaka University

Abstract. We propose a new rendering system for large-scale, 3D geometic data that can be used with web-based content management systems (CMS). To achieve this, we employed a geometry hierarchical encoding method “QSplat” and implemented this in a Java and JOGL (Java bindings of OpenGL) environment. Users can view large-scale geometric data using conventional HTML browsers with a non-powerful CPU and low-speed networks. Further, this system is independent of the platforms. We add new functionalities so that users can easily understand the geometric data: Annotations and HTML Synchronization. Users can see the geometric data with the associated annotations that describe the names or the detailed explanations of the particular portions. The HTML Synchronization enables users to smoothly and interactively switch our rendering system and HTML contents. The experimental results show that our system performs an interactive frame rate even for a large-scale data whereas other systems cannot render them.

1

Introduction

The widely used World Wide Web provides many types of information including texts, sounds and 2D images for various purposes. These sorts of multimedia data are very helpful to enrich contents and enhance users’ experiences. Use of 3D geometric data is an effective method for understanding contents because users can browse them from arbitrary viewpoints. Our goal is to develop a 3D geometric data browser along with user interfaces for the purpose of webbased contents management systems (CMS) into World Wide Web. Our browser can render large-scale geometric data very quickly even on not so powerful user terminals. We designed this system to compensate for three issues. First, this software should work independently of users’ hardware and software environments. Previous solutions have needed additional plug-in software to render 3D geometric data, thus users had to install the software in advance. Additionally, this sorts of plug-in software can not always be ported to all platforms. To resolve this issue, we developed our software as a Java applet so that it would work on any platform including mobile terminals. The next issue is the rendering performance. Compared to texts or images, rendering 3D geometry requires much computation cost, in particular for rendering large-scale geometric data. Further, our rendering system must work at Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1212–1222, 2006. c Springer-Verlag Berlin Heidelberg 2006

Hierarchical 3D Data Rendering System Synchronizing with HTML

1213

Fig. 1. Architecture of Our Rendering System

an interactive frame rate in any environment - from desktop PCs to mobile terminals. For this issue, we introduce a hierarchical geometry encoding method “QSplat[1]” as a 3D geometry description format. With the QSplat rendering system, we can adjust the image quality of the rendered 3D model and its frame rate according to the terminal’s environment and users’ interactions. The third issue is the synchronization between the geometric data and other data in CMS. Because we want to use 3D geometric data as multimedia data in web-based CMS, the objects’ locations or objects’ portions in the data should be associated with other information, such as texts or sounds in the CMS database. Therefore, we develop a method to draw annotations at the particular positions of the objects, and for HTML synchronization functions. Annotations show names and details onto the geometric data, and HTML synchronization offers user’s much interactive browsing experience between the 3D object and the HTML browsers. If users click the links in the HTML browser, the viewing position in the 3D geometric data browser smoothly moves to the particular position. On the other hand, when annotations in the 3D geometric data browser are clicked, the associated HTML or other multimedia data are activated and users can smoothly see and hear a detail description of the portion. Several solutions, such as VRML, have been proposed to show 3D data in web browsers, called Web3D. But these usually cannot be applied when the objects are complicated, because the size of such data is enlarged by a complicated 3D geometric object and it is hard to render in an interactive frame rate. Several commercial Web3D technologies, such as Cult3D[2], Viewpoint[3], Shockwave 3D[4] solve the data size problem and have various functions for more effective rendering or animation. Point-based rendering is more useful than polygon-based rendering in cases where the shape of the 3D object is complex[5]. Here, the model is assumed to be of a lot of points of various sizes. This sorts of rendering method can easily introduce multiresolution rendering, which enables dynamic changes in the resolution of models while the rendering procedure proceeds. QSplat, Surfels[6], and several other systems[7][8][9] are examples of point-based rendering

1214

Y. Kimura et al.

methods. On our approach, we employed the QSplat method for rendering and data transmission. In the reminder of this paper, we first give an overview of our systems in Sect. 2. Next, we describe QSplat, on which we base our rendering system, in Sect. 3. We explain the way of storing and rendering annotations in Sect. 4. In Sect. 5, we present an example of synchronization of object rendering and HTML browsers. The experimental results are shown in Sect. 6. Finally, we show applications of our method and give conclusions in Sect. 7 and Sect. 8 respectively.

2

System Overview

The architecture of our system is shown in Fig. 1. This system works on a web browser, Java Virtual Machine (JVM) and JOGL (Java Bindings for OpenGL) [10]. First, the web browser downloads the html file and applet. The applet loads the 3D data (QSplat file) and annotation files from the server, then it renders the 3D model using JOGL. The annotation files manage information such as annotation IDs and descriptions, or relationships to the 3D model. Annotation IDs are used in both HTML (JavaScript) and the Java applet for HTML synchronization. The description can contain various information including words, sentences, hyperlinks or numerical data. The field is used not only for text but also for hyperlinks of sound data or viewing position and directions.

3

QSplat Rendering

QSplat is a kind of point-based rendering system. The object is expressed as a set of the points (SPLATs) of various sizes. Each splat is defined by the parameters of; center position, radius, normal, normal cone, and optionally color. Further, the whole object geometry is constructed as a tree structure (Fig. 2). Here, one node corresponds to a splat. One parent node has up to 4 child nodes. Higher level nodes correspond to a rough geometric structure, and lower level nodes correspond to the details. In the QSplat file, each splat’s parameters are quantized so that one splat data is encoded to 4 bytes (if color data, 6 bytes). On the rendering stage, the tree is searched from high-level to low-level nodes. Tree traversal is stopped at the appropriate level and the nodes of that level are rendered. Here, the visibilities of the nodes are also considered. As the target nodes are at a lower level, the result becomes finer and more time is necessary for rendering. Thus, if the processing power is limited, the program draws higher level nodes and keeps interactive frame rates (Fig. 3). The system can draw the data while the data file is being loaded from server, because the QSplat data are stored from high-level to low-level data in the file. Thus, the user doesn’t have to wait until the system loads all the QSplat data from the server. This is a good feature for low bandwidth environments and


1215

Fig. 2. QSplat Tree

Fig. 3. Trade-off between Rendering Speed and Resolution

reduces user irritation. For the same reason, our system draws low resolution (high level) data while a user is changing viewpoints in the browser. During this operation, a quick and interactive response is achieved for user operations.

4

Annotation

Annotations show names or explanations of 3D models or portions of 3D models, and are rendered with the 3D model. Each annotation has a unique ID (annotation ID). Annotation IDs are commonly used in rendering systems and HTML. How to use annotation ID is described in detail in Sect. 5. 4.1

Annotation Files

Annotations are related to the nodes or leaves in a QSplat tree. If the associated QSplat nodes are visible and rendered, the annotation is also rendered near the splats. In most cases, one annotation corresponds to multiple nodes. In such cases, we associate the annotation and the root-nodes of the highest subtrees. Figure 4 shows the case that the annotation “Head” is related to the 3 splats indicated by arrows. Because the annotation and the splat nodes are not in a one to one correspondence, we used two files to describe the annotation: annotation and association files. An annotation file describes unique annotation IDs, view information (position, rotation, rotation axis, field of view) and descriptions. The view information

1216

Y. Kimura et al.

Fig. 4. Relating Annotations to a QSplat Tree

Fig. 5. Samples of Annotation and Association Files

is used for the HTML synchronization function. When a user wants to see a particular portion of the 3D data that accords with the annotation name, this view information is used to set the correct viewpoint. The association file contains the relationship between the annotation and the splat indexes. Here, one annotation may correspond to multiple Splats. Figure 5 shows an example. To make annotation and association files, we developed the editing tool shown in Fig. 6. Designers can easily create and edit annotations by clicking the splats of 3D data and writing texts or other descriptions. 4.2

Drawing Annotations with a 3D Model

Annotations must be drawn at the ”free space” of the 3D model or other annotations; arrows indicate correspondences between annotations and corresponding splats (Fig.7). Here, two issues need considering; 1. how to find the ”free space,” and 2. how to avoid ”crossing” of the arrows. We describe the algorithm below. First, we find the rectangular region where the 3D model is rendered. We call this region the “model-rendered area”(Fig. 7(a)). The model-rendered area is then split into 4 areas (Fig. 7(b)). The position of the annotation is decided by which area the target splat is located in (Fig. 7(b)). For example, if the splats are rendered in area 1, the annotation is located in the right side of the model-rendered area. If multiple sets of annotated splats are located in the same area, the vertical order within the area is used for the arrangement of the annotations. Namely, if three sets of annotated splats are rendered in an area in a particular order, the


1217

Fig. 6. Annotation Editing Tool

(a)

(b)

Fig. 7. Locating Annotation. (a) Compute model-rendered area. (b) Divide modelrendered area and locate annotations.

three annotations are drawn in the free space of the area with the same vertical order of the splats. Finally, arrows are drawn to connect the annotations and the splats.

5

Synchronization with HTML

Our applet can interact with other HTML contents thorough JavaScript, which can call methods of the Java applet, and the Java applet can call functions of JavaScript. We utilize these features. In our applet, some methods are defined for interacting with JavaScript. The HTML synchronization is realized by JavaScript functions calling these methods and controlling the HTML contents. The architecture of synchronization is shown in Fig. 8. If the user clicks a content (Fig. 8 [A]), the associated JavaScript function is called with an annotation ID as an argument (Fig. 8 [B]), where the annotation IDs are predefined to

1218

Y. Kimura et al.

Fig. 8. Architecture of Synchronization with HTML Contents

identify annotations and contents associated with a particular annotation. This function calls the applet method with the same annotation ID (Fig. 8 [C]), and the view of QSplat is changed to a predefined one according to the annotation ID (Fig. 8 [D]). On the other hand, if the user clicks an annotation in the applet (Fig. 8 [a]), the predefined method is called with an annotation ID (Fig. 8 [b]), Then the predefined method calls the JavaScript function with an annotation ID (Fig. 8 [c]), and these functions control particular contents corresponding with the annotation ID(Fig. 8 [d]). We have made a sample website that expounds on the Great Buddha, as shown in Fig. 9. In this site, our applet is located on the left, and the descriptions of each part of the Great Buddha are located on the right. These descriptions correspond to each annotation in the applet. If headings of the descriptions are clicked, JavaScript functions are called to change the view of QSplat and to show the detailed description. If the user clicks an annotation in the applet, a predefined JavaScript is called, and a detailed description is shown on the right side of web page and the voice guidance is played.

6

Experiments

Two experiments were conducted to verify the usefulness of our rendering system in terms of rendering performance. 6.1

Experiment 1: Comparing Existing Rendering Systems

We compared the rendering speed of our system and that of a VRML viewer, the Cortona VRML Client[11]. VRML viewers are widely-used 3D rendering


1219

Fig. 9. Sample Website with Our Applet Synchronizing with Other Contents

systems that can be executed on web browsers. In Experiment 1, we prepared 4 kinds of VRML data and corresponding QSplat data (Fig. 10) and located them on a server. Then, we downloaded them with a client PC (Intel Pentium 4 2.5 GHz, 768MB memory, NVIDIA GeForce4 Ti4200, Internet Explorer 6.0, JRE 1.5.0.07) through a wired LAN (about 85 Mbps), rendered them with Cortona or our system, and measured their execution time. Table 1 shows the results of Experiment 1. First, our system’s executing time to finish rendering was much shorter. Specifically, the larger the data size, the larger the difference between the execution times. In the case of data 4, Cortona could not finish execution because the data size was too large. When a viewer renders VRML data, a scene graph is made, thus if the data size is large, execution time increases explosively. On the other hand, with QSplat, a scene graph is not made, so even if the data size is large, execution time increases only linearly. Next, because our system renders from lower to higher-resolution data sequentially, users can reasonably quickly see the first view as the time to first view depends on the lowest data. In VRML, users have to wait until all data are rendered. This waiting time gets longer proportional to the data size. In terms of frame rate, the frame rate of the VRML viewer in changing the view was smaller than that of our system for each data size. These experimental results show that users can not smoothly browse large size 3D geometric data with a VRML viewer. However, using our system, the resolution of the image is adjusted to maintain the frame rate so that users can browse even large size 3D geometric data.

1220

Y. Kimura et al.

Data 1

Data 2

Data 3

Data 4

Fig. 10. Experimental 3D Geometry Data Table 1. Comparison of Our Rendering System with a VRML Viewer Number Number Time to Time to Frame Rate of of Begin Finish While Polygons Splats Displaying Rendering Changing [thousands] [thousands] Image [sec.] [sec.] View [fps] VRML QSplat VRML QSplat VRML QSplat VRML QSplat VRML QSplat Data 1 4.4 0.2 69 58 2 1 2 2 *1 8.0 Data 2 71.2 6.7 1087 1757 71 1 71 4 1.2 8.0 Data 3 56.8 8.1 871 2129 51 1 51 5 1.2 8.0 Data 4 142.5 9.9 3109 2591 *2 1 *2 6 *2 8.0 Data Size [MB]

∗1 Enough high speed. ∗2 The system can’t render the model because the data sizes are too large.

6.2

Experiment 2: Executing Our System in Several Environments

We executed our systems on a desktop PC (same one used in Experiment 1) thorough a wired LAN (about 85 Mbps), on a mobile PC (Intel Pentium M 1.1GHz, 504 MB memory, Intel 82852/82855 GM/GME Graphics Controller, Internet Explorer 6.0, JRE 1.5.0.07) through a wired LAN, and on a mobile PC through wireless LAN (about 8Mbps), and measured and compared their execution times. Table 2 shows the results of Experiment 2. The difference of terminal doesn’t materially affect the execution time. It shows that our system has the potential to be used on any terminal. On the other hand, network speed enormously influences execution time. In the case of a wireless LAN, execution time increases in proportion to data size. In actual use, if the data size is too large, our system stops downloading at a proper resolution level and renders the model at that level.


1221

Table 2. Comparing Execution on a Desktop Terminal with on a Mobile Terminal, and Comparing Execution through a Wired LAN with through a Wireless LAN Data Size [MB] Data Data Data Data

7

1 2 3 4

0.2 6.7 8.1 9.9

Number Download + Re-rendering Time of Rendering Time [sec.] After Changing View [sec.] Splats Desktop/ Mobile/ Mobile/ Desktop Mobile [thousands] Wired Wired Wireless 58 1.4 1.6 6.1 0.2 1757 4.2 4.8 59.8 3.0 3.6 2129 4.4 5.4 70.2 3.1 3.9 2591 5.3 6.2 86.4 4.0 4.5

Applications

We designed this rendering system to use with content management systems. For example, e-learning systems such as those involving biology, art or history need this sort of geometric data. Also, applications for digital archive systems need this sort of rendering system. We are now designing a new CMS that can deal with QSplat data and their annotations. Our rendering system is also included in this CMS. The architecture of this CMS is shown in Fig. 11. The system consists of contents registering, website editing, and 3D data rendering systems. The contents registering system manages data including texts, images or other multimedia files. The website editing system generates HTML files from source scripts that users edit with a special markup language. The 3D data rendering system is downloaded with HTML files and renders 3D geometric data on a HTML browser. Our applet is used as the 3D data rendering system. Using this CMS, a user can design websites with various contents including 3D geometric and annotation data, even if the user doesn’t have detailed knowledge of HTML or the architecture of the CMS.

Fig. 11. CMS We Design

1222

8

Y. Kimura et al.


We proposed a new 3D geometric data rendering system that can be used in conventional Web browsers or Content Management Systems. We implemented QSplat rendering as a Java Applet so that it can quickly and precisely render large scale 3D geometric data. Our rendering system can dynamically control the rendering frame rate and the preciseness of the rendering. Thus, it can be used in a wide variety of hardware/ software/network environments such as powerful desktop PCs or mobile wireless terminals. The HTML synchronization technique enables users to smoothly view 3D geometry contents. We will extend this system for use with the Content Management System. We are now trying to implement a website editing system that allows users to make a website with registered 3D geometric data and synchronizing functions. We are also going to implement a content registering system that users can register 3D geometric data through a web interface. For this, we are presently designing a markup language and implementing functions to convert it into HTML.

References [1] S. Rusinkiewicz, M. Levoy, “QSplat: A Multiresolution Point Rendering System for Large Meshes,” Proc. ACM SIGGRAPH 2000, pp.343-352, 2000. [2] “Cult3D,” http://www.cult3d.com/ [3] “Viewpoint,” http://www.viewpoint.com/pub/ [4] “Adobe Shockwave Player,” http://sdc.shockwave.com/products/shockwaveplayer/ [5] T. Fujimoto, K. Konno, N. Chiba, “Introduction to Point-based Graphics,” The Journal of the Society for Art and Science Vol.3 No.1, pp.8-21, 2004. [6] H. Pfister, M. Zwicker, J. Baar, M. Gross. “Surfels: Surface elements as rendering primitives,” Proc. ACM SIGGRAPH 2000, pp.335-342, 2000. [7] F. Duguet, G. Drettakis, “Flexible Point-based Rendering on Mobile Devices,” IEEE Computer Graphics and Applications, Vol.24, No.4, pp.57-63, 2004. [8] Y. Okamoto, S. Yamazaki, K. Ikeuchi, “Efficient Point-based Rendering Method for Huge 3D Models using Sequential Point Clusters,” Proc. Meeting on Image Recognition and Understanding 2004 (MIRU 2004), Vol.1, pp.207-212, 2004. [9] Q. Peng, W. Hua, and X. Yang, “A new approach of point-based rendering,” Proc. Computer Graphics International 2001, pp.275-282, 2001. [10] “Java Bindings for OpenGL,” https://jogl.dev.java.net/. [11] “Cortona VRML Client,” http://www.parallelgraphics.com/products/cortona/.

Fragment Indexing of XML-Based Metadata for Digital TV Contents* Hyoseop Shin1, Minsoo Lee2,**, and Seokhyun Son1 1

Dept. of Internet and Multimedia Engineering, Konkuk University, Seoul, Korea [email protected], [email protected] 2 Dept. of Computer Science and Engineering, Ewha Womens University, Seoul, Korea [email protected]

Abstract. TV-Anytime metadata is being widely accepted as a metadata standard format that describes digital TV contents. It provides a wide spectrum of information of TV programs such as program information, program groups, program schedules, program reviews, program segments, user preferences, etc. These metadata are structured in a large hierarchical XML document. For efficiency, the metadata is split into a set of smaller pieces of metadata fragments that can be transmitted and accessed independently over a broadcast network. In this paper, we propose a fragment-based XML data indexing and querying scheme for efficient processing of TV-Anytime metadata. The experimental results show that the proposed scheme performs better than the generic XML data indexing and querying methods. Keywords: TV-Anytime metadata, XML indexing, metadata fragment.

1 Introduction As an emerging standard for data representation and exchange on the Internet, XML is being adopted and used in more and more industry area. Although many methods of storing and querying XML data have been proposed in the literature [1,2,3,4,5,7,8,9,10,11], they are often not optimal or adequate for a system with domain-specific requirements. While systems in a special domain may be more restrictive in computing power, main-memory size, and secondary disk spaces, they also pose new chance of customization and optimization in storing and retrieving XML data. This is because the characteristics and behaviors of the XML data in a specific domain are likely to be known a priori. TV-Anytime metadata[6] is being widely accepted as a metadata standard format that describes digital TV contents. It provides a wide spectrum of information of TV programs such as program information, program groups, program schedules, program reviews, program segments, user preferences, etc. These metadata are structured in a *

This work was supported by Korea Research Foundation Grant funded by Korea Government(MOEHRD, Basic Research Promotion Fund) (KRF-2005-003-D00281). ** Corresponding author. Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1223 – 1232, 2006. © Springer-Verlag Berlin Heidelberg 2006

1224

H. Shin, M. Lee, and S. Son

large hierarchical XML document and thus for efficiency reason, the metadata is split into a set of smaller pieces of metadata fragments that can be transmitted and accessed independently over a broadcast network. In this paper, we propose a fragment-based XML data indexing and querying scheme for efficient processing of TV-Anytime metadata. The remainder of the paper is organized as follows. Section 2 describes the background of TV-Anytime metadata. Section 3 explains the querying model of TVAnytime metadata. Section 4 presents fragment-based indexing scheme. Section 5 describes the process of the query processing of TV-Anytime metadata. Section 6 gives performance evaluations and our conclusion is given in Section 7.

2 Background: TV-Anytime Metadata The TV-Anytime metadata [6] is conceptually categorized into four areas: the content description metadata which describes the contents, the instance description metadata which describes the broadcasted instances of the contents, the consumer metadata which describes the consumers, and the segmentation metadata which describes the segments of the contents. The metadata deal with the different aspects of the content and the consumers in broadcast environment. TV-Anytime Metadata

Content Description Metadata

Instance Description Metadata

Consumer Metadata

Segmentation Metadata

Program Information

Program Location

Usage History

Segment Information

Group Information

Service Information

User Preference

Segment Group Information

Credits Information Program Review

Fig. 1. Outline of TV-Anytime Metadata Specification

Each metadata is further classified into sub-categories as illustrated in Fig. 1. The content description metadata consists of the program information, group information, credits information, and program review information. Each program and program group is given an identifier called CRID (content referencing identifier) which is used for referring to a content in the other parts of the metadata (e.g. within the instance description metadata or segmentation metadata). The instance description metadata

Fragment Indexing of XML-Based Metadata for Digital TV Contents

1225

consists of the program location information and service information. Instances are divided into the scheduled events and the on-demand events each of which is described in the program location information. The service information describes the broadcast channels. <Title>Sunrise News <Synopsis>Morning News politics economy News Reporter Richard Perry Producer Tom Martin PROG-123 2003-04-29T09:40:00 2003-04-29T10:30:00 <ServiceInformationTable> NBC <ServiceURL>www.nbc.com

Fig. 2. An Example of TV-Anytime Metadata

In Fig. 2, regarding a program identified as ‘PROG-123’, its program information, program location, and service information are described. The program information and the broadcast event are interconnected via the identifier ‘PROG-123’. The consumer metadata describes the consumers rather than the contents, thus this metadata is not usually transmitted from the broadcasters to the consumers. The

1226


consumer metadata consists of the usage history and user preference. The segmentation metadata describes information regarding the concept, ‘segment’ which represents a continuous part of a program. The segmentation metadata consists of the segment information describing segments and segment group information describing group of segments. Each segment group enables non-linear play of contents (e.g. highlights, table of contents, bookmarks, preview, etc). 2.1 Carriage of TV-Anytime Metadata As a TV-Anytime metadata document consists of the various parts as mentioned above and is described in a large hierarchical XML document, it is not efficient to transmit the document in the original format in the broadcast transport layer. The preprocessing of the TV-Anytime metadata for transmitting over uni-directional broadcast streams like MPEG-2 TS consists of fragmentation, encoding, and encapsulation steps. In fragmentation step, the metadata is split into a set of fragments that can be transmitted and updated independently. The standard fragment types defined in TVAnytime metadata includes TVAMain, ProgramInformation, GroupInformation, OnDemandProgram, OnDemandService, BroadcastEvent, Schedule, ServiceInformation, CreditsInformation, PersonName, OrganizationName, Review, ClassificationScheme, CSAlias, SegmentInformaton, SegmentGroupInformation. For example, in Fig. 2., the ProgramInformation (which contains title, genre, etc. of a program), the BroadcastEvent (which contains publishedtime and endtime, etc. of an event), and the ServiceInformation are independently transmitted as differenet fragments. Fig. 3. illustrates the fragmentation of the TV-Anytime metadata documents. TVAMain

ProgramInformation

ServiceInformation

BroadcastEvent

Fig. 3. Fragmentation of TV-Anytime Metadata


1227

After the fragmentation step, the resulting fragments can be encoded into a binary format for space efficiency and then be encapsulated with additional information such as version number and fragment identifier. After the encapsulation step, the resulting metadata stream is inserted into the transport stream as appropriate.

3 TV-Anytime Metadata Querying Model In querying TV-Anytime metadata, the target of users’ queries is metadata fragments that match query conditions which are based on the values of sub-elements of the target fragment type. The typical target fragment types are ProgramInformation, GroupInformation, BroadcastEvent, Schedule, ServiceInformation, CreditsInformation, PersonName, SegmentInformaton, SegmentGroupInformation. The typical examples of sub-elements used for specifying query conditions against tartget fragment types are title, actor, genre, program start/end time etc. The query conditions are represented by conjunctive forms of atom boolean expressions each of which is formed as ‘element-name binary-operator literal’. Assume that a TV-Anytime metadata document D and its tree representation TD are given and a target fragment type is specified by TF, a sub-tree of TD and the query condition qF of which all the element-names exist within the sub-tree TF. The TVAnytime metadata querying model is defined as: Q(TF, qF ) = { f | type(f) = TF ∧ qF(f) = true } For instance, consider a query for the TV-Anytime metadata document in Fig. 2 that requests “the information of programs of which genre is ‘News’”. In this query, the target fragment type is ‘ProgramInformation’ and the query condition is ‘Genre = News’. The result of the query is the ‘ProgramInformation’ metadata fragments whose ‘Genre’ is ‘News’. In general, the target fragment type and the query condition can be expressed in an XML path language such as XPath. The target fragment type is expressed in an absolute path from the root and the sub-elements within the query condition in a relative path from the node that represents the target fragment type. Regarding the query above, the target fragment type is ‘/TVAMain/ProgramInformationTable/ ProgramInformation’, and the sub-element tied with the query condition is ‘BasicDescription/Genre’. Alternatively, the query can be expressed in XPath as : /TVAMain/ProgramInformationTable/ProgramInformation [ BasicDescription/Genre = ‘News’ ]. The naive method of processing the TV-Anytime queries is to uncover every target fragment and to check if it matches the given query condition. However, as the entire TV-Anytime metadata is broken up into many fragments of small size, the method suffers from the performance degradation. In the following section, we propose an indexing mechanism for speeding up searching among a large number of fragments.

4 Fragment Indexing of TV-Anytime Metadata It is not efficient to look into the content of every fragment delivered over the broadcast network only to find some fragments that match the query conditions specified by the values of certain nodes of a fragment type. To avoid this overhead,

1228


fragment indexing is required so that the system is able to find fragments efficiently. From the TV-Anytime querying model in the previous section, we propose the indexing method that frequently referred nodes in query conditions are indexed in advance. Indices are defined on a fragment type using single or multiple nodes within a fragment type as the key. For the XML example in the previous section, an index can be defined on ‘ProgramInformation’ fragments using the element ‘Genre’ to accelerate searching the information of the programs of specific genres. An index consists of a set of entries each of which is formatted as key_value, fragment_id. The key_value represents the value of the index key. If an index key is a composite type, the key_value consists of multiple fields. The fragment_id is the identifier of a fragment. An index of XML fragments can be built in common access methods like B+-trees. As there can be more than one index in the repository, two supplemental structures are provided to assist selecting indices for user-given queries: IndexDictionary, IndexKeyfield. The IndexDictionary maintains the information of each index available in the repository. The structure of the IndexDictionary is . The index_id identifies an index. The fragment_type represents the type of fragments on which an index is built on. The num_key_field is the number of key fields which constitute the index key and the key_field_list lists the key fields of an index. The IndexKeyfield describes the information of the key fields used in the index keys. The structure is . The field_id is the identifier of a key field. The field_path represents the absolute path from the root node to the designated node. The field_type is the type of the node in the repository.

5 Processing of TV-Anytime Metadata Queries The TV-Anytime metadata stored in the repository are retrieved in the fragment basis. A fragment type is specified as the query target and query conditions are specified against the values of the nodes within the fragment type. For example, the fragments in the fragment type 'Program Information' can be filtered on their genres. After the matching fragments are fetched from the fragment repository, they get parsed into a form as desired in the applications. The retrieval process consists of four steps: 1) Selecting Indices select the suitable indices to process the user-given query by looking into the IndexDictionary and IndexKeyfield structures. 2) Filtering Fragment Identifiers filter the identifiers of the fragments that match the query condition by looking into the relevant indices. 3) Retrieving Fragments retrieve the content of the fragments indicated by the fragment identifiers by looking into the fragment repository. 4) Parsing Fragments parse the content of the fragments which are returned as the query result.


1229

6 Performance Evaluations In this section, we examine the performance of the proposed fragment-based indexing scheme (fragment) in the aspects of data loading, reconstruction, and querying, by comparing with the other approaches such as the node numbering scheme (numbering) plus structural joins[10,11] and the attribute inlining scheme[9] (inlining). In the node numbering scheme[10], each element or attribute in the XML document is separately stored as a tuple within a table. Each tuple has the structure of < doc_id, begin_pos, end_pos, level >, where doc_id is a document identifier, and begin_pos and end_pos are the beginning and ending word positions of a tag within the document, and level is the depth of a tag when the XML document is represented as a tree. Under this node numbering scheme, the ancestor-descendant and parentchild relationships between nodes can be verified by examining the tuple values. If a node 'A' is an ancestor of a node 'B'(i.e., A//B), the conditions 'A.doc_id == B.doc_id, A.begin_pos = B.end_pos' are satisfied. If a node 'A' is the parent of a node 'B' (i.e., A/B), the condition 'A.level == B.level - 1' is added to the conditions above. According to these properties, to obtain node pairs that satisfy the 'A//B' or 'A/B' relationships, a structural join operation on the node table is performed using the join conditions above. The attribute inlining scheme[9] handles excessive fragmentation of XML documents by assembling multiple XML elements into attributes of a same relational table. The method inlines as many descendants of an element as possible into a single relation unless a descendant node has an in-degree greater than oneor zero. Otherwise, plural descendants will create another relations. The attribute inlining scheme can automatically generate a relational database schema for XML documents based on a given DTD or XML Schema. Also, it may suffer less from several joins in processing path expression queries than the node numbering scheme. In practice, however, the resulting schema may not be considerably different from that of the node numbering scheme, especially when many of nodes in a DTD specification have in-degree greater than one. In our applications, it turned out that the TV-Anytime metadata contains about 80% plural XML tags. Another serious problem with the attribute inlining scheme in our application is that the approach is so sensitive to the XML data schema that a change of a node would trigger the rebuilding of the database. Meanwhile, the proposed fragment-based approach does not have to rebuild the database unless a fragment type is deleted or a new fragment type is added. We implemented these schemes on top of a relational database engine in a linux computer which has 1GMhz CPU, 512 mega byte main memory and a hard disk installed. Two different datasets were used for the experiments: small synthetic data set which includes 12,000 fragments ( i.e., about six mega byte) and large real data set that represents 14 days program data for 100 channels which is equal to 210,000 fragments (i.e., 70,000 for each one of three fragment types or about 30 mega byte). The experiment was performed in two categories: data loading and reconstruction and data querying. In the loading and reconstruction experiments( Fig.4-(a), (b) ), the fragment method was faster than the other methods. Especially, the reconstruction of the proposed method was faster by up to 20 times. This is because the fragment method suffers less from the overhead of splitting and restructuring of XML data.

1230


In the querying experiments(Fig. 4-(c), (d) ), we tested two different queries. The simple query has single filtering condition and single return field. The complex query has multiple filtering conditions and multiple return fields. The numbering scheme was the worst as in the previous experiments. For the simple query, the inlining 350 300 ) c e s (

250 200

e m it e s n o p s e r

150 100 50 0

small

numbering

dataset

large

inlining

fragment

(a) metadata loading 250 200

c e s ( e m i t e s n o p s e r

150 100 50 0

small numbering

large dataset

inlining

fragment

(b) metadata reconstruction 3 2.5

c e s ( e

2

m i t

e1.5 s n o p s e r

1

0.5 0 small

large dataset

numbering

inlining

fragment

(c) simple query Fig. 4. Performance Evaluations


) c e s (

1231

70 60 50

e im t e s n o p s e r

40 30 20 10 0

small numbering

dataset

inlining

large fragment

(d) complex query Fig. 4. (Continued)

scheme showed similar performance to the fragment scheme, but the performance gap became wider for the complex query. This is because the inlining scheme is required to join many normalized tables to perform the complex query.

7

Conclusion

TV-Anytime metadata as a standard metadata format for describing digital TV programs is getting more and more attention from the industry standard bodies such as ATSC[12] and DVB[13]. TV-Anytime metadata can provide rich descriptions about TV contents. As the metadata is orgarnized in a large hierarchical XML document, it is necessary for the document to be split into smaller pieces called fragments. In this paper, we proposed the fragment-based indexing scheme to accelerate the processing of the typical TV-Anytime metadata querying model. In the experiments, the proposed method outperformed the well-known generic XML data indexing and querying methods.

References 1. Shu-Yao Chien, Zografoula Vagena, Donghui Zhang, Vassilis J. Tsotras, and Carlo Zaniolo, Efficient structural joins on indexed XML documents, In Proc. of VLDB 2002, pages 263-274, Hong Kong, China, August 2002. 2. B. F. Cooper, N. Sample, M. J. Franklin, G. R. Hjaltason, and M. Shadmon, A fast index for semistructured data, In Proc. of VLDB 2001, pages 341-350, Roma, Italy, September 2001. 3. A. Deutsch, M. F. Fernandez, and Dan Suciu, Storing semistructured data with STORED, In Proc. of ACM SIGMOD 1999, pages 431-442, Philadelphia, USA, June 1999. 4. M. Fernandez and D. Suciu, Optimizing regular path expressions using graph schemas, “ In Proc. of ICDE 1998, pages 4-13, Orlando, Florida, February 1998. 5. R. Goldman and J. Widom. Dataguides: Enalbing query formulation and optimization in semistructured databases, In Proc. of VLDB 1997, pages 436-445, Athens, Greece, August 1997.

1232


6. TV-Anytime Metadata Working Group, Specification Series: S-3 on Metadata Version 1.3, TV-Anytime Forum, January 2003. 7. Torsten Grust, Accelerating XPath location steps, In Proc. of ACM SIGMOD 2002, pages 109-120, Madison, Wisconsin, USA, June 2002. 8. Quanzhong Li and Bongki Moon, Indexing and querying XML data for regular path expressions, In Proc. of VLDB 2001, Rome, Italy, September 2001. 9. J. Shanmugasundaram, K. Tufte, C. Zhang, G. He, D. J. DeWitt, and J. F. Naughton, Relational databases for querying XML documents limitations and opportunities, In Proc. of VLDB 1999, pages 302-314, Edinburgh, Scotland, September 1999. 10. D. Srivastava, S. Al-Khalifa, H.V. Jagadish, N. Koudas, J. M. Patel, and Y. Wu, Structural Joins: a primitive for efficient XML query pattern matching, In Proc. of ICDE 2002, San Jose, USA, February 2002. 11. C. Zhang, J. F. naughton, Q. L. David, J. DeWitt, and G. M. Lohman, On supporting containment queries in relational database management systems, In Proc. of ACM SIGMOD 2001, Santa Barbara, CA, USA, May 2001. 12. Advanced Television Systems Committee, http://www.atsc.org. 13. Digital Video Broadcasting, http://www.dvb.org.

Using a MyoKinetic Synthesizer to Control of Virtual Instruments Duk Shin1 , Atsushi Katayama2, Kyoungsik Kim2,3 , Hiroyuki Kambara2,3, Makoto Sato1 , and Yasuharu Koike1,3 1

2

Precision and Intelligence Laboratary, Tokyo Institute of Technology Department of Intelligence and Systems Science, Tokyo Institute of Technology 3 CREST, Japan Science and Technology Agency {shinduk, akatayama, kyoungsik, hkambara}@hi.pi.titech.ac.jp, {sato, koike}@pi.titech.ac.jp

Abstract. We have been developing a new type of human-computer interface, myokinetic synthesizer (MyoKinSynthesizer), using electromyography (EMG) signals. It enables a user to select virtual instruments and to control its properties such as volume and tone without any position or force sensors. The virtual marimba system emulates the basic properties of the real instrument by producing a sound depending on in which of eight zones and how hard the user is hitting. The virtual drum set is composed of 4 different virtual drums controlled by arms and legs. We used a three-layer neural network to estimate position and force of the forearm from EMG signals. After training the neural network and obtaining appropriate weights, the subject was able to control the movement of virtual avatar and to play virtual instruments. The system was destined to be used as a demonstration of VR entertainment and music therapy rehabilitation.

1

Introduction

Muscle contraction is the fundamental unit of body motion and posture control. Coordinated contractions by several muscles allow for various movements of human limbs. Human limb movements are controlled by the central nervous system (CNS). The CNS controls muscle contraction using electrical impulses. Surface EMG signals captured by sensors attached to the skin above the muscle typically contain superpositions of those impulses and these signals connotes the fact that these signals are related to the amount of contraction. By measuring EMG signals, it is possible to predict the next stage of movement or force before motion [1]. Therefore, it is also possible to use EMG signals as an input signals for a human interface system and robot control. There are some approaches to using EMG signals for control of prosthetic arm or robotic hand and control command of human interfaces [2, 3]. The EMG signals in these studies, however, were used just on/off control. NASA scientists have used EMG signals to substitute for mechanical joysticks and keyboards [4]. They demonstrated the ability to control and land Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1233–1242, 2006. c Springer-Verlag Berlin Heidelberg 2006

1234

D. Shin et al.

a computer-simulated airplane using only EMG signals from the forearm. The commands were corresponding to the gesture for controlling the joystick. They also succeeded to recognize typing on a numeric keyboard from EMG signals. These techniques were developed in the field of biomedical engineering such as prosthetic arm. However, in these approaches, the command was corresponded to the specific gesture for controlling the joystick or the keyboard, not continuous motion. The BioMuse has been demonstrated as an interface to music synthesizers [5]. The BioMuse translated the amplitude of EMG signals to MIDI signals directly. However, it takes time for beginners to generate the desired sound. Resently, we proposed that the MyKinSynthesizer approximates the 3D motion and force of the user’s hand based on a musculo-skeletal model synthesized from EMG signals [6]. Here, we have created virtual instruments using MyKinSynthesizer. The proposed system enables a user to control and play virtual instruments continuously using his arms and legs. The user manipulates the instruments in free space and a computer generates the appropriate sound. There is also a computer which is rendered virtual representation of the user and the instruments. The instruments can be controlled in the same way as normal instruments but the technique offers great freedom when engineering the interface since there are no physical or acoustic constraints.

UCRL

DELC PMJC

FCUL

TRAP

(b) Fig. 1. The experiment’s configuration (a) and The electrode positions in EMG measurement for marimba (b)

2

Methods

As an application of this technology we have created a virtual marimba and a virtual MIDI sound player as shown in figure 1 (a). The subject sat on a chair

Using a MyoKinetic Synthesizer to Control of Virtual Instruments

1235

in front of the desk. The subject hit a specified position on a paper marimba or a drum-set with stick. The arm posture of the subject (the choice of a drum or a marimba bar) was calculated from the EMG signals of shoulder muscles. The timing and magnitude of the hit force was also calculated from the EMG signal of elbow muscles or leg muscles. A monitor in front of the subject displayed the virtual avatar and virtual instrument. To create the system a series of experiments aimed at data collection used for algorithm prototyping in MATLAB were first carried out and the creation of a standalone application was implemented using C++. The stand-alone application fetches data from a PC Card A/D-converter and uses a software MIDI device for sound output. To make the interaction more intuitive a 3D graphical representation was also implemented. 2.1

EMG Signals

During the experiment, surface EMG signals were recorded from 6 (virtual marimba) or 12 (virtual drum) muscles as shown in figure 1 (b). For the shoulder posture detection, pectralis major clavicular head (PMJC; drum and marimba), deltoid- scapular part (DELS; drum and marimba), deltoid- clavicular part (DELC; marimba only), Trapezius (TRAP; marimba only) were measured. For the hit detection, the flex. carpi ulnaris (FCUL; drum and marimba), ext. carpi radialis longus (ECRL; drum and marimba), triceps lateral head (TRIA; drum only), Tibialis Anterior (TIBA; drum only) were measured. We recorded EMG signals by using pairs of Ag-AgCl surface electrodes in bipolar configuration. Each signal was sampled at 2 kHz with 16-bit resolution. The signal was digitally rectified, integrated for 0.5 ms, sampled at 200Hz, and finally filtered with a cutoff frequency of 3Hz. If one rectifies, integrates and passes the raw signal through a low-pass filter, the output is a signal proportional to the muscle force. We called the signal gQuasi-tensionh due to their similarity to true muscle tensions [7]. The preprocessing algorithm is visualized in figure 2. Using pairs of silver-silver chloride surface electrodes, EMG activity was recorded. After the raw EMG signal has been sampled at 2 kHz with 16-bit resolution, it is then rectified, integrated over 10 ms and passed through a lowpass FIR filter (finally producing a signal from on referenced as EM Gq ) with an impulse response of the following form: h(t) = 6.44 × (exp−10.80t − exp−16.52t )

(1)

(the sampled filter’s coefficients were normalized by dividing them with the sum of the unnormalized coefficients) 2.2

Position Estimation

For a very rough classification of the horizontal position of a stick held by the user (i.e. three zones indicating only left, center or right), the similar thresholding algorithms could be used to solve problem 2. However, if one needs a more finegrained classification a different approach of using a neural network combining

1236

D. Shin et al.

Fig. 2. Block diagram of the signal preprocessing

input from several muscles can be used to estimate a parameter that is related to the position. With the aim of making the relationship between the network inputs and outputs less complex (as well as having the added benefit of better ergonomics) marimba’s horizontal bars (or drum set) structure was used by an arc of length π divided into discrete zones. A three-Layer neural network, shown in figure 2, was used to identify the input EMG signals as associated shoulder posture, i.e. marimba’s bar detection. The first layer of the neural network consists of the EMG input signals from each muscle. The second layer is the middle layer, and third layer is the output shoulder posture for each bar. The position estimation of the stick used for hitting is performed by using EM Gq from three muscles related the horizontal movement of the shoulder (see figure 1(b)) as inputs to a neural network trained using a back-propagation algorithm. The back-propagation is a supervised training method in which a training session allows the system to learn the relationship between EM Gq and the position of the stick. Because of various muscle activation patterns between the left and right shoulder separately trained networks are used for each shoulder. Since it was desired to minimize reliance on external equipment (such as a motion capture system) to keep the system portable a heuristic training is used: The user is instructed to perform static co-contractions followed by hit movements in predefined zones designated on a screen. An angle θ located in the middle of the zone is then used to represent the target output used for network training. Apart from the input layer the neural network is composed of a 7-node hidden layer (using sigmoid transfer functions) and a single-node output layer (linear transfer function). To increase the precision of the network the number of zones used for training (12) were twice of that used when playing (6 per shoulder with


1237

4 overlapping resulting in a possible spectrum of 8 tones). Because the network output often was found to oscillate around the desired value a moving average filter was placed at the network output to dampen this effect. The network performance was rather poor for extreme outward rotations (i.e. to the extreme right for the right shoulder) and for this reason θ was limited to the the interval π −π [ π2 , −π 4 ] for the right shoulder and [ 4 , 2 ] for the left (θ = 0 located right in front of the user). This means that zones to the extreme left can only be accessed by using the right hand and vice versa. The algorithm used for the position estimation is visualized in figure 2. Capture of training data takes about one minute and the actual weight adaption calculations (incorporating various measures to prevent over-learning) takes less than one minute on an IBM Thinkpad laptop equipped with a 1.6 GHz Pentium M processor. 2.3

Hit Detection and Intensity Estimation

By capturing data from a lower-arm muscle involved in the hit movement we can find satisfying solutions to the problems by only regarding the amplitude and derivative of the filtered signal. In particular, comparing the signal to a threshold value and monitoring when the derivative of the signal changes from positive to negative provides a good estimate of when the user is hitting. Since the filtered EMG signal is a measure of the force used when hitting one way to estimate the intensity is to make it a function of the maximum value seen during the hit movement. Detection of when the user has performed a hit movement is carried out with thresholds being computed from the training data considering only the relationship between the maximal and minimal values of EM GQ (see figure 2). Because multiple local maxima sometimes occurs after the main peak of a hit no new hits are detected until 250 ms has passed since the last detection. The intensity of the sound (as represented by the parameter velocity in MIDI terminology) is calculated according to equation (2) v = a ln(x − b)

(2)

with x being the value of EM Gq immediately after the derivative changes from positive to negative and a and b is calculated to match the function’s range to the available intensity levels according to the hit threshold and maximum value seen during training.

3

Results

Using data from EMG sensors attached to the user’s skin we can estimate the user’s muscle tension, joint stiffness and posture giving us many degrees of freedom to use when creating the interface. The instruments we have created to illustrate the technology are a virtual marimba and a virtual drum set. The marimba is controlled with the arms and has virtual bars laid out in a horizontal fashion like a real marimba. According to the applied force and at which

1238

D. Shin et al.

virtual bar (which can be marked out in an arbitrary way) the user aims the appropriate sound is played. The drum set is composed of 4 different virtual drum controlled by arms and legs. As for the arms, according to where the user is aiming (to the left, center or right) and the force used the proper drum sound is generated. Because of the fact that EMG signals are roughly 100 ms ahead of the actual movement they evoke we can make highly responsive systems (with very good synchronization of sound and computer graphics) compared to other approaches of realizing similar systems (using video camera, motion capture etc). Moreover the system is more portable, easier to set up and could be fabricated to a lower cost. 3.1

Virtual Marimba

The virtual marimba system is controlled with the right arm and has virtual bars laid out in a horizontal fashion like a real marimba. According to the applied force and at which virtual bar (which can be marked on a paper) the user aims the appropriate sound is played. The proposed virtual marimba system used 6 muscles (for hit detection and intensity; FCUL and ECRL, for posture detection; PMJC, DELC, DELS, TRAP see figure 1(b)). Figure 3 shows that the user play the virtual marimba. In this section we analyze two aspects related to the performance of the system. In figure 5, EM Gq is plotted together with the detected hits. As can be seen, although multiple local maxima occurs after the main peak of a hit, the hit detection algorithm performs very well. And also the hit detection algorithm performs well although the threshold level used for detection is mostly a matter of personal preference as long as it is sufficiently elevated to ignore background noise present in the signal. Figure 6 shows the system’s estimation of the position (as represented by the zone number). One can see that the system has problems performing the correct classification just after user’s hand moves between zones. During this time the stick is close to the borders of a zone but since the user typically aims for the middle of a zone this does not lead to problems in practice.

Fig. 3. Virtual marimba system

Fig. 4. Virtual drum system


1239

5

Tutorial signal Network output 1

1 4

Zone

Signal Amplitude

3

0.72

2

0.5

0.5

Detected hits 0

50

100

1

0

150

200

250

300

350

400

450

500

Time

Fig. 5. Example of detected hits (dotted lines with circles) in the case of a threshold of about 0.72 (dashed line)

0

500

1000

1500

2000

2500

3000

3500

4000

Time

Fig. 6. The estimated zone (dashed line) compared to the correct zone (solid line) associated with the right arm

We have created a system for virtually playing instruments. The user manipulates the instruments in free space and the approriate sound is generated by a computer. There is also a computer rendered virtual representation of the user and the instruments. The instruments can be controlled in the same way as normal instruments but the technique offers great freedom when engineering the interface since there are no physical or accoustical constraints. 3.2

Virtual Drum

The proposed virtual avatar system used 2 muscles of legs and 10 muscles of arms. For the shoulder posture, EMG signals of PMJC and DELS were measured. These muscles also used to select virtual drum (snare drum, hi-hat cymbal and crash cymbal). EMG signals of FCUL, ECRL and TRIA were also measured for the elbow hit movement. TIBA of left leg used to hit the hi-hat pedal and TIBA of right leg used to step on base drum pedal. The virtual drum set is composed of 4 different virtual ”drums”; crash/ride cymbal, snare drum, hi-hat and bass drum. 1. Cymbal : The cymbal is played by 4 EMG signals of the right arm corresponding to the beat. 2. Snare-drum : The snare drum is controlled by 8 EMG signals in both arms. 3. Hi-hat : A user can control Hi-hat with two arms. The opening and shutting of Hi-hat is adjusted by On/Off signal from an EMG activity of the left-leg. 4. Bass drum : The bass drum is controlled by EMG signals from the right leg. Virtual drum can generate sound without time delay. Because an EMG is about 100ms ahead of movement, we can discriminate the next posture prior to movement. Based on that the estimation is quicker than the movement, we can generate the appropriate sound and volume of the virtual drum while synchronizing the motion of a virtual skeleton. We perform calibration prior to playing

1240

D. Shin et al.

Detection & Intensity

stick position[mm]

Left(hi-hat)

Right(cymbal) -2680

-2700

-2700

-2750 -2750

-2720

-2800 0

-2800 1

2 3

4

5

6

7

8

9

10

0

-2740 1

2 3

4

5

6

7

8

9

0

1

2 3

4

5

6

7

8

9

10

0 0

1

2 3

4

5

6

7

8

9

10

1

2 3

4

5

6

7

8

9

10

10

30

30

30

20

20

20

10

10

10

0 Estimation of Shoulder position

Center(snare-drum) -2700

0

1

2 3

4

5

6

7

8

9

10

0 0

1

2 3

4

5

6

7

8

9

10

10

10

10

0

0

0

-10

0

1

2 3

4

5

6

7

8

9

10

-10

0

1

2 3

4

5

6

7

8

9

10

-10

0

Fig. 7. Detection and Intensity of a virtual drum set

because there are personal differences on EMG activities. We show in figure 4, a subject plays the virtual drum system. The player throws his arms on the air as if he beats a drum, but the sound of drum is heard. In figure 7, we show some plots of hit detection and intensity during tasks of experiment. Each first plots shows the end postion of the stick. The stick posture of the subject was measured by an OPTOTRAK. Each second plots show the detection of timing and the intensity of sound from EMG signals processing algorithm (see figure 2). The bottom plots show the estimation of shoulder posture (see also figure 2). The data of the plot was sampled at 200Hz. As can be seen at center plots, whether the subject hit quickly or not, the hit detection algorithm performs very well. And the stick movement was always the same but the sound (force) was different. By using an external measurement sensor, such as camera, it is impossible to measure an intrinsic time-varying force. This intrinsic force works as an input variable provided for the human interface resulting in an intuitive operation. we can estimate intrinsic forces and the posture of the limb prior to movement, and then these variables are used as input for the human interface.

4

Conclusions

There are many application to interactive music performance systems in a virtual environment [8, 9, 10, 11, 12]. Most conducting systems use gesture recognition for use by professional conductors, but play back only synthesized audio (MIDI).


1241

However, it is impossible to measure an intrinsic time-varying force by using an external measurement sensor, such as camera or motion capture. We proposed a novel method to describe the design and implementation of a virtual system emulating a pulsatile instrument using EMG measurement. We have also investigated the feasibility of using MykinSynthesizer for human interfaces. Since EMG activities are different depending on the posture and intensity, we can discriminate diverse posture and control the volume corresponding to the amplitude of muscle tension. This intrinsic force works as an input variable provided for the human interface resulting in an intuitive operation. In our syatem, human beings use a diverse combination of muscles to take pose. The avatar posture and the sound selection is controlled by the presence of the user’s hand in different zones as estimated by a neural network and thresholding algorithm. The intensity of the sound is related logarithmically to the amplitude of the filtered EMG signal. Since the system is based on a neural network architecture it is very flexible. Although the usage scenarios described here emulates the traditional style of interfacing with a marimba (or drum) any muscles whose activity can be measured using surface EMG can be used. Sujects will be able to experience what it is like to use virtual instruments and realize the potential of virtual user interfaces that could be controlled by almost any muscle of the body. Since the instruments are virtual not much space is needed. The equipment used are laptop computers, EMG sensors attached on the users skin and a screen showing the computer generated graphics. These virtual pulsatile instrument systems are not aware by the users, so the users would feel that they control the instrument directly by their arms and legs. Because the intensity of the sound is proportional to the degree of his muscle activity. This portable virtual system that could be used at conferences (for technology demonstration purposes) or hospitals (for music therapy rehabilitation).

Acknowledgment This work was supported by grants from the Ministry of Education, Science, Sports and Culture, Grant-in-Aid for Scientific Research (B), 16360169 and JST CREST Program to Y. Sakurai.

References 1. Koike, Y. and Kawato, M. : Estimation of arm posture in 3D-space from surface EMG signals using a neural network model. IEICE Transactions Fundamentals D-II(4), (1994) 368-375. 2. Jacobsen, S.C., Wood, J.E., Knutti, D.F., and Biggers, K.B. :The Utah/MIT dexterous hand: work in progress. Int. J. Robotics Research, vol. 3, no. 4, (1984) 21-50 3. Meek, S.G., Jacobsen, S.C., and Straight, R. : Development of advanced bodypowered prosthetic arms. Journal of Rehabilitation Research and Development, vol. 26, (1989)

1242

D. Shin et al.

4. Bluck, J. : Nasa scientists use hands-off approach to land passenger jet. http://amesnews.arc.nasa.gov/releases/2001/01 08AR.html. 5. Tanaka, A. : Musical technical issues in using interactive instrument technology with application to the BioMuse. in eICMC Proceedingsf (2001) 124-126. 6. Koike, Y. : Tele-Kinesthetic Interaction: Using Hand Muscles to Interact with A Tangible 3D Object, SIGGRAPH2006. (2006) 7. Koike, Y. and Kawato, M. :Estimation of dynamic joint torques and trajectory formation from surface electromyography signals using a neural network model. Biological Cyberbetics, vol. 73 (1995) 291-300 8. Mathews, V. : The Conductor Program and Mechanical Baton, in Max V. Mathews and J. R. Pierce, eds.: Current Directions in Computer Music Research, MIT Press, Cambridge, (1991) 9. Pogue, D. : The dangers of the Digital Orchestra. New York Times Direct, Apr 5, (2001) 10. Borchers,O. : WorldBeat: Designing a baton-based interface for an interactive music exhibit. Proc. CH197, ACM, (1997) 131-138 11. Lee, M., Garnett, G., and Wessel , D. : An Adaptive Conductor Follower. Proc. International Computer Music Conference, ICMA, San Francisco, (1992) 12. Kragtwijk M., Nijholt A., and Zwiers J. : Implementation of a 3D Virtual Drummer. Computer Animation and Simulation, N.Magnenat-Thalmann and D. Thalmann, eds., Springer Verlag Wien, (2001) 15-26.

View-Dependent Simplification of Complex Urban Scenes Using Weighted Quadtrees Bum-Jong Lee, Jong-Seung Park, and Mee Young Sung Department of Computer Science & Engineering, University of Incheon, 177 Dohwa-dong, Nam-gu, Incheon, 402-749, Republic of Korea {leeyanga, jong, mysung}@incheon.ac.kr

Abstract. This article describes a new contribution culling method for the view-dependent real-time rendering of complex huge urban scenes. As a preprocessing step, the view frustum culling technique is used to cull away invisible objects that are outside the view frustum. For the management of the levels-of-detail, we subdivide the image regions and construct a weighted quadtree. The weight of each quadtree node is defined as the sum of weights of all objects contained in the node or its child nodes. The weight of an object is proportional to the view space area of the projected object as well as the distance from the viewpoint. Hence, large buildings in the far distance are not always culled out since their contributions to the rendering quality can be larger than those of near small buildings. We tested the proposed method by applying it to render a huge number of structures in our metropolitan section which is currently under development. Experimental results showed that the proposed rendering method guarantees real-time rendering of complex huge scenes.

1

Introduction

When rendering a large scene, a huge number of objects can reside in the viewing frustum. Urban environments present challenges to interactive visualization systems, because of the huge complexity. Many previous works have been focused on high quality interactive visualization of complex urban environments [1]. A huge number of objects should be drawn to ensure the high quality rendering. However, it slows down the rendering speed, which is not acceptable for realtime rendering. Some insignificant objects need not be drawn and they should be culled away from the set of objects in the scene. Such a culling process should be performed for each rendering cycle and it should be proceeded fast enough to satisfy the real-time requirement. A suitable spatial data structure is required for the fast culling. Visibility culling reduces the number of polygons sent to the rendering pipeline by excluding faces that need not be drawn. There are several different stages of culling[2]: backface culling, view frustum culling, portal culling, and occlusion culling. For the real-time rendering for complex scenes, the use of levels-of-detail (LODs) is necessary. The LOD management involves decreasing the details of Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1243–1252, 2006. c Springer-Verlag Berlin Heidelberg 2006

1244

B.-J. Lee, J.-S. Park, and M.Y. Sung

an object as it moves away from the viewer. The use of LOD increases the efficiency of rendering by reducing the number of polygons to be drawn. An appropriate level of the LOD should be determined automatically to provide the unnoticeable change of the rendering quality. The LOD selection techniques can be classified into three categories: ranged-based method, projected area-based, and hysteresis method [2]. The ranged-based method is to associate the different LODs of an object with different ranges. The projected area-based method uses the projected area of the bounding volume to associate the LODs. The projected area-based method uses the number of pixels of that area, called the screen space coverage. If the LOD varies around some transition value, there is a risk to switch back and forth between two levels from frame to frame. The hysteresis method prevents the LOD flickering by distinguishing between upwards translations and downwards transitions. This paper describes a weighted quadtree structure for fast rendering of complex huge urban scenes. Objects that are far away from the camera do not contribute significantly to the rendering quality. We throw away objects whose projected areas into the screen space are below a given threshold. This is called the contribution culling. The weighted quadtree structure supports both the fast view frustum culling and the contribution culling well. In Section 2, we describe general rendering methods of complex scenes. In Section 3, the proposed weighted quadtree structure is described. In Section 4, experimental results for complex urban scenes are presented. Finally, concluding remarks and future works are described in Section 5.

2

Rendering Complex Scenes

Geometry-based rendering has been actively studied to satisfy the requirement of real-time computation in drawing large complex scenes. The view frustum culling is the most obvious visibility culling[3]. It is based on the fact that only the objects in the current view frustum need to be drawn. A view frustum is usually defined by six planes, namely the front, back, left, right, top, and bottom clipping planes, which together form a cut pyramid. The six bounding planes are represented by πi : ni · x + di = 0 for i = 1, . . . , 6, where ni is the normal of the plane πi , di is the offset of the plane πi , and x is an arbitrary point on the plane. Whether an object is in the frustum is determined by an intersection test of the object and the six planes. If the left side of the plane equation is greater than zero, then a point x is outside the plane. If the value equals to zero, then the point is on the plane. If the point x is inside all planes, then the point is inside the view frustum. As well as the simple intersection test method, there are several optimization methods for the view frustum culling. The octant test reduces the number of intersection tests that should be done. The view frustum is split in half along each axes, resulting in eight parts. Then, if the bounding sphere is inside the three nearest planes, it must also be inside all planes of the view frustum. There is another way to reduce the number of intersection tests. If the bounding volume

View-Dependent Simplification of Complex Urban Scenes

1245

1 2 5

1 3

C 0.3

2 4

B 0.5

5

6 8

10

11

D 0.5 13

12

C

7E 0.2 9

E

6 7

3 4

B

8 9

D

12

A

A 0.8

10 11

13

(a)

(b)

Fig. 1. Terrain cells in the screen space (a) and corresponding regions in the world space (b) 2.3

0.7

0.8

A 0.8

0.2

D

C

B

0.5

0.3

0.5

E 0.2

Fig. 2. A quadtree for the objects in Fig. 1

of a node is completely inside one of the plane of the view frustum, then the bounding volumes of the node’s children also lie completely inside that plane. Hence it is not required to test the plane for the child nodes of the node. The temporal coherence and the camera movement coherence can be exploited to reduce the intersection tests. The temporal coherence is based on the fact that if a bounding volume of a node is outside of a view frustum plane at the previous frame, then the node is probably outside of the same plane. The camera translation and rotation coherency exploits the fact that, when navigating in a 3D space, only one of a rotation around an axis and a translation is commonly applied to the camera. In that case, many objects that were outside the plane at the last frame still remain outside. The fact inspires us to use a hierarchical structure for the view frustum to speed up the culling process [1][4]. As an effective view frustum culling implementation, we use a quadtree data structure. The classical quadtree can be used for the view frustum culling to remove many objects at once from the graphics pipeline [5][6]. The terrain area is divided into cells of regular squares. A bounding box centered at the camera

1246


position is managed during the camera movement. The quadtree structure is recomputed only when the camera crosses the bounding box. When cells are on a frustum boundary, low level quadtree nodes are used to refine the areas in order to reduce the number of objects to be sent to the rendering pipeline. Gudukbay[7] presented a view frustum culling algorithm that traverses the quadtree in a preorder traversal. The frame coherency property simplifies the view frustum culling algorithm. Yin[8] introduced a method which constructs a static quadtree in a preprocessing step and culls out invisible objects by view frustum culling using the quadtree. From the next section, we describe a hierarchical weighted quadtree structure for the efficient implementation of the view frustum culling and the contribution culling. When rendering a complex environment containing a huge number of objects, the rendering speed should not be slowed down. To avoid such dependency on the number of visible objects, we investigate a criteria to choose the most contributive objects for high quality rendering. Small objects in the screen space are culled out even though they are near to the camera position.

Fig. 3. Rendered scenes using simple methods: culling by the distance measure (upper), culling by the number of polygons (lower)

3

Contribution Culling Using Weighted Quadtree

Drawing all visible objects is too slow and inappropriate to real-time rendering systems. Rendering only the most important visible objects is necessary and, hence, appropriate LOD techniques for urban objects are essential. If small details contribute little to the scene then it should be discarded. The contribution of each object is measured as the area of the projection in the screen space. When computing the contribution measure, we consider only the bounding volume of the object. The bounding volume is projected onto the projection plane and the


1247

area of the projection is then estimated in pixel units. If the number of pixels in the area is smaller than a given threshold value then the object is culled away from the rendering pipeline. Our contribution culling method uses a weighted quadtree structure for the rendering of complex urban scenes. We denote the j’th coordinate vector of an object in the world space by Xj = (xj , yj , zj ) and the coordinates in the screen space by xj = (uj , vj ). The overall steps of the proposed algorithm are as follows: Step 1 (Initialization): 1. Compute C1 , the quadrilateral region corresponding to the overall terrain in the screen space. 2. Create an empty quadtree with a root node n1 and associate the node to C1 . Set w1 (the weight of n1 ) to zero. Step 2 (Computing object weights): For each object obji (i = 1 . . . N ), where N is the number of total objects, perform the following steps. 1. Compute xj , the projection of the object point Xj to the screen space. 2. Compute Ai , the area of the bounding rectangle of the object. 3. Compute the weights wi using Ai . Step 3 (Inserting objects to the quadtree): Initialize the node index k to zero and perform the following recursive quadtree construction process. 1. If there are no objects in Ck , goto Step 4. 2. If there is a single object in Ck , set wk to the weight of the node and goto Step 4. 3. If there are more than one objects in Ck , split the associated region Ci into four subregions and create corresponding four child nodes. Set wk to the sum of all object weights inside the quadrilateral region. 4. Repeat Step 3 for the four child nodes. Step 4 (Culling): Choose a fixed number of the most significant objects from the quadtree using the node weights. 1. The sum of the weights of the chosen objects is maximized while traversing the quadtree in the breath-first-order. We describe the above steps in more detail. In Step 1, C1 is the quadrilateral region of the entire terrain, as shown in Fig. 1(a). To split the region into four subregions, we calculate four corners of the region ek (k = 1, . . . , 4). If the four corners of the terrain in the world space are Ek (k = 1, . . . , 4), the corresponding corners in the screen space can be calculated by projecting each Ek onto screen space using the projection matrix. In Step 2, the projection xi of each object point Xi is computed by projecting the point in the world space into the screen space. Then, we calculate Ai , the area of the bounding rectangle of the i’th object, using Eq. (1): Ai = (max(uj ) − min(uj ))(max(vj ) − min(vj ))

(1)

where uj and vj are the screen coordinates of the i’th object. The weight is set to Ai , the area of the bounding rectangle in the screen space.

1248


The quadtree is constructed in Step 3. Each object is inserted to the quadtree together with its weight. The weight is the sum of the weights of all objects in the node and its child nodes. The associated quadrilateral screen region for a node is split into four subregions if the node has more than one object. In Step 4, a fixed number of significant objects are selected. We traverse the quadtree in a breathfirst-order as shown Fig. 2 and choose the fixed number of the most significant objects. Let M be the desired number of objects to be drawn. Then, we try to choose M objects starting from the root node n1 . It is equivalent to choose M objects from the four subnodes of n1 . For each subnode nk (k = 2, . . . , 5), we try to choose (M ∗ wk /w1 ) objects from the node. The selection process is repeated recursively until the desired number of objects are actually chosen. It is possible that the number of the selected nodes is less than the desired number of nodes since some internal nodes and their subnodes have no significant nodes at all or they have fewer significant nodes than expected. Fig. 1(a) shows terrain cells and their associated objects. Each cell region is marked with an identification number and a capital letter. A number for each object indicates the weight of the object. Fig. 1(b) shows the corresponding terrain cells in the world space. Fig. 2 shows the quadtree for the scene in Fig. 1. A circled node is either the root node or an internal node and the squared node is a leaf node. Each non-empty node has a number that is the weight of the node. After constructing the quadtree, the quadtree is searched in the breathfirst-order. For example, if we want to render only two objects, then the second node and the third node are chosen since they have two highest weights in the second level. Then, since the third node has no child nodes, object A is chosen. Then, in the third level, object B is chosen among the four child nodes of the second node in the second level.

4


To demonstrate the effectiveness of the proposed method we have implemented a rendering engine using C++ and DirectX 9.0 on the Windows platform. We tested on a 2.8GHz Pentium 4 PC with 512MB DDR RAM and an ATI RADEON 9200 GPU with 64MB DRAM. For the comparison purpose, two simple methods of LODs are implemented. The first method limits the number of polygons and only the fixed number of the nearest polygons are rendered. The first row of Fig. 3 shows rendered urban scenes using the method. The number of rendered polygons for the images at the upper row of the figure are 960, 3,000, and 5,040, respectively, from left to right order. The second method limits the the visibility range. All the polygons outside the visibility range are culled away. The images at the second row of Fig. 3 are the rendered urban scenes using the method. The visibility ranges are 300, 1,000, and 2,750 meters, respectively, from left to right order. As well as the rendering quality, the rendering frame rate should also be stable when the viewer is navigating in a virtual world. The proposed method has provided almost the same quality of rendering with the same frame rate. The


1249

Fig. 4. Rendered scenes using our method when the camera is walking forward (upper) and looking around at a fixed position (lower)

view magnification experiments and camera rotation experiments are shown in Fig. 4. When the scene is magnified, small objects at far distances are gradually scaled larger, as shown in the upper six images of Fig. 4. The lower nine images of Fig. 4 show the rendered scenes when the camera is rotated 0, 40, 80, 120, 160, 200, 240, 280, and 320 degrees, respectively, at the fixed camera position. During the rotation, the number of rendered polygons and the variation of the rendering frame rate are shown in Fig. 6.

1250


Fig. 5. Comparison of rendered scenes: using no LODs, using view frustum culling, using constant distance, using constant number of polygon, using proposed method

The proposed method has kept high-quality rendering without significant performance depreciation. We compared the rendering quality of five different methods: a simple method without using LOD, the view frustum culling method, the fixed visibility range method, the fixed number of polygons method, and the proposed method. The rendered scenes using those five methods are shown in Fig. 5. The images in the first row are the rendered scenes using the


6

x 105

4 3 2

frustum culling constant distance constant polygon our method

0.09 0.08 speed (1/fps)

5 #polygons

0.1

frustum culling constant distance constant polygon our method

1251

0.07 0.06 0.05 0.04 0.03 0.02

1

0.01 0

0

50

100

150 200 250 camera angle

(a)

300

350

0

0

50

100

150 200 250 camera angle

300

350

(b)

Fig. 6. Comparison of rendering performance for the 360◦ rotating camera: (a) the number of polygons, (b) the rendering speed

novice method without using LOD, which means all the objects are sent to the rendering pipeline. The images in the second row are the rendered scenes using the view frustum culling method. The method renders only the objects that are inside the view frustum and the rendering frame rate is much faster than that of the first method. The images in the third row are the rendered scenes using the fixed visibility range method. The objects that are placed outside the distance over 900 meters do not be rendered. Though the fixed visibility range method provides faster frame rate, the rendering quality is not satisfied. The images in the fourth row are the rendering results using the fixed number of polygons method. The number of polygons was set to 82,720 polygons. Significant objects in far distances were disappeared and the quality is bad. The images in the last row are the rendering results using our proposed method. The number of polygons is always equal to or less than that of the view frustum culling method. However, the rendering quality is as good as that of not using any LOD method. In our methods, both the huge number of tiny objects and the mostly occluded objects are not drawn. We measured the performance when the camera is turning around 360 degrees at a fixed position, whose rendered scenes are shown in lower side of Fig. 4. The camera is rotated with respect to Y -axis at a fixed position. The measurements are shown in Fig. 6. Fig. 6(a) and Fig. 6(b) show the number of polygons and the rendering frame rate, respectively, for the corresponding camera angle. Two graphs are similar since the rendering frame rate is proportional to the number of polygons. The figure indicates that the proposed method reduces the number of polygons by about 50,000 and it is faster than using the view frustum culling by about 0.005sec. The rendering speed is 0.008sec when the camera angle is around 75 degree and the rendering speed is 0.093sec when the camera angle is around 175 degree. The variation is due to the fact that the number of significant objects is varying, as you can check it in lower side of Fig. 4.

1252

5


Conclusion

This article has presented a novel contribution culling method for the real-time rendering of complex urban scenes using the weighted quadtree structure. Objects are culled away from the rendering pipeline when their screen projected areas are too small to contribute to high-quality rendering. Contributions of objects are represented by weights on nodes of the created quadtree. By traversing the nodes, a fixed number of significant nodes are selected to be drawn. The rendering speed is determined by the number of significant objects to be drawn. The proposed method has applied to our new metropolitan region which is currently under development. Experimental results showed that the proposed rendering method guarantees real-time rendering of complex huge scenes. We compared the rendering quality of five different methods and found that the rendering quality of the proposed method is significantly better than those of classical LODs methods at the same rendering frame rate. Acknowledgements. This work was supported in part by grant No. RTI05-0301 from the Regional Technology Innovation Program of the Ministry of Commerce, Industry and Energy(MOCIE) and in part by the Brain Korea 21 Project in 2006.

References 1. Tsuji, T., Zha, H., Kurazume, R., Hasegawa, T.: Interactive rendering with lod control and occlusion culling based on polygon hierarchies. In: Proceedings of the Computer Graphics International (CGI’04). (2004) 536–539 2. Akenine-M¨ oller, T., Haines, E.: Real-Time Rendering. A. K. Peters, Ltd., Natick, MA, USA (2002) 3. Assarsson, U., Moller, T.: Optimized view frustum culling algorithms for bounding boxes. Journal of Graphics Tools 5(1) (2000) 9–22 4. Samet, H., Webber, R.E.: Hierarchical data structures and algorithms for computer graphics. i. fundamentals. IEEE Computer Graphics and Application 8 (1988) 48–68 5. Falby, J.S., Zyda, M.J., Pratt, D.R., Mackey, R.L.: Npsnet: Hierarchical data structures for real-time three-dimensional visual simulation. Computers and Graphics 17(1) (1993) 65–69 6. Pajarola, R.: Large scale terrain visualization using the restricted quadtree triangulation. In: Proceedings of IEEE Visualization’98. (1998) 19–24 7. Gudukbay, U., Yilmaz, T.: Stereoscopic view-dependent visualization of terrain height fields. IEEE Transactions on Visualization and Computer Graphics 8(4) (2002) 330–345 8. Yin, P., Shi, J.: Cluster based real-time rendering system for large terrain dataset. Computer Aided Design and Computer Graphics (2005) 365–370

Indoor Environment Modeling for Interactive VR – Based Robot Security Service Sangwoo Jo1,2, Yong-Moo Kwon1, and Hanseok Ko2 1

39-1 Hawolgok-dong, Sungbuk-ku Imaging Media Research Center Korea Institute of Science and Technology, Seoul, 136-791, Korea 2 5ka-1 Anam-dong, Seongbuk-Gu, Eng. Bldg. Room 419 or 439, Dept. of Electronics and Computer Engineering, Korea University, Seoul, 136-713, Korea [email protected], [email protected], [email protected]

Abstract. This paper presents our simple and easy to use method to obtain a 3D textured model. Our algorithm consists of building a measurement-based 2D metric map which is acquired by laser range-finder, texture acquisition/stitching and texture-mapping to corresponding 3D model. The algorithm is applied to 2 cases which are corridor and space that has the four walls like room of building. The proposed algorithm can be applied to 2D/3D model-based remote surveillance system through WWW. The application terminals of this environment model for interactive VR that we consider are PC, PDA and Mobile phone. Especially, this paper introduces a case of service on WIPIbased mobile phone. Keywords: Indoor environment modeling, Surveillance robot, 2D and 3D model, Human robot interface.

1 Introduction In large scale environments like airport, museum, large warehouse and department store, autonomous mobile robots will play an important role in security and surveillance tasks. Robotic security guards will give the surveyed information of large scale environments and communicate with human operator with that kind of data such as if there is an object or not and a window is open. In view of visualization of information and human machine interface, a 3D model can give much more useful information than the typical 2D maps used in many robotic applications. It should be also noted whether windows and doors are opened or closed can be rendered with 3D model, while 2D map specifies only the location of window or door. VR (Virtual Reality) technology can provide several advantages in view of intuitive understanding, real-time interactivity and virtual experience based on simulation model. This paper aims to achieve the technology fusion between VR and robot technologies. The research goal includes intuitive interaction between user and robot based on virtual robotic space that corresponds to physical space and robot. Several research results have been reported for 2D and 3D modeling of indoor environment for robot application. 3D modeling techniques are typically divided into two groups. The first one is composed by active techniques based on rage sensors Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1253 – 1262, 2006. © Springer-Verlag Berlin Heidelberg 2006

1254

S. Jo, Y.-M. Kwon, and H. Ko

measuring directly the distance between the sensor and points in the real world. Researches in the group are generally implemented with 2D or 3D laser scanner for geometry and cameras for texture [1-7]. Techniques in the second group are passive and get the 3D information from several 2D digital photographs or video sequences (such as in stereo or photogrammetry) [8-10]. This approach use only camera device and get 3D information using image processing. Each one of these techniques has advantages and limitations. However, in view of accuracy of model, we use both the range-sensor and camera and simplify the modeling method in terms of interactive virtual reality. In this paper, we present our simple and easy to use method to build both of 2D map and 3D textured model. Here, we build 2D map using 2D laser scanner while acquiring texture images at the corresponding position. Our researches on the virtual indoor environment modeling can be used for the application of remote surveillance, game contents, museum indoor modeling and so on. The developed indoor 2D/3D map is based on standard format, SVG (Scalable Vector Graphics) format for 2D map and VRML(Virtual Reality Modeling Language) format for 3D map. For interactive VR application, we will present mobile phone for remote access. Our 2D map can be accessed with WIPI [13] (Wireless Internet Platform for Interoperability) based mobile phone which is Korean Standard Wireless Internet Platform for mobile device.

2 System Overview This section gives an overview of our method to build a 3D model of an indoor environment. Fig. 1 gives an overview of the indoor map modeling method. In Fig. 1, M, A, and SA stand for manual, automatic and semi-automatic respectively.

Fig. 1. An overview of our method to build 2D/3D model of an indoor environment

In 2D map & 3D model, the geometry data is acquired with 2D laser scanner manually. Based on these geometry data, TMM (Table Metric Map), GMM (Graphic Metric Map) and 3D EM (Environment Map) is generated automatically. The TMM,

Indoor Environment Modeling for Interactive VR – Based Robot Security Service

1255

Fig. 2. Data flow of building 2D and 3D model for indoor environment

GMM procedure will be explained in detail in Section 4. Fig. 2 shows the presented method for building 2D and 3D model for virtual indoor environment. Fig. 3 shows our indoor modeling system using LMS200 laser scanner and two IEEE-1394 cameras. LMS200 laser scanner has 180° scan range and 1.5cm accuracy in 8m apart. Two cameras are set up like Fig. 3(right) in order to get wide view of texture. In our research, we use two approaches for localization of device. One is using an automatic IR landmark-based localization device, named as starLITE [12] which is developed in ETRI, the other is using dimension of floor square tile (DFST) manually. The DFST approach is applied when starLITE is not installed. The DFST method can be used easily in the environment that have reference dimension without the additional cost for the localization device. In our indoor environment, there are a 0.6m square size tiles on the floor. We can use these tiles for localizing the kart position when automatic localization is not available.

Fig. 3. Laser scanner and IEEE-1394 dragonfly camera pair are set up in moving kart

3 Calibration Camera calibration process consists of two steps. First calibration is for getting radial distortion parameter and the last one is for obtaining external parameter between two

1256


cameras. Since we use 2mm lens for obtaining textures, we need to remove lens distortion first. To get radial distortion factor, we calibrate camera by using typical camera calibration method [11]. After we get the radial distortion parameter and get rid of it, we acquire image that has square pattern which is used for obtaining camera information. Now we would like to know camera external parameter between two cameras. Here we use homography method. Two square pattern images are taken from two camera respect and from 4 correlation points give us the relation between two cameras. In Fig. 4, one camera (left) is set up as orthogonal to pattern plane and from the homography matrix we can warp the image of other camera to left one. After right image is warped to right image, the overlapped region is blended in weighted averaging with two pixel values in following basic equation (1). The notation R and L mean that corresponding Right and Left camera pixel each other. Alpha and beta values are determined by the horizontal distance between correspondence pixel point and the edge of left or right image. Calibration pattern and camera distance is 70cm so that if the sharpness of object image which is located far or near than 70cm may be not as good as object near the 70cm because of incorrect homography matrix value. So when we calibrate the two cameras, we get the different homography matrix by setting unit distance between object and camera as 30cm from 70cm to 190cm.

Color =

α ⋅R+β ⋅L α+β

(1)

Fig. 4. Radial distortion removed image. (left) Two camera images that has pattern (right) Right camera image is warped to left camera by homography matrix.

The external calibration between camera and laser scanner is done by matching at center point in vertical direction. As we can see in Fig. 3, the center points of entire devices are set up as in one vertical line. From this external calibration, we can get the data from LS and camera consistently according to localization data.

4 Indoor Modeling 4.1 2D Modeling A 2D map is the basis of our algorithm because an accurate 2D map could build an accurate 3D model. Typical approach of 2D map building method using laser scanner


1257

is scan matching. To use scan matching method, the system needs to have moving vehicle with odometry equipment like robot. Robot platform with laser scanner equipment can solve the problem of pose and position but it is quite complicate. In our case, we use metric map which is generated by laser scanner with known orientation. We use DFST method or StarLITE system to get orientation of 2D metric map data so that we do not have to worry about pose and position problem of camera or data gathering device. If we use the 2D map data from 2D flat drawing, we just need to process that kind of data to be converted into line-based presentation data which represent the basic wall plane. We use user interface program is shown in Fig. 5(left) and the graphical 2D map is shown in Fig. 5(center). Here, we define line-based 2D map as shown in Fig. 5(left). We use two points’ coordinates for specifying one line and each line is connected to 2D map model. Because each line is specified in two point data and it is shown in table-like display, so we call it as TMM. Fig. 5(center) shows a graphic version of TMM that we made in order to view the line-based data in visualization by the base coordinate, so we call it as GMM. Lastly Fig. 5(right) shows converted 2D metric map which is a SVG format. By converting into this format, we can see our 2D map in any internet explorer browser just with SVG plug-in and interact with robot in remote environment through 2D map.

Fig. 5. 2D metric map. (left) Tabled metric map (TMM) data and user interface (center) Graphical metric map (GMM) (right) 2D Map in SVG format.

The laser scanner is positioned in the same center point of camera in vertical direction. And then we can determine the laser scanner pose by doing that set the laser scanner orthogonal to wall. Now we know the laser position information so that we could know the part of shape in environment from laser scanner. We gather the geometric data twice in 1 location by rotating the laser scanner in 180° to scan the opposite side. We can finally generate total 2D metric map by gathering and merging these geometric data. Since we use a line (wall plane) - based raw data format, we can extend the 2D map data to various standard file format like SVG. A SVG based 2D model is generated from TMM automatically. SVG file can be rendered with internet explorer browser. We also implement SVG browser for cellular phone application with WIPI based mobile phone. Fig. 6 shows the rendering of 2D map model with mobile phone.

1258


Fig. 6. 2D map model in WIPI based phone

4.2 3D Modeling Here is a basic idea of building a 3D model. We assume that wall plane is orthogonal to floor plane. This assumption is quite very adequate for indoor environment because in most case of indoor environment a wall is basically orthogonal to a floor in most case of building. The geometry of the 3D model consists of two parts: the floor and the walls. The floor is modeled by a single plane and the walls are the main model of the 3D model. Since we use the 2D metric map data, we just need to show the wall and floor plane in 3D model. More accurate 2D map data enables to build a 3D model so that accurate 2D map measuring is very important process in our method. This generation which is from 2D map data to 3D model is a full of automatic step. This method gives us various advantages. First of all, it is very simple so that we don’t need to setup expensive equipment like moving platform. And if it is possible, we can make 2D and 3D map model using a 2D flat drawing which is easily available in these days. Secondly, we do not consider if doors might open or not and if there is an object in front of walls. Since we focus on the presentation of the wall in our modeling, we basically do not want the temporarily changing objects and linear features which occlude the walls. By omitting the front object of walls, a 2D map and 3D model depends only on measuring exactitude so our method has good advantage in terms of simplification.

Fig. 7. 3D Map Model and textured 3D Model


1259

Our 3D model is generated in VRML format as shown in Fig. 7(left) and the final 3D model combined with processed texture is shown in Fig. 7(right). Since we use a raw data which is line-based table metric map data from 2D metric map, we can render the 3D model in other 3D renderer such as OpenGL which means easy data conversion is available.

5 Texture Generation In this paper, the texture generation is focused on the walls. First, the images which are gathered from walls are stitched in one image automatically. And then, we process the image by cutting to represent different depth part. Finally, we acquire textures for the 3D model to match especially for walls. The camera pose and position is known when we gather textures of walls. Both floor and walls are given by known planes in 3D: the floor is simply the ground plane, and a walls’ plane is given by assigning the respective wall of the 2D map. Instead of using moving platform like robot, we use the known point of texture acquisition device and pose of device that is very still when it moves to other point. There are 2 types shape in indoor environment which is the corridor and room-like shape. The former looks like just as corridor. It has long sequence walls and we have to gather part of it at one time and merge them to get the whole texture the wall. The latter is enclosed area with wall like room. In this case we gather the images in the 4 directions of room orthogonal to walls. In these two gathering method, we can generate whole texture of indoor environment easily. To get wide area of texture in one time, we use small value of focal length. As we use small value of it, we have to remove radial distortion of images firstly. There is a demerit of using small value of focal length lens that image resolution is lowered due to distortion removed image. We omit the image warping process because we know the camera pose is orthogonal to wall plane. The stitching process is processed semiautomatically. We basically find the correspondence points between two input images and we stitch two sequence images just like in calibrating two cameras by these corresponding points. So after we take images for both sides of wall, we can get stitched two big images for both walls respectively. Finally we cut stitched images for fitting to the 3D model for walls. If the wall plane of indoor environment is one plane and there is no disparity, we do not need to make a 3D model in different depth or cut stitched images for texture generation. However, there are several doors and other different depth object in our wall plane and we already build a 2D map considering the depth. So, we process images taking the different depth into account to produce individual texture for the 3D model for walls each other. Fig. 8 shows the processed image in each step. Acquired images are stitched by homography method and applied to basic blending processing which is mentioned in chapter 3. Fig. 8(upper) shows acquired radial distortion free images respectively and Fig. 8(down) shows the stitched image of Fig. 8(upper). There is different depth in stitched image so that we cut stitched image in different depth plane. Also image might have the ceiling or floor plane so we cut those kinds of plane after stitching all images.

1260


Because most people get the idea of environment through wall than floor, we apply the simple way to the floor plane texture generation. We use one pictured tile image for floor texture and duplicate it to floor plane.

Fig. 8. Texture Processing

6 Interactive VR – Based Robot Security Service As we shortly mentioned in chapter 4, we would like to control the remote robot and recognize the circumstance of that environment from robot using our result which is 2D and 3D map. The 2D and 3D map which we see are the VR space and in VR space, we can interact with real object which is robot. In order to that, the user has to have the 2D and 3D map to control the robot and there has to be somewhat connection between user device terminal and robot. Following Fig. 9 shows the network connection for each user device between user and robot. The robot position and viewing angle are continuously updated to web server and user get that information from server and see that in map that user has. When user wants to move the robot to check the status of some point, user designate the desire point using map and that information is passed on to robot trough web server. Finally user can see that robot reaction in the map and get the required status of environment using robot. We take the PC, PDA and Mobile phone as the terminal of our interactive robot service into account. In case of PC, the resource is much more abundant than PDA or mobile phone so that it is comparatively easy to implement. However, in case of PDA or mobile phone, we firstly have to consider the platform in view of computing resource. We develop in Window CE and WIPI platform for PDA and mobile phone respectively and those terminals including PC give interactive robot service to user according to upper network concept. WIPI platform for mobile phone especially support the C and Java language in implementation. This is one of the important reasons why we choose the WIPI platform in case of mobile device.


1261

Fig. 9. Network concept between map and robot

7 Conclusion and Future work This paper presents the 2D and 3D modeling method for interactive virtual reality taking robot security into account and mobile phone application of it. As a real application of the presented algorithm, data set of 83 images and 2D metric map data was recorded at ETRI 7th building 1st floor, covering parts of a region of about 25 × 25 meters. The 2D map data that we generate are 112 wall planes. The 2D and 3D map is generated using those data. The model is generated as SVG and VRML format and it can be viewed in a web browser with a SVG and VRML plug-in. We see our technique as a successfully easy and simple method to generate 3D model. The big advantage of our method is in vivid described model that is texture of model is very realistic. In order to communicate with robot and human through virtual model, it needs to be very accurate and realistic. We are confident to build a 3D model simply and easily in human’s wants that is realistic enough to interact with robot. Moreover, in real application, we can easily see the whole environment, understand the situation and interact with the robot by using user friendly device like Mobile Phone. Regarding our future work, we will develop our algorithm in terms of more automatic processing of 2D map generation and texture matching. And by modeling the object in environment, we will be able to get more intuitive and understandable 2D and 3D model to interact with robot in remote distance. Moreover, interacting information that we use now are just robot position and rotation. The user might have more immersed feeling through additional environment information like sensor status in that environment.

1262


Acknowledgments. This work was supported in part by MIC & IITA through IT Leading R&D Support Project.

References 1. Y. Liu et al, “Using EM to Learn 3D Models of Indoor Environments with Mobile Robots”, 18th Int’l Conf. on Machine Learning, Williams College, June 28-July 1, 2001. 2. D. Hanel, et al., “Learning Compact 3D Models of Indoor and Outdoor Environments with a Mobile Robot”, Elsevier Science, Robotics and Autonomous Systems, Vol. 44, No. 1, pp. 15-27, 31 July, 2003. 3. O. Wulf, et al., “2D Mapping of Cluttered Indoor Environments by Means of 3D Perception”, Proceedings of the 2004 IEEE, Int’l Conf on Robotics & Automation, New Orleans, Apr. 2004. 4. B. Peter, et al., “3D Modeling of Indoor Environments by a Mobile Robot with a Laser Scanner and Panoramic Camera”, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2004), 2004. 5. Kai Pervölz, et al., “Automatic Reconstruction of Colored 3D Models”, Robotik 2004, Munich, Germany, June 2004. 6. D. Paulo, et al., “Registration and fusion of intensity and range data for 3D modeling of real world scenes”, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 7. I. Stamos, P. K. Allen, “3-D Model Construction Using Range and Image Data”, IEEE International Conference on Computer Vision and Pattern Recognition, pages 531536(volumn І), South Carolina, June 2000. 8. D. Aliaga, D. Yanovsky, and I. Carlbom, “Sea of images: A dense sampling approach for rendering large indoor environments”, IEEE Computer Graphics & Applications, Special Issue on 3D Reconstruction and Visualization, pp. 22-30, Nov/Dec 2003. 9. Faugeras O.D., Three-Dimensional Computer Vision: A Geometric Viewpoint, Cambridge, MA: Mit Press, 1993. 10. P. Debevec, “Reconstructing and Augmenting Architecture with Image-Based Modeling, Rendering and Lighting”, Proceedings of the International Symposium on Virtual Architecture (VAA’ 01), pp.1-10, Dublin 21-22 June 2001. 11. Zhang. Flexible Camera Calibration by Viewing a Plane from Unknown Orientations. ICCV99. 12. Heeseoung Chae, Jaeyeong Lee and Wonpil Yu, “A Localization Sensor Suite for Development of Robotic Location Sensing Network,” International Conference on Ubiquitous Robots and Ambient Intelligence (ICURAI 2005), 2005. 13. www.wipi.or.kr

Research on Mass Terrain Data Storage and Scheduling Based on Grid GIS Zhang Jing1 and Huang Hai2 1

College of Resources Environment and Tourism The Capital Normal University, Beijing 100037, P.R. China [email protected] 2 School of Computer Science and Engineering Beihang University, Beijing 100083, P.R. China [email protected]

Abstract. In order to meet the need for 3D real-time dynamic visualization of mass terrain data, the spatial database in Grid GIS is taken as the data source of large-scale terrain scenes. Mass terrain data from distributed heterogeneous GIS spatial database are integrated, and their storage and access methods are designed on the basis of grid concept, ensuring that 3D terrain scene from any spatial information source can serve users on demand anywhere at any time. In addition, according to the requirements for low delay and high throughput of terrain data transmission, both the filtering mechanism of terrain data and multi-path data transport service are proposed so as to achieve the best possible viewing experience under the current network conditions. The experimental results show that the proposed mechanism is effective.

1

Introduction

Large-scale terrain visualization has been widely used in GIS, virtual reality, games and simulation, etc. In particular, the virtual battlefield simulation environment requires for real-time display of large-scale terrain model. In this process, two problems occur. The first problem is the source of 3D terrain mass data. Currently, the highprecision terrain data storage and processing system is mainly GIS (Geographical Information System). A large number of GIS systems have been established with extensive and in-depth GIS application. These GIS systems are mostly designed for special GIS data sources and applications. With the development of network technology and practical demands, people not only access local geographical information, but also access wide area distributed geographical information. The focus of GIS research has transferred from the traditional data structure and algorithm to WebGIS [12]. However, utilizing WebGIS a user can only visit a spatial information system at a time. If he/she wants to access the data in other

This paper is supported by the Beijing Natural Science Foundation of China under the grant No. 4042011.


1264

J. Zhang and H. Huang

spatial databases simultaneously, it is very difficult or even impossible to integrate these data in different databases. Therefore, research & development of Grid GIS based on grid computing technique have become one of the focuses in GIS field [2], [4], [7]. A key problem that Grid GIS should resolve is to collect and share distributed spatial information resources, implement integrated organization and processing for spatial information data, and provide services on demand. The second problem is transport and display of 3D terrain scene data, which involves elaborate models and high-resolution texture data. Even through the wide band network, scene transport also requires for a longer time, and the user has to wait for a long time to observe the scene. Therefore, real time and display fidelity have to be balanced in 3D terrain scene transport, such as adoption of simplified scenes (rough model and low-resolution texture) or reservation of more bandwidth for scene transport. However, the more vivid and more refined scene display requirements of mass terrain elevation data constantly expand the scale of scenes. Simply relying on rapid drawing technology of hardware acceleration cannot meet the requirements for drawing speed in practical applications. Especially for the terrain, the number of triangles increases four times of the increase of terrain size. It is necessary and inevitable to design the rapid drawing algorithm primarily irrelevant to or weakly relevant to the scene scale [1], [3], [6]. In this paper, based on previous research, the spatial databases in Grid GIS are taken as data source of large-scale terrain scenes. 3D terrain mass data from distributed and heterogeneous GIS spatial databases are integrated and organized according to grid concept. In addition, the progressive transmission mechanism, which takes not only transport delay but also throughput into account, is studied. As a result, 3D terrain scene from any spatial information source can be transported at any time after process and can serve the spatial information user with the corresponding authority at any location on demand.

2

Terrain Data Processing and Storage

The sources of terrain data in Grid GIS are distributed geographically with heterogeneous data formats and storage mediums. There are three traditional spatial data storage mediums: spatial data documents, object-oriented database, and spatial database. The middleware able to access different spatial data directly is first established in order to eliminate the differences in grammatical structure of spatial data, i.e., the connector able to access spatial data documents, object-oriented database, and spatial database directly is developed. The heterogeneous spatial data are accessed through spatial data connector, and the diversity of original GIS data formats and access ways is shielded. However, in addition to the grammatical heterogeneity, there is also semantic heterogeneity in spatial data [5]. In order to resolve the problem of semantic heterogeneity, mass data storage and management capacity of Grid GIS are fully utilized in this paper to map terrain data in every source to the uniform data organization formats at the preprocessing stage. The method proposed in [8] is adopted to

Research on Mass Terrain Data Storage and Scheduling Based on Grid GIS

1265

divide elevation data and texture data in terrain data into regular terrain pieces, which will further adopt embedded zerotree coding to obtain progressive compression code stream [10], [13]. Elevation data, texture data, and location name identifiers in terrain data are associated through the coordinates. The progressive compression code streams, location name identifiers, and center coordinates of every terrain piece are stored in terrain database as the final existence. At the same time, the metadata of the database are stored in IS (Index Service). The client can access IS through RB (Resource Broker) to obtain the storage information of the required data, and the data will be transported to and displayed at the client in the form of stream media. In the process of data display, the client can also locate and search for the geographical position through name identifier or the coordinates interactively (see Fig. 1).

Fig. 1. Terrain data storage and scheduling based on Grid GIS

We may suppose D as large-scale terrain mass data which will be stored at n Grid GIS nodes in distributed way. Di is stored at node i, and Di ⊆ D(i = 1, 2, · · · , n) , then the logic view of D stored in Grid GIS is shown as Fig. 2, in which terrain data storage system is mainly composed of RB Service, IS, etc. RB Service provides the user with data access, storage and management functions. It adopts distributed structural design with each LAN (Local Area Network) domain owns a RB server providing data access service independently. When the user makes a request, RB will provide various data operation services for the user. The distributed RBs are able to work together to provide WAN (Wide Area Network) data services. RB is able to analyze user’s request and find the suitable storage resources to provide the user with data operation and management services. After RB obtains the data, it will adopt streaming media transport protocol to transmit the data to the client directly. IS is a hierarchical distributed service structure composed of local IS and central IS, with every local IS responsible for the corresponding local resource & metadata information and central IS responsible for establishing index of every local metadata information & data buffer to provide global metadata view.

1266

J. Zhang and H. Huang Index Service Index Service

Index Service Client

Dj

D1

Di

Dj+1

WAN

RB

LAN

Dn

RB

LAN

Fig. 2. Logic view of terrain data storage and accessing

IS and RB are designed and realized independently with the service relation established through system deployment and configuration. A kind of important metadata stored in IS is the boundary coordinates of Di (i = 1, 2, · · · , n). Suppose when D is stored in a distributed way, Di and Dj are adjacent terrain scenes, then when the terrain scenes displayed at the client change from Di to Dj , RB will access IS to search for the boundary coordinates of Dj to determine the storage node of Dj , read Dj data, and then transport the data to the client so as to ensure the smoothness and continuity of scene display at the client. Notice the boundary coordinates here are those of Di boundary points, and are different from those of central points of terrain pieces stored in the database when Di is divided into terrain pieces at the storage node.

3

Terrain Data Scheduling and Transport Mechanism

Given the current network conditions, in order to achieve the best possible viewing experience, it is necessary that data should be transferred in a timely way; and at the same time, high throughput is preferred. In this paper, we leverage the scheduling method of terrain data to achieve this purpose. The scheduling method is based on two techniques. One is the data filtering method based on viewing volume projection. The original intention of this method was only to transport most ”useful” terrain data to the client and filter the nonessential data so as to reduce the overall transport time and bandwidth occupation. The other is Multi-path Data Transport Service, which will select multiple paths with less transport delay to transport terrain data so as to obtain high throughput. 3.1

Data Filtering Mechanism Based on Viewing Volume Projection

This method is based on the following fact: when the user observes a scene, he/she will not notice every terrain piece in the scene, and cannot see clearly all the visible terrain pieces. Less data are required to be transported for the more distant terrain pieces. Therefore, in this paper, only the visible terrain pieces are selected to be transported and more code stream data are transported


1267

for the nearer terrain pieces so that the scene transport time is reduced. When the user has changed the view point, new terrain pieces can be transported. Terrain piece is used as the unit for visibility determination. And the view points are divided into low view points and high view points to be handled separately. If the view point is low, the viewing volume is projected to x-y plane according to the algorithm in [9], i.e. the upper, lower and near clipping planes are ignored, as shown in Fig. 3(a), among which the triangle represents the projection of the cone on x − y plane, the grey panes represent the visible data pieces, and V represents the projection point of view point on x − y plane. Suppose the side of 1 terrain piece on x−y plane is a; there are nk (k = 0, 1, · · · , x2 −x ) terrain pieces a with the x-coordinate of x1 + ka; if the x-axis distance between the terrain piece and V increases a, the data to be transported from this terrain piece will decline λ times; the bandwidth available for terrain data transport is B; transport of the terrain piece where V is located in requires for the network bandwidth of β. Then: (x2 −x1 )/a nk λk β = B. k=0

Therefore, the network bandwidth required to be allocated for every terrain piece with the x-coordinate of x1 + ka is: λk β = λk B/

(x2 −x1 )/a

nk λk .

(1)

k=0

View Direction

Y axis

Y axis

View Direction

V V V

x1

x2

X axis

(a) Visibility determination of the terrain piece with low view point

X axis

(b) Visibility determination of the terrain piece with high view point

Fig. 3. Visibility determination of the terrain piece with different view point

If the view point is high, the viewing volume is projected to x − y plane. As shown in Fig. 4, after the terrain data are divided into pieces, viewing point V will always be located near the center of the scene. The closer the terrain piece is to V , the more data of the terrain piece are required to be transported through the network so as to obtain clearer visualized results; The farther away the terrain

1268


piece is to V , the less data of the terrain piece are required to be transported. The density of concentric circles of dotted line in Fig. 3(b) represents the contrast of required data transport volumes of terrain pieces in the visible area. Similarly, if the view point is high, the network bandwidth can also be computed for every terrain piece in the visible area similar to the low view point. The description is omitted here due to space limitation. No matter the view point is low or high, only the terrain pieces in the visible area are required to be transported with the motion of view point. However, part of invisible terrain data in the view direction are transported in advance in order to increase the real-time nature of terrain display, as the terrain pieces covered by dotted lines shown in Fig. 3(a) and Fig. 3(b). 3.2

Multi-path Data Transport Service

In order to obtain high throughput, the most direct way is to broaden the bandwidth. However, the grid is established on the basis of the current Internet infrastructure, so it is unfeasible to obtain high throughput simply by broadening the bandwidth. In this paper, we utilize multiple concurrent paths through the Internet to improve the throughput [11]. Where standard Internet routing protocols try to find the single shortest path route from source to destination, we try to discover multiple routes across the Internet. Although the discovered routes may have higher latency than the shortest path, the ensemble of path(s) can offer higher throughput. However, when using paths with length (hop counts) greater than the shortest path, no matter how the flow rates are distributed across the paths, not only the total utilization but also the transfer delay is greater than that used by shortest path routing. In order to trade off between high throughput and low delay, we should restrict length of any selected path within q times length of shortest routing path. The underlying communication network is modeled by a directed graph G = (V, E), which is a structure consisting of a finite set of vertices V = {v1 , v2 , · · · , vn } and a finite set of edges E = {(vi , vj )|vi , vj ∈ V, vi = vj }, where each edge is an ordered pair. V corresponds to the set of routers and E to the set of physical links between them. Without loss of generality, let v1 be the source node, vn destination node. Also, let P (v1 , vi , · · · , vk ) denote a path formed by v1 , vi , · · · , vk connected in sequence. P = {P (v1 , vi , · · · , vk )|i, k ∈ I, 1 < k ≤ n} is a group composed of paths; A is a linear list, the data elements in which are vertexes in graph G; the length of the shortest path routing between v1 and vn is h. Table 1 shows the operations used in path selection. Based on the assumptions and definitions above, the path-selection algorithm is given in Table 2. After the path-selection approach is carried out, the final remaining paths in group P are those connecting the source node v1 and the destination node vn . All these paths are disjointed, so that the bottleneck link is avoided. However, although none of the paths in P is longer than q times the length of shortest routing path, there may be major differences in actual transmission delay of these paths due to the uncertainty of network performance. Therefore, we further select N paths with minor differences in actual


1269

Table 1. Operations used in path selection Operation Length(P(v1, vi, , vk)) Neighbor(vi)

Description Return the link number of path P(v1, vi, , vk). Return the next adjacent node to vi; when Neighbor(vi) is called for the 1st time, return the 1st adjacent node to vi; when Neighbor(vi) is called for the 2nd time, return the 2nd adjacent node to vi, and so forth; if there is no subsequent adjacent node to vi, return NULL. P(v1, vi, , vk)+vm Return P(v1, vi, , vk, vm). Append(A,vi) Append vi to linear list A. Delete(A,vi) Delete vi from linear list A. Get(A) Return a data element in linear list A randomly; if there is no data element in linear list A, return NULL. Get(P,vi) Return a path in P with the last node of vi. Add(P, P(v1, vi, , vk)) Add path P(v1, vi, , vk) to Group P. Remove(P, P(v1, vi, , vk)) Remove path P(v1, vi, , vk) from Group P.

Table 2. Path-selection algorithm

Algorithm: Path selection [BEGIN] P = {P(v1)}; A = (v1); Vertex=Get(A); WHILE (Vertex != NULL) {Temp = Neighbor(Vertex); WHILE (Temp != NULL) {IF (Temp = = vn) {Add(P, Get(P, Vertex)+Temp); Temp = NULL;} ELSE IF (Length(Get(P,Vertex)+Temp) < q*h ) {Add(P, Get(P, Vertex)+Temp); Append(A, Temp);} ELSE Temp = Neighbor(Vertex); } Remove(P, Get(P, Vertex)); Delete(A, Vertex); Vertex = Get(A); } [END]

transmission delay from Group P as the transmission paths for terrain data. More specifically, we calculate weight wi for every path I in Group P according to Formula (2), and select N paths with higher weights:

1270


wi = M in(

M ax(RT Tk ) , F ) × G, RT Ti

(2)

among which k = 1, 2, · · · , n; F = 6; G = 5, and RT Ti represents RT T of path i. Formula (2) shows the higher the weight of the path, the smaller the RT T . Therefore, the transmission delays of N paths selected are minor. 3.3

Scheduling Method of Terrain Data

Suppose the bandwidths of N paths selected according to Formula (2) are N B1 , B2 , · · · , BN , respectively, then B = i=1 Bi . Suppose the size of data packet in network transmission is S, the frame rate is f , then once a frame is drawn, B the number of data packets to be transmitted in the network is f ×S . Combining the interlacing transmission method for code stream data proposed in [8], we can build the running logics for client and server respectively. The differences lie in how to calculate how many bits of terrain pieces should be coded into a data packet; meanwhile, load balance should be realized among the selected paths, so in the running logics packets should be allocated to different paths for transmission according to the bandwidth residual of every path. On the basis of the interlacing transmission method for code stream data, ac(x −x )/a cording to Formula (1), every data packet should contain λk S/ k=02 1 nk λk bits of every terrain piece with the x-coordinate of x1 + ka in Fig. 3. In order B to realize load balance of transmission data volume among the paths, f ×S data packets to be transmitted for every frame should be allocated to the selected N N B i paths reasonably. Therefore, f ×S × Bi / i=1 Bi = fB×S data packets should be transmitted through path i.

4


According to the method above, we have implemented preliminary performance test. The testing environment is 100M Ethernet. There are two grid storage nodes. The host configurations are P4 2.8, 512M memory, and P4 2.4G, 512M memory. The client host configuration is P4 2.4G, 512M memory. The video card is GeForce FX 5200. At the pre-processing stage, terrain data are divided into 16*16 pieces and compressed into code streams to be stored in SQL Server 2000 database of grid nodes. Testing by adjusting the available bandwidth B of data transmission path, we find that when the available bandwidth B=700K, the real-time frame rate may exceed 20fps (see Fig. 4), and the smooth dynamic visualization effect is obtained. Meanwhile, we have implemented simulation test for throughput of multi-path data transport service. The simulation network environment is the complete graph composed of 5 nodes (see Fig. 5(a)). We chose uniform link bandwidth and uniform link delay in this simulation. The bandwidth of every link is 1M. The source node is s, and the destination node is d. Let q = 2, then the changes


1271

Frames per second

60 50 40 30 20 10 0 0

2

4

6

8

10

12

Time (unit: 10s)

Fig. 4. Record of frame rates

s

1

3

d

2

(a) Network topology in simulation

(b) Average throughput using shortest path / multi-path routing

Fig. 5. Experimental results

of throughput with the changes of network load under the shortest path routing and multi-path routing are shown in Fig. 5(b). We can see in Fig. 5(b) that under the same network load conditions, adopting multi-path routing is able to increase the throughput of mass terrain data transmission significantly in comparison with adopting the shortest path routing.

5

Conclusions

Real-time visualization of large-scale terrain scene is of great significance. In this paper, the storage and access methods for mass terrain data are designed on the basis of grid concept by integrating mass terrain data from distributed heterogeneous GIS spatial database. The distributed heterogeneous IS provides the users with global terrain data resource view. RB provides the users with various data operation services. And the users can access terrain data anytime anywhere. Also, the existing terrain data stream media transmission technologies

1272


are leveraged. Then the requirements for delay and throughput of mass terrain data in network transmission are integrated to propose the filtering mechanism to reduce unnecessary data transmission and the multi-path data transmission services to fully utilize the existing network infrastructure to increase throughput of data transmission. This experiment demonstrates the proposed methods can realize real-time online display of large-scale terrain scene on PC.

References 1. Zhuo, Y.F., Zhao, Y.B., Shi J.Y.: A Survey of Real-time Terrain Rendering Algorithms. Computer Simulation, Vol. 22, No. 3 (2005) 4–7 2. Jiang, Y.F., Lv, G.N.: Grid computing and infrastructure and key technology of grid GIS. Science of Surveying and Mapping, Vol. 30, No. 4 (2005) 16–19 3. Bo-Yin Li, Horng-Shyang Liao, Chang, C.H., San-Liang Chu: Visualization for HPC data - large terrain model. Proceedings of the Seventh International Conference on High Performance Computing and Grid in Asia Pacific Region, (2004) 280–284 4. Shen, Z.F., Luo, J.C., Cai, S.H., Zheng, J., Chen, Q.X., Sun, Q.H.: Architecture of Grid-GIS and Its Key Techniques. Geo-Information Science, No. 4 (2003) 57–62 5. Deng, S.J., Wu, C.H., Meng, L.K., Xu, L.: Spatial Data Integration Services for Grid GIS. Computer & Digital Engineering, Vol. 34, No. 2 (2006) 39–43 6. Ng, C.M., Nguyen, C.T., Tran, D.N., Tan, T.S., Yeow, S.W.: Analyzing pre-fetching in large-scale visual simulation. Proceedings of Computer Graphics Intemational 2005, (2005) 100–107 7. Sun, Q.H., Chi, T.H., Wang, X.L., Zhong, D.W.: Design of middleware based grid GIS. Geoscience and Remote Sensing Symposium, (2005) 854–857 8. Ma, Z.G., Zhang, K., Wang, G.P., Dong, S.H.: Streaming 3D Terrain over Network. Acta Scientiarum Naturalium Universitatis Pekinensis, Vol. 42, No. 1 (2006) 116– 120 9. Zhao, Y.B., Shi, J.Y., Zhou, J., Pan, Z.G.: A Fast Algorithm for Large Scale Terrain Walkthrough. Journal of Computer-Aided Design & Computer Graphics, Vo l. 14, No. 7 (2002) 624–628 10. Nagatomo, K., Yoshikai, D., Okuda, M.: Progressive coding of textured 3D models. Multimedia and Expo, No. 1 (2002) 357–360 11. Shi, S., Wang, L.L., Calvert, K.L., Griffioen, J.N.: A multi-path Routing Service for Immersive Environments. 2004 IEEE International Symposium on Cluster Computing and the Grid, (2004) 699–706 12. Rishe, N., Sun, Y., Chekmasov, M., Selivonenko, A., Graham, S.: System architecture for 3D terrafly online GIS. Proceedings of the IEEE Sixth International Symposium on Multimedia Software Engineering, (2004) 273–276 13. Zhang, L.B., Wang, K.: An Image Compression Algorithm Based on Integer Wavelet Transform. Journal of Software, Vol. 14, No. 8 (2003) 1433–1438

Multi-touch Interaction for Table-Top Display Song-Gook Kim, Jang-Woon Kim, Ki-Tae Bae, and Chil-Woo Lee Department of Computer Engineering, Chonnam National University Yong-bong Dong 300, Gwang-Ju, South Korea [email protected], [email protected], [email protected], [email protected] Abstract. In this paper, we describe a system of multi-touch tabletop display and its control algorithm for Google-earthT M that utilizes hand gesture as mouse free interaction. The system adopts well known FTIR technique. The hand gesture of necessary instruction is predefined according to the number, position, and movement of the fingertips which are obtained from multi-touch of a single user. The system consists of a beam-projector and an acrylic screen attaching infrared LED as light source, and an infrared camera for capturing the fingertip image. Input image is preprocessed with simple morphological method to remove the noise and to determine the fingertip region. In recognition process, gesture commands are recognized by comparing the preprocessed image with predefined gesture model of fingertips. In detail, firstly the number of the spot; fingertip mark, angles among the spots, and Euclidean distance between two spots are calculated, and then the values are compared with the information of the model. The effectiveness of the system can be proved through the experimental results. Keywords: Multi-touch, Tabletop display, FTIR (Frustrated Total Internal Reflection), human computer interaction.

1

Introduction

Recently, according as frequent use of information system in daily life, the research on understanding user intention and action awareness is being popular because if the system understands it we can construct the user-friendly interaction easily. There are many methods to establish the purpose, and hand touch is considered as the most intuitive and convenient method for controlling the system. So, constructing multi-touch screen and hand gesture recognition for display are receiving much attention as core technologies for advanced information system in the human interface research field. Therefore, the researches about tabletop display with large screen installed on table are very popular [1]. Generally, in designing a tabletop display, we have to consider five key aspects in order to make an ideal system; (1) multi-touch interaction using bare hands, (2) implementation of collaborative work, (3) direct touch interaction, (4) the use of physical objects as interaction tool, (5) not expensive for the public large display. And with these aspects, tabletop display ultimately integrates

Corresponding author (Tel.: 82-62-530-1803).


1274

S.-G. Kim et al.

interaction of three elements; Computer, physical objects and displayed objects, to human user as the central figure very easily. Additionally, tabletop display can help cooperative interaction of multi-users as medium of communication. However until now most of displays are generally recognize only a single point touch. Multi-touch recognition has many difficulties to be implemented, so rich interaction for the system is limited. The design of multi-touch tabletop user interfaces is still in its infancy and is not yet well understood. In this paper, we will describe a system of multi-touch tabletop display and its control algorithm for Google-earthT M that utilizes hand gesture as most intuitive tool of human computer interaction. Also, we are going to describe the technologies which include multi-touch sensing technique in the previous researches.

Fig. 1. The outline and the goal of tabletop display

2

Related Work

Myron Kreuger can be considered as one of the pioneers in interactive computer based art. His focus was to create unencumbered, artificial realities where the humans could participate with their entire body without wearing any special instruments in an experience created by the computer. His works have affected many things to tabletop display researches. In the section of ”Emerging Technologies”, SIGGRAPH 2006’ we can see various multi-touch technologies which are being developed in the world [1]. From having experience of the technologies we can see that the terminology called ”table” becomes interface of synthetic device that both sensing and output means are integrated. Most of the researches about tabletop display are classified into several types according to system configuration and sensing method. We are going to explain the types as follows. 2.1

Capacitive Sensing Based Interaction

There are a number of systems which embed electronic capacitive sensing device into the surface itself such as SmartSkin [5] and DiamondTouch [2]. These

Multi-touch Interaction for Table-Top Display

1275

systems typically result in very fast and precise detection of touch compared to the following vision based approaches. However, these systems still require very complicated hardware configuration with low resolution. Also the systems usually rely on top-down projection. So occlusion can frequently occur between hands and displayed images and also the distortion of image can occur. As a result, a user frequently tries to see the distorted images on their hands and tries to move it away from screen. Consequently, it disrupts the focus of attention. Especially, SmartSkin [5] analyzes sensor value of two dimension called ”proximity pixels” using the capacitive sensing method. Therefore, the recognition of hand shape and the discrimination of hand proximity are available with the very fast speed. 2.2

Vision Based Interaction

Computer vision based method has capable of sensing capabilities including detection and recognition of objects placed on the screen itself. Previously, the AugmentedDesk [6] uses a thermo-infrared camera mounted above the table to extract the shape of hands from the background. And also template matching method used to detect the 2D positions of all the fingertips of both hands for such tasks as two-handed drawing and GUI navigation. But their single camera setting cannot determine whether a finger is touching the table surface or not. Lumisight Table [4] is based on the optical design of a special screen system composed of a building material called Lumisty and a Fresnel lens. This system combines films and a lens with four projectors to display four different images, one for each user’s view. Thus, a method for a mixed and integrated display of shared and private information on a single screen could become a new paradigm in human-computer interaction. Therefore, the Lumisight Table [4] is physically single device but can be visually used multiple person. Recently, New York University announced the multi-touch sensing display [3] using FTIR (Frustrated Total Internal Reflection) technique to the public. Because such methods use all video cameras, there are many advantages to get image data of high resolution inexpensively and so on. In vision based method, both top-down and rear projection methods have been used but latter one becomes main approaches to many systems. Because the discrimination of fingers touch is difficult and inexact on screen in top-down method. Additionally, rear mounting configuration can gradually expand into implementation of augmented reality using physical objects. 2.3

Display System of Top-Down Projection

One popular approach is to mount a camera and projector high on a shelf or on the ceiling [2, 5, 6]. Such mounting configurations are typically necessary because of the throw requirements of projectors and the typical focal length of video cameras. Such a configuration has the following drawbacks. Ceiling installation of a heavy projector is difficult, dangerous, requires special mounting hardware. Once the installation is complete, the system and projection surface cannot

1276

S.-G. Kim et al.

be moved easily. Often minor vibrations present in buildings can create problems during operation and make it difficult to maintain calibration. The user’s own head and hands can occlude the projected image as they interact with the system. 2.4

Display System of Rear Projection

A second approach is to place the projector and camera behind a diffuse projection screen [3, 4]. While this enables the construction of a self-contained device, allows the placement of codes on the bottom of the objects, and eliminates occlusion problems, this approach also has drawbacks. It is difficult to construct such a table system with large display area which also allows users room enough to put their legs under the table surface. Because the camera is looking through a diffuse surface, the imaging resolution is limited. High resolution capture of documents, for example, is impossible. A dedicated surface is required, and the resulting housing for the projector and camera can be quite large. This presents manufacturing and distributing problems for a real product.

Fig. 2. The principle of FTIR (left). Detailed specification of infrared LED and infrared pass filter (right).

3 3.1

System Overview FTIR (Frustrated Total Internal Reflection) Sensing Technique

Our tabletop display makes use of FTIR (frustrated total internal reflection), which is a phenomenon that accompanies total internal reflection in a waveguide medium. When the fingertips touch the surface of a waveguide medium, the condition of the total internal reflection is disturbed. Therefore, a scattering occurs along the surface where FTIR happens. Consequently, the infrared camera with IR pass filter placed below the screen can capture the scattering as a bright spot on the homogenous surface. Infrared LED used in our system has peak emitting power at 950nm in wavelength. So we have utilized the B+W infrared pass filter of the Schneider 093 version See Fig 2.


3.2

1277

Hardware Configuration

We adopted the multi-touch display concept developed by NewYork University using two rows of infrared LED as light sources on either side of an acrylic screen, which acts as a waveguide. In our prototype, we used a 660mm (wide) × 510mm (high) × 8mm (thick) sheet of acrylic, whose four edges have been polished clearly Fig 3. This acrylic sheet is four edges-lit by infrared LED which is placed directly against the polished four edges so as maximize coupling efficiency into total internal reflection, while the infrared camera with the infrared pass filter is mounted orthogonally to the screen. The screen is covered by a special film and rear-projection composition that acts as a diffuser and prevents the light from escaping. And the gradient of acrylic screen is 30 degrees for an arrangement of equipment. Tabletop display also has the air-cap of only 2mm between screen and special film for preventing light spreads. Fig 3 shows the prototype of our table-top display, which includes a projector, a infrared camera attached the infrared pass filter and infrared LEDs. The infrared camera can capture 320*240 images at 30 frames per second on a Pentium IV 3.0GHz.

Fig. 3. The acrylic screen attaching infrared LED (left). The inside of the prototype system, and the configuration of camera and projector (right).

4

Interaction Techniques

The recognition system of multi-touch interaction was developed by using the Microsoft Visual C++ compiler with the Image Processing Libraries (IPL) and Open Computer Vision library (OpenCV) offered from Intel. These libraries contain special image-processing functions optimized for the Pentium architecture to ensure efficient computational performance. Our table-top display is capable of tracking fingertip movement and counting the number of fingertips that the user is using for interaction. The only one time calibration is required, in the process the view of the camera has to be

1278

S.-G. Kim et al.

Fig. 4. The algorithm flowchart of our tabletop display system

adjusted to the maximum width of X axis (660mm) and the maximum height of Y axis (510mm). The captured image is successively processed through a series of pipelined stages shown in Fig 4. The system uses acquired images to process the image through a series of pipelined stages, successively refining the information about the position of the fingertips by removing noise and segmenting the touch points from the background. Once the position of the hands is known, the system then counts the number of bright spots where the user is touching on the screen. Each stage of this flowchart is discussed in the followings. The Fig 5 shows the segmentation result of touched fingertips. Firstly, the RGB color images obtained from infrared camera are converted to the Gray scale images. And then binary image is obtained by using appropriate threshold value. Continuously, we conduct noise reduction process of smoothing, opening, closing in consecutive order for detecting touch points. In the next process, we also apply the labeling for counting the number of touched fingertips. And then related commands (shown in Table 1) ;Click or Pointing, Move, Left Rotation, Right Rotation, Zoom-In, Zoom-Out, Tilt-Up, Tilt-Down ,and Play Tour, are executed according to the number of bright spots. We are going to describe interaction techniques for controlling Google-earth T M via one hand or both hands. The instructions are divided into four major categories; static gesture, spatial dynamic gesture, temporal dynamic gesture, and (spatial+dynamic) gesture. 4.1

One-Handed Interaction

In case of using only one hand, a user can use the move, zoom-in, zoom-out, leftrotation, right-rotation instructions for controlling of Google-earthT M . When the user is contacting on the screen, she/he is apts to use more than one finger. If the bright spot is one, the click event of left button of mouse happens and acts as role of catch. So, by moving the fingertip to forward, backward, left, right or arbitrary direction the user is able to explore of satellite picture. And, if the


1279

Fig. 5. Segmentation results of touched the fingertips

fingertip is contacted at arbitrary position, the instruction of movement happens in relation to the contacted direction. If the contacted fingertip recedes from the center of the screen, it is possible to explore the satellite picture with the faster speed. If the user selects arbitrary two places on screen with click command, then the system shows the shortest route between the two places. A user can additionally select arbitrary area in the mini-map of the Google-earthT M at the right side top portion for direct movement. When a user touches on screen using a thumb and a forefinger, namely it has two touch points, zoom-in and zoom-out instructions are achieved. If the user does not move in the state that the thumb and the forefinger are touched, any reaction will not occur. But if the distance between two fingers becomes closer, the zoom-out instruction is achieved. On the other hand the distance of two fingers become away, the zoom-in instruction is executed. Both left-rotation and right-rotation instructions are also controlled by using the thumb and the forefinger. At this time, the moving index finger indicates rotation amount and direction while the thumb is fixed as principal axis. That is, the differences of zoom instruction from rotation instruction are caused by variation of angle and Euclidean distance of two bright spots. In detail, angles among the bright spots and Euclidean distance between two bright spots are calculated, and then the values are compared with the information of the model. Also, contacted points are initially stored to buffer and continuously observed to check the variation. Fig 6 shows the instruction set that use one hand motion. 4.2

Two-Handed Interaction

In case of using both hands, other instructions ;zoom-in, zoom-out, tilt-up, tiltdown, and play-tour, can operate the system depend on the number of fingertips that are contacted on the screen. If the distance of two fingertip spots made by one finger of both hands become away from each other, zoom-in instruction is executed, and in the opposite case, zoom-out instruction is achieved. In this case, we can use faster and more dynamic command than one handed instruction. We detected the center of two fingertips and calculated the Euclidean distance between two bright spots in this case. Also the user can execute tilt-up and

1280

S.-G. Kim et al.

Fig. 6. One handed instruction set. (a)Pointing or Click: single point touch. (b)Move: move to arbitrary direction. (c)Zoom-In (d)Zoom-Out (e)Right-Rotation: the thumb is fixed. (f)Left-Rotation: the thumb is fixed.

tilt-down instruction. For the instruction, the user activate the tabletop display with two fingers of left hand which are fixed on the screen, on the contrast to the fingers, one finger of right hand determines the amount of tilt variation by moving the finger position. To performed instruction predefined information are continuously compared with the calculated information from input image. This instruction permits exploring 3D shape of buildings and outdoor scenery with solid model. The last instruction of Google-earthT M is play-tour command for traveling the historic and famous places. A user contacts four fingertips of both hands on screen and wait for few seconds. This instruction is peculiar and special

Fig. 7. Two handed instruction set. (a)Zoom-In (b)Zoom-Out (c)Play-Tour: both hands are fixed for a few seconds. (d)Tilt-Up: fingers of Left hand are fixed. (e)TiltDown: fingers of Left hand are fixed.


1281

Table 1. The control commands for the Google-earthT M Command

Control result of Google-earth

Move Rotate Clockwise Zoom-In Zoom-Out Tilt-Up Tilt-Down Play Tour

Moves the viewer in the direction of the arrow. Rotates the view clockwises. The earth spins counter-clockwise. Zooms the viewer in. Zooms the viewer out. Tilts the viewer toward ”top-down” view. Tilts the viewer toward ”horizon” view. Plays the tour mode for all selected items in the Places pane.

instruction of Google-earthT M . Fig 7 shows instruction set that consists of using both hands.

5

Concluding Remarks and Future Work

In this paper, we describe a unique tabletop display system which adopts FTIR (Frustrated Total Internal Reflection) sensing technique and its control command for well-known Google-earthT M system. In the system we develop very simple instruction set which used variation of the number of finger spots and location. The recognition algorithm is very simple so that the system implemented in real-time by using ordinary personal computer. However the system has some drawbacks which should be improved in the next version. The first one is to develop more sensitive technology for capturing light tap gesture on the screen since usually the users want to use light touch. Also, if we can develop the technique it may be used for recognize the pressure of the touch. The second thing is to construct more reliable technologies which can avoid the interference of surrounding illumination. Since the system uses the infrared light source, it is so sensitive to the brightness of ambient light. This drawback makes the system not to be used in usual indoor environment. The next one is to have algorithm which can identify owners of fingers in multiuser interaction. If we realize the algorithm we can develop many interesting application software.

Acknowledgments This research has been supported by research funding of ”Culture Technology Research Institute”, and ”Center for High-Quality Electric Components and Systems”, Chonnam National University, Korea.

1282

S.-G. Kim et al.

References [1] In the section of ”Emerging technologies” of ACM SIGGRAPH 2006 Full Conference. [2] Dietz, P. and Leigh, D. ”DiamondTouch: A Multi-User Touch Technology.” In Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology (Orlando, Florida, November 11 - 14, 2001). UIST ’01. ACM Press, New York, NY, pp. 219-226. [3] Han J. Y. ”Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection.” In Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology, ACM Press, New York, NY, 2005, 15-118 [4] Matsushita, M., Iida, M., Ohguro, T., Shirai, Y., Kakehi, Y., and Naemura, T. 2004. Lumisight Table: A Face-to-face Collaboration Support System That Optimizes Direction of Projected Information to Each Stakeholder. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (Chicago, Illinois, USA, November 06 - 10, 2004), CSCW ’04. ACM Press, New York, NY, pp. 274283. [5] Rekimoto, J. 2002. SmartSkin: An Infrastructure for Freehand Manipulation on Interactive Surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’02. ACM Press, New York, NY, pp. 113-120. [6] K. Oka, Y. Sato, and H. Koike, ”Real-Time Tracking of Multiple Fingertips and Gesture Recognition for Augmented Desk Interface Systems,” Proc. IEEE Int’l Conf. Automatic Face and Gesture Recognition (FG 2002), IEEE CS Press, 2002, pp. 429-434.

Reliable Omnidirectional Depth Map Generation for Indoor Mobile Robot Navigation Via a Single Perspective Camera Chuanjiang Luo1,2, Feng Zhu1 , and Zelin Shi1 1

Optical-Electronic Information Laboratory, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China {lcj0603, fzhu, zlshi}@sia.cn 2 Graduate School of the Chinese Academy of Sciences, Beijing, China

Abstract. This paper deals with the problem of finding the largest navigable areas around a mobile robot, which is important for navigation and action planning. We propose a method to obtain reliable dense 3D maps using a novel omnidirectional stereo vision system. The vision system is composed of a perspective camera and two hyperbolic mirrors. Once the system has been calibrated and two image points respectively projected by upper and below mirrors are matched, the 3D coordinate of the space point can be acquired by means of triangulation. To achieve the largest reliable dense matching, our method are divided into three steps. First reliable FX-dominant matching; then feature matching and ambiguous removal; finally the remaining points between features are matched using dynamic time warping(DTW) with modified energy functions adapted well to our system. Experiments show that this proposed vision system is feasible as a practical stereo sensor for accurate 3D map generation.

1 Introduction Obtaining panoramic 3D map information for mobile robots is essential for navigation and action planning. Although there are other ways to fulfill this task, such as ultrasonic sensors or laser range finders, stereo vision system, especially catadioptric vision system excels them in its precision and wide angle of view without energy emission. Mobile robot navigation using binocular omnidirectional stereo vision has been reported in [1], [2], [8]. Such two-camera stereo system are costly and complicated compared to single camera stereo system. Omnidirectional stereo based on a double lobed mirror and a single camera was developed in [3], [4]. A double lobed mirror is a coaxial mirror pair, where the centers of both mirrors are collinear with the camera axis, and the mirrors have a profile radially symmetric around this axis. This configuration has the merit to produce two omnidirectional scene in a single image. However, the effective baseline is quite small due to the closeness of the two mirrors. To overcome this drawback, we have developed a large baseline panoramic vision system to be described in Section 2. To build a depth map for mobile robot navigation, the most important and difficult process is omnidirectional stereo matching. Once two image points respectively Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 1283–1292, 2006. c Springer-Verlag Berlin Heidelberg 2006

1284

C. Luo, F. Zhu, and Z. Shi

a

b

Fig. 1. a: The appearance of the stereo vision system. b: The configuration of the system.

projected by upper mirror and below mirror are matched, the 3D coordinate of the corresponding space point can be obtained by triangulation. State of the art algorithms for dense stereo matching can be divided into two categories: Local method: These algorithms calculate some kind of similarity measure over an area [11]. They work well in relatively textured areas in a very fast speed, while they cannot gain correct disparity map in textureless areas and areas with repetitive textures, which is a unavoidable problem in most situations. In [13] a method of finding the largest unambiguous component has been proposed, but the density of the disparity map varies greatly depend on the discriminability of the similarity measure in a given situation. Global method: These method make explicit smoothness assumptions and try to find a global optimized solution of a predefined energy function that take into account both the matching similarities and smoothness assumptions. The energy function is always in the form of E(d) = Edata (d) + λ · E smooth (d) , where λ is a parameter controlling the proportion of smoothness and image data. Most recent algorithms are belong to this category [9], [10]. The biggest problem of global method is that the data term and the smoothness term represent two process competing against each other, resulting in incorrect matches in areas of weak texture and areas where prior model is violated. Although numerous methods exist for stereo matching, they are designed towards ordinary stereo vision purpose. As there are some particularities in the catadioptric images and the high requirement of reliability for navigation, we propose a three step algorithm that combines the advantages of feature matching and dense global matching. The rest of the paper is organized as follows: in Section 2, we briefly describe principle of our catadioptric stereo vision system; a full model of calibrating the system is presented in Section 3; in Section 4, we detail the omnidirectional stereo matching and depth map generation; our experimental results are shown in Section 5 and in Section 6 we conclude the paper.

Reliable Omnidirectional Depth Map Generation for Indoor Mobile Robot Navigation

1285

2 Principle of Our Vision System The system we have developed [5] is based on a common perspective camera coupled with two hyperbolic mirrors, which are separately fixed inside a glass cylinder (Fig.1a). The two hyperbolic mirrors share one focus which coincides with the camera center. A hole in the below mirror permits imaging via the mirror above. As the separation between the two mirrors provides much enlarged baseline, the precision of the system has been improved correspondingly. The coaxial configuration of the camera and the two hyperbolic mirrors makes the epipolar line radially collinear, thus making the system free of the search process for complex epipolar curve in stereo matching (Fig.2). To describe the triangulation for computing 3D coordinates of space points, we define the focal point O as the origin of our reference frame, z-axis parallel to the optical axis pointing above. Then mirrors can be represented as: (z − ci )2 (x2 + y2 ) − =1. a2 b2

(i = a, b)

(1)

Only the incident rays pointing to the focus Fa (0, 0, 2ca), Fb (0, 0, 2cb) will be reflected by the mirrors to pass through the focal point of the camera. The incident ray passing the space point P(x, y, z) reaches the mirrors at points Ma and Mb , being projected onto the image at points Pa (ua , va , − f ) and Pb (ub , vb , − f ) respectively. As Pa and Pb are known, Ma and Mb can be represented by: x Mi yM zM = i = i . ui vi −f

(i = a, b)

(2)

Since point Ma and Mb are on the mirrors, they satisfy the equation of the mirrors. Their coordinates can be solved from equation group (1) and (2). Then the equation of rays Fa P and Fb P are: x p y p z p − 2ci = = . (i = a, b) (3) xi yi zi − 2ci We can finally figure out coordinate of the space point P by solving the equation (3).

3 Calibrating the Camera and Mirrors Camera calibration is a vital process for many computer vision jobs. In using the omnidirectional stereo vision system, its calibration is also important, as in the case of conventional stereo systems. Compared with the present calibrating techniques only concerning the camera Intrinsics [7], our calibrating algorithm presents a full model of the imaging process, which includes the rotation and translation between the camera and the mirror, and an algorithm to determine this relative position from observations of known points in a single image. We divide the calibration into two steps. Firstly we calibrate the camera’s intrinsics without the mirrors in order to reduce computational complexity. Once the camera intrinsics are known, we estimate the pose parameters of the CCD camera with respect to the mirrors by LM algorithm. The function to be optimized is the sum of squared difference between coordinates of targets and the locations calculated from the targets’ image points from the projection model.

1286

C. Luo, F. Zhu, and Z. Shi

a

b

Fig. 2. a: A calibration image used in our experiment. b: Real indoor scene captured for depth map generation. Table 1. Calibration results with real data value

α −0.9539◦

β 0.1366◦

γ 0.1436◦

tx −0.0553mm

ty −0.1993mm

tz 1.8717mm

Fig.2a shows one example of the calibration image in our experiment. The calibration was performed using a set of 81 points equally distributed on a desk with different heights from 0 to 122mm around the vision system. The calibration results with real data are listed in Table 1. The calibration result was verified using a new set of 40 untrained points, the average square error of the set points is 34.24mm without considering the camera-to-mirror transformation while 12.57mm with the calibration result. For more details about the calibration process, see [6].

4 Stereo Matching 4.1 Overview The images acquired by our system(Fig.2b) have some particularities in contrast to normal stereo pairs as follows, which may lead to poor result using traditional stereo matching methods: (1) The upper mirror and below mirror have different focal length that the camera focal length has to compromise with the two, thus causing defocusing effect. As a result, similarity measures take on much less discriminability. (2) Indoor scene has much more weak textured and textureless areas than outdoor scene. (3) The resolution gets lower when moving away from the image center. The result is the farther off the center, the more unreliable the matching result is. To solve problem (1), we propose a three-step method that allows matching distinctive feature points first and breaks down the matching task into smaller and separate subproblems. For (2) we design a specific energy function used in the third step DTW, in which different weights and penalty items are assigned to points of different texture level and matching confidence. Then we throw away the matching result of the most indiscrminable points, replacing it with interpolation. For (3), we regard points farther

Reliable Omnidirectional Depth Map Generation for Indoor Mobile Robot Navigation

a

1287

b

Fig. 3. a: The curve of texture intensity along the epipolar line. To show it clearly, we removed the part the curve is higher than 100 only labeling the detected feature points. b: Detected reliable FX-dominant matching result in the MZNCC matrix. The black region of the matrix is formed since it is out of the disparity search range.

than the most farthest matched feature point off the center as unreliable, leaving them as unknown areas. This is also required by DTW. Epipolar geometry makes the stereo matching easier by reducing the 2D search to a 1D search along the same epipolar line in both images. To handle epipolar property conveniently, we unwrapped the raw image to two panoramic images which corresponding to images via below and upper mirrors respectively(Fig.7 a,b). The matching process is done on every epipolar pair respectively. The red line labeled in the two panoramic images are the same epipolar line for the subsequent illustration of our proposed method, of which the one above has 190 pixels and the one below 275 pixels. 4.2 Similarity Measure and Defined Texture Level The similarity measure we choose here is zero-mean normalized cross correlation (ZNCC), since it is invariant to intensity and contrast between two images. But directly using this measure would result in low discriminability. Chances exist that two templates with great difference in average gray-level or standard deviation which cannot be deemed as matched pair may have high ZNCC value. To avoid this possibility, we modified ZNCC(called MZNCC) by multiplying a window function as follows: (Ia (i, j + d) − μa ) · (Ib (i, j) − μb ) max(σa , σb ) C(p, d) = · w(|μa − μb |) · w( − 1) , (4) σa · σb min(σa , σb ) 1, x 2),

where α and β is a constant, and the t is the frame index. 3.2 Curve Deformation After the initial curve is localized, the curve is deformed until it matches the object boundary using region competition based active contours. Region competition is proposed by Zhu and Yuille as a hybrid approach to image segmentation [6]. Their basic functional is as follows: M ⎫ ⎧μ E[Γ, {α i }] = ∑ ⎨ ∫ ds − log P({I s : s ∈ Ri }| α i ) + λ ⎬ (2) Ri 2 ⎭, i =1 ⎩ where Γ is the boundary in the image, P(·) is a specific distribution for region Ri, αi is its parameters, M is the number of the regions, s is a site of image coordinate system, and μ and λ are two constants. To minimize the energy E, steepest descent can be done with respect to boundary r v Γ. For any point . On the boundary Γ we obtain: r δE [Γ, {α i }] dv =− r (3) dt δv , where the right-hand side is (minus) the functional derivative of the energy E. r v Taking the functional derivative yields the motion equation for point : r r r ⎫ dv ⎧ μ = ∑ ⎨− k k ( vr ) nk ( vr ) + log P( I ( vr ) | α k )nk ( vr ) ⎬ (4) dt k∈Q( vr ) ⎩ 2 ⎭ , r Q r = {k | v lies on Γk } where ( v ) , i.e., the summation is done over those regions Rk for r r r r k n r which v is ron Γk. k (v ) is the curvature of Γk at point v and k (v ) is the unit normal to Γk at point v . Region competition contains many of the desirable properties of region growing and active contours. Indeed we can derive many aspects of these models as special cases of region competition . Active contours can be a special case in which there are two regions (object region Ro and background region Rb) and a common boundary Γ as shown in follows:

Hand Shape Recognition Using a Mean-Shift Embedded Active Contour

r r r dv = − μk o ( vr ) no ( vr ) + (log P( I ( vr ) | α o ) − log P ( I ( vr ) | α b ) )no ( vr ) dt

1337

(5)

The active contour evolution was implemented using the level set technique. We represent curve Γ implicitly by the zero level set of function u : ℜ2 → ℜ, with the region inside Γ corresponding to u > 0. Accordingly, Eq. (5) can be rewritten by the following equation, which is a level set evolution equation [4]: du ( s) = − μk s ∇u + (log P( I s | α o ) − log P( I s | α b ) ) ∇u , dt

(6)

where u xx y 2y − 2u y u x u xy + u yy u x2

k=

(u x2 + u y2 ) 3 / 2

.

4 Hand Shape Recognizer The shape of an extracted hand region is recognized by comparing with templates. Then, to describe hand shapes, we use moment invariants proposed by Hu [7], since the invariants are independent of position, size and orientation. Let f(x,y) denote the binary image which indicates a pixel belong to hand region or not. Then, the two-dimensional (p+q)th order moments for f(x,y) are defined as follow:

…

m pq = ∫

∞

−∞

∫

∞

−∞

x p y q f ( x, y )dxdy

(7)

,

where p,q=0,1,2 . In digital image, Eq.(7) can be described as m pq = ∑ x

∑x

p

y q f ( x, y) .

(8)

y

The central moments are defined as

μ pq = ∑ x

−

∑ ( x − x)

p

−

( y − y ) q f ( x, y )

y

,

(9)

where −

x=

m10 m00

−

and

y=

m01 . m00

(10)

The normalized central moments, denoted ηpq, are defined as η pq = μ pq / μ 00γ

…

,

(11)

where γ=(p+q)/2 + 1 for p+q=2,3, . A set of seven invariants moments can be derived from the second and third moments.

1338

E.Y. Kim φ1 = η 20 + η 02 , φ 2 = (η 20 − η 02 ) 2 + 4η112 φ3 = (η 30 − 3η12 ) 2 + (η 03 + η 21 ) 2 ,

φ4 = (η 30 + η12 ) 2 + (η 03 + η 21 ) 2

φ5 = (3η 30 − 3η12 )(η 30 + η12 )[(η 30 + η12 ) 2 − 3(η 21 + η 03 ) 2 ] + (3η 21 − η 03 )[3(η 03 + η12 ) 2 − (η 21 + η 03 ) 2 ]

φ6 = (η 20 − η 02 )[(η 30 + η12 ) 2 − (η 21 + η 03 ) 2 ] + 4η11 (η 30 + η12 )(η 21 + η 03 )

(12)

φ7 = (3η 21 − η 03 )(η 30 + η12 )[(η 30 + η12 ) 2 − 3(η 21 + η 03 ) 2 ] + (3η12 − η 30 )(η 21 + η 03 )[3(η 30 + η12 ) 2 − (η 21 + η 30 ) 2 ]

This set of moments is invariant to translation, rotation, and scale change [7]. To recognize the hand shape, Euclidean distance between moment invariants of hand in current frame and templates is used.

5 Application to a Robot Control User can control the mobile robot with predefined hand shapes in front of camera without any hand-held device. The distance between user and camera is about a miter. The system was implemented using a PC and a web cam without additional devices such as data gloves and frame grabber boards. In the proposed system, the mobile robot is controlled by right hand shapes. For the purpose of experiment we selected five different hand shapes. These were logically associated with five different commands needed for robot motion. Fig. 3 shows the five predefined hand shapes and corresponding commands.

Fig. 2. System Environment

(a)Stop

(b)Go forward

(c)Shrink

(d)Turn left

Fig. 3. Predefined hand shapes and commands

(e)Turn right


1339

If the system finds the hand whose shape is “stop,” the tracking process is started and the tracked hand is recognized. When a hand shape is recognized in successive three frames, the corresponding command is performed. Command “stop” makes robot stop their action and maintain standing pose. And other commands make robot take corresponding actions.

6 Experimental Results The proposed system was implemented using MS Visual C++ 6.0 and OpenCV beta3.1 to get 320×240 and 24-bit color images captured 3 frames/s from camera mounted mobile robot without an additional frame grabber board. The used robot is KHR-1 which can controlled servo motor, KRS-784. 6.1 Results of Tracking The proposed system was evaluated through testing 529 frames for 4 persons. Each person tried many times to change hand shapes. And to show the effectiveness of the proposed hand tracking method, we compare the results between the proposed method using mean shift and the method only using active contours without mean shift. Fig. 4 shows the tracking results of the hands.

5th frame

6th frame

7th frame

8th frame

9th frame

10th frame

Fig. 4. Results of the hand tracking

The quantitative comparison in term of convergence speed is shown in Fig. 5. In the proposed method, because initial curve is re-localized neared the hand using mean shift, iteration of curve evolution is fast converged. Due to it, the method only using active contour model takes lager time to track the object than the proposed method as shown in Fig. 5.

1340

E.Y. Kim

ʹΠΟΧΖΣΘΖΟΔΖ͑ΥΚΞΖ͙͑ΞΤ͚

ͥ͡͡ ͤͦ͡

Ψ ΚΥΙΠΦΥ͑ΞΖΒΟ͑ΤΙΚΗΥ

ͤ͡͡

Ψ ΚΥΙ͑ΞΖΒΟ͑ΤΙΚΗΥ

ͣͦ͡ ͣ͡͡ ͦ͢͡ ͢͡͡ ͦ͡ ͡ ͢

ͧ

͢͢

ͧ͢

ͣ͢

ͣͧ

ͤ͢

ͤͧ

ͥ͢

͔͑ΠΗ͑ͷΣΒΞΖ

Fig. 5. Comparison of two methods in term of convergence time

6.2 Results of Static Shape Recognition Table 1 shows the hand shape recognition results using moment invariants. Overall, the approach classified 96.8% of the examples correctly and erred in 17 of the 529 testing frames. Table 1. Hand shape recognition results

Shape recognized Shape given Stop Go forward Shrink Turn left Turn right

529 107 103 112 106 101

Stop

Go forward

Shrink

101 2 2

2 101

4 109 1

2

Turn left

Turn right

1 103 1

2 98

7 Conclusions This paper presents a vision based walking robot control system using hand shape recognition. The proposed system proposed of four modules: a hand detector, a hand tracker, a hand shape recognizer and a robot controller. The hand detector detects a skin color region, which has a specific shape, as hand in an image. Then, the hand tracking is performed using an active contour model with mean shift. Thereafter the hand shape recognition is performed using Hue moments. To show the effectiveness of the proposed system, we tested the proposed system to a walking robot, RCB-1. Acknowledgments. This paper was supported by Konkuk University in 2004.


1341

References 1. William T. Freeman, and Craig D. Weissman.: Television control by hand gestures. IEEE International. Workshop. on Automatic Face and Gesture Recognition. (1995) 179-183 2. Hyeon-Kyu Lee and Jin H. Kim.: An HMM-Based Threshold Model Approach for Gesture Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. Vol. 21, No. 10 (1999) 961-973 3. Min C. Shin, Lenonid V. Tsap, and Dmitry B. Goldgof.: Gesture Recognition using Bezier curves for visualization navigation from registered 3-D data. Pattern Recognition. Vol. 37, No. 5 (2004) 1011-1024 4. Freedman, D., and Zhang, T.: Active Contours for Tracking Distributions. IEEE Transactions on Image Processing. Vol. 13, No. 4 (2004) 518-526 5. Kim, K. I., Jung, K., and Kim, J. H.:Texture-Based Approach for Text Detection in Image Using Support Vector Machines and Continuously Adaptive Mean Shift Algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence. Vol. 25, No. 12 (2003) 16311639 6. Zhu, S. C., and Yuille, A.: Region Competition: Unifying Snakes, Region Growing, and Bayes/MDL for Multiband Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. Vol. 18, No 9 (1996) 884-900 7. Gonzalez, R. C., and Woods R. R., Digital Image Processing, Prentice Hall, New Jersey, 2002

Author Index

Ahn, Honam 980, 1098 Ahn, Jaewoo 58 Ahn, Jung-Ho 575, 584 Andel, Miroslav 1008 Arif Setiawan, Nurul 1, 732 Bae, Doo-Hwan 989 Bae, Ki-Tae 1273 Bai, Jianjun 1136 Bannai, Yuichi 143, 1322 Bao, Weidong 354 Barnaghi, Payam M. 901 Ben Amor, Heni 65 Billinghurst, Mark 85, 185, 272 Brandl, Peter 185 Bu, Jiajun 960 Byun, Hyeran 575, 584 Chae, Songah 372 Chen, Cheng 594 Chen, Chongcheng 724 Chen, Chun 960 Chen, Ge 1156 Chen, Guojun 695 Chen, Hua-hong 764 Chen, Ji-hua 437 Chen, Jim X. 918 Chen, Jun 1136 Chen, Qingzhang 1018 Chen, Wang 354 Chen, Yen-wei 802 Chen, Yimin 155 Chen, Yong 1156 Chen, Y.P. 175 Chen, Zhihua 38 Cheon, Jaeyoung 392 Cho, Dong-uk 1078 Choi, Byoungjo 113 Choi, Cheolmin 575, 584 Choi, Jin Ho 1055 Choi, Seungmoon 392 Chou, Wusheng 486 Choy, Yoon-Chul 512 Cui, Jian 918

Dai, Wenjun 928 Deng, Tao 416 Deng, Yu 77 Dinh, Toan Nguyen 548 Dong, Jinxing 819 Du, Zhiqiang 880 Duenser, Andreas 272 Duh, Henry Been-Lirn 1166 Duthen, Yves 11 Ehnes, Jochen

262

Fan, Chunyu 540 Fan, Jinglong 105 Fei, Guangzheng 633 Feng, Yan 155 Frimor, Troels 998 Furuno, Koki 122 Gao, Chengying 557 Gao, Enke 540 Gao, Jinzhu 314 Gao, Tao 861 Gao, Yan 38 Gong, Jianhua 344, 705 Green, Scott 272 Gu, Hui 785 Gu, Lixu 666 Gu, Qinglong 660 Gu, Xiaokai 416 Ha, Hojin 604 Ha, Taejin 207, 384 Haller, Michael 185, 948 Han, Jianghong 1018 Han, Ray P.S. 425 Han, Woo-Jin 604 Han, Xuesong 651 Hao, Aimin 889 Hartley, Michael Ian 901 Hashimoto, Takashi 122 He, Yuanjun 871 Henrysson, Anders 1008 Heumer, Guido 65 Hirose, Michitaka 262

1344

Author Index

Hong, Seok-Ju 1, 732 Hong, Tao 880 Hu, Jia 494 Hu, Jing 660 Hua, Shungang 1146 Huang, De-cai 494 Huang, Hai 1263 Huang, Mingxiang 705 Huang, Tinan 105 Hwang, Jane 1204 Ignatov, Artem 292 Irawati, Sylvia 272 Ishizawa, Masayuki 1322 Jang, Kyung Ho 1193 Jang, Yun Seok 1055 Jeon, Dongwoon 372 Jeong, Seung-Dae 1193 Jiang, Xiaohong 361 Jin, Shijiu 614 Jin, TaeSeok 530 Jin, Xianlong 1128 Jin, Xiaogang 715 Jin, Yicheng 742 Jing, Hao 502 Jo, Sangwoo 1253 Jun, Kyungkoo 113 Jung, Bernhard 65 Jung, Changduk 1045 Jung, Gi Sook 1193 Jung, Jaehoon 1204 Jung, Soon Ki 1193 Kambara, Hiroyuki 1233 Kameda, Yoshinari 302 Kanbara, Masayuki 217, 228 Kang, Bo 284 Katayama, Atsushi 1233 Ke, Changzhi 284 Kim, Bong-hyun 1078 Kim, Chungkyue 239 Kim, Doo-Hyun 372 Kim, Eun Yi 1333 Kim, Gerard Jounghyun 1204 Kim, Gye Young 980 Kim, HyungSeok 447, 1045 Kim, Jang-Woon 732, 1273 Kim, Jee-In 324 Kim, Jeong Seob 643

Kim, Junhwan 989 Kim, Ki Chang 643 Kim, Kilcheon 575 Kim, Kyoungsik 1233 Kim, Lila 372 Kim, Manbae 292 Kim, Sehun 384 Kim, Song-Gook 1, 1273 Kim, TaeYong 197 Kim, Wonil 1045 Kim, Yeongmi 207, 384 Kimura, Asako 122 Kimura, Yousuke 1212 Kitahara, Itaru 302 Kiyokawa, Kiyoshi 1212 Klinker, Gudrun 998 Ko, Hanseok 1253 Ko, Heedong 272 Koike, Yasuharu 1233 Kourogi, Masakatsu 1310 Krokos, Meleagros 105 Kurata, Takeshi 1310 Kurian, Jayan C. 901 Kurumi, Yoshimasa 802 Kwak, Sooyeong 575 Kwon, Dong-Soo 406 Kwon, Hwasun 372 Kwon, Koo-Joo 1062 Kwon, Yong-Moo 1253 Kye, Heewon 624 Kyung, Ki-Uk 406 Lassabe, Nicolas 11 Lau, Rynson W.H. 753 Lee, Bae-Keun 604 Lee, Bum-Jong 1243 Lee, Chil-Woo 1, 732, 1273 Lee, Dae Sung 643 Lee, Dongwook 1293 Lee, Guee Sang 548 Lee, Hanku 1045 Lee, JangMyung 530 Lee, Jin-Sang 512 Lee, Jun 324 Lee, Minsoo 1223 Lee, Se-hwan 1078 Lee, Seungwon 584 Leithinger, Daniel 185 Leitner, Jakob 185

Author Index Li, Deren 880 Li, FengXia 686 Li, Frederick 753 Li, Guangxin 908 Li, Hua 77 Li, Lan 437 Li, Qicheng 811 Li, Wenhui 676 Li, Yi 467, 522, 1035 Li, Zhenbo 77 Li, Zhide 889 Li, Zhilin 1136 Liang, Ronghua 105 Liang, Weiming 666 Liang, Ying 1118 Lim, Soon-Bum 512 Lim, Yo-An 1175 Ling, Ruo-tian 764 Liu, Da 457 Liu, Feng 796 Liu, Guixi 540 Liu, Ning 557 Liu, Ruian 614 Liu, Wei 871 Liu, Zhen 48 Lou, Jilin 785 Lu, Difei 474 Lu, Jun 28 Lu, Yaya 785 Luga, Herve 11 Luo, Chuanjiang 1283 Luo, Lufeng 28 Luo, Shiwu 1088 Luo, Xiao-nan 522, 764, 849 Luo, Xiwen 28 Luo, Zhiqiang 1166 Ma, Jian-ping 849 Ma, Lizhuang 38 Ma, Shi-jia 849 Ma, Z.Y. 175 Machado, Liliane dos Santos Machida, Takashi 1212 Makita, Koji 217 Mao, Keji 1018 Mariette, Nick 132 Mashita, Tomohiro 1212 Miichi, Yasuo 1026 Min, Kyungha 58

Miyamoto, Toru 302 Moraes, Ronei Marcos de 97 Morikawa, Shigehiro 802 Nakazato, Yusuke 228 Nakazawa, Atsushi 1212 Oakley, Ian 384 Oh, Kyoungsu 624 Ohta, Yuichi 302 Okada, Kenichi 143, 1322 Okuma, Takashi 1310 Ollila, Mark 1008 Pan, Zhigeng 742, 938 Pang, Chen 960 Pang, Mingyong 928 Park, Jaehyun 643 Park, Jangwoo 1175 Park, Jinah 1293 Park, Jong-Seung 197, 239, 1243 Park, Mi Seon 548 Park, Sungjun 324 Park, Wei-jin 989 Peng, Guojun 742 Peng, Huiling 21 Peng, Wen 819 Peng, Yanjun 1070 Petrovski, Alexander 1008 Prihatmanto, Ary Setijadi 948 Pustka, Daniel 998 Qi, Hua 344 Qian, Feng 666 Qian, Guiping 819 Qin, Aihong 48 Qin, Xujia 839 Quan, Xiao-Lin 1118

97

Ren, Lifeng 938 Ren, Peng 284 Ren, Tongwei 1108 Ryu, Jeha 207, 384, 1175 Saito, Hideo 163 Sakata, Nobuchika 1310 Sato, Makoto 1233 Schmalstieg, Dieter 85 Schwerdtfeger, Bj¨ orn 998 Seifried, Thomas 185

1345

1346

Author Index

Sharlin, Ehud 249 Shen, Jianbing 715 Shen, Jie 1128 Shen, Rongrong 775 Shen, Yong-zeng 796 Shi, Jian 1186 Shi, Jiaoying 48, 361, 1070 Shi, Minyong 633 Shi, Zelin 1283 Shi, Zhou 705 Shibata, Fumihisa 122 Shigeno, Hiroshi 143, 1322 Shin, Byeong-Seok 980, 1062, 1098 Shin, Duk 1233 Shin, Hyoseop 1223 Sim, Dong-Gyu 565 Son, Hwa Jeong 548 Son, Seokhyun 1223 Song, Mingli 960 Song, Tao 344 Song, Teuk-Seob 512 Sun, Hongjie 416 Sun, Jingjing 724 Sun, Shouqian 1088 Sun, Zhengxing 676 Sung, Mee Young 113, 1243 Suzuki, Yuji 143 Tam, Gary 753 Tamaki, Hidekazu 143 Tamekura, Haruo 1212 Tamura, Hideyuki 122 Tan, Jiawan 742 Tan, Zheng 1186 Tanaka, Hiromi T. 1026 Tang, Liyu 724 Tang, Songyuan 802 Tang, Xiaohui 811 Tang, Yiping 416 Tang, Yong 830 Tian, Guifen 889 Tong, Ruofeng 819 Tong, Xiaoming 1186 Uematsu, Yuko

163

Wagner, Daniel 85 Wagner, Roland 948 Wang, Dong 960, 1156 Wang, Guoping 811

Wang, Lidan 1146 Wang, Minghui 155 Wang, Ruomei 522 Wang, Tianmiao 457, 486 Wang, Wen-Yong 1118 Wang, Yao 1118 Wang Yongtian 802 Wang, Zonghui 361 Weber, Matthias 65 Wechsler, Harry 918 Wei, Sheng 334 Wen, Peizhi 861 Woo, Woontack 207, 384 Wu, Chengke 861 Wu, Gangshan 928, 1108 Wu, Ke-qin 1156 Wu, Qishi 314 Wu, Qun 1088 Wu, Wei 334 Wu, Xiaojun 861 Wu, Xiaomao 38 Wu, Xiaorong 614 Wu, Zheng-yi 1186 Xi, Haixu 467 Xiao, Hui 753 Xiao, Jun 594 Xin, Min 249 Xiong, Hua 48 Xu, Duan-qing 1035 Xu, Qian 1118 Xu, Rui 802 Xu, Yunlong 938 Yang, Kun 811 Yang, Minghao 633 Yang, Xinhua 839 Yao, Linqiang 724 Yao, Minghai 660 Ye, Xiuzi 474 Yim, Changhoon 604 Yin, Xin 1026 Yokoya, Naokazu 217, 228 Yoo, Jae Myeong 548 Yoon, Hyo Sun 548 Yoon, Jong-Hyun 197, 239 Yoon, Jungwon 1175 Yoon, Kyoungro 1045 Yu, Huijie 425 Yu, Yi-min 1035

Author Index Yuan, C.M. 175 Yuan, Yuan 494 Yun, Ruwei 467 Zhan, Haisheng 908 Zhan, Yongsong 633 Zhan, Yongzhao 775 Zhan, ShouYi 686 Zhang, Dandan 1070 Zhang, Fuyan 928 Zhang, Jianming 775 Zhang, Jing 695, 1263 Zhang, Jingsi 666 Zhang, Lihui 705 Zhang, Maojun 354 Zhang, Mingmin 938 Zhang, Min-jie 796 Zhang, Qing-Rong 1118 Zhang, Shaoting 666 Zhang, Weitong 785 Zhang, Weixiang 502 Zhang, Xiaoyun 1128 Zhang, Xin 522 Zhang, Xuefan 830

Zhang, Yan 676 Zhang, Zhaohua 1305 Zhao, Chengling 1305 Zhao, Hanli 715 Zhao, Lei 1035 Zhao, Qinping 889, 969 Zhao, Weidong 1070 Zhao, Xin 686 Zhao, Xuesheng 1136 Zheng, Guifeng 849 Zheng, Hongbo 839 Zheng, Yongguo 1070 Zhong, Shao-Chun 1118 Zhou, Bingfeng 502 Zhou, Feng 811 Zhou, Yanqiong 28 Zhou, Z.D. 175 Zhou, Zhong 334, 969 Zhu, Feng 1283 Zhu, Jun 344 Zhu, Mengxia 314 Zhu, Yixuan 880 Zhuang, Yueting 594 Zou, Xiangjun 28

1347

Advances in Artificial Reality and Tele-Existence: 16th International Conference on Artificial Reality and Telexistence, ICAT 2006, Hangzhou, China, November

Artificial Life and Virtual Reality

Artificial Life and Virtual Reality

MICAI 2006: Advances in Artificial Intelligence: 5th Mexican International Conference on Artificial Intelligence, Apizaco, Mexico, November 13-17,

AI 2006; Advances in Artificial Intelligence

MICAI 2005: Advances in Artificial Intelligence: 4th Mexican International Conference on Artificial Intelligence, Monterrey, Mexico, November 14-18, 2005,

Advances in Artificial Intelligence

Innovations in Applied Artificial Intelligence: 18th International Conference on Industrial and Engineering Applications of Artificial Intelligence

AI 2006: Advances in Artificial Intelligence: 19th Australian Joint Conference on Artificial Intelligence, Hobart, Australia, December 4-8, 2006: Proceedings

New Frontiers in Artificial Intelligence, JSAI 2006 Conference and Workshops

Advances in Artificial Intelligence: 9th Mexican International Conference on Artificial Intelligence, MICAI 2010, Pachuca, Mexico, November 8-13, ... Lecture Notes in Artificial Intelligence)

Advances in Artificial Intelligence SBIA

GWAI-92: Advances in Artificial Intelligence: 16th German Conference on Artificial Intelligence, Bonn, Germany, August 31 - September 3, 1992. Proceedings ... Lecture Notes in Artificial Intelligence)

Reality

Advances in Computer Graphics: 24th Computer Graphics International Conference, CGI 2006, Hangzhou, China, June 26-28, 2006, Proceedings

Advances in Applied Artificial Intelligence

Logic for Programming, Artificial Intelligence, and Reasoning: 13th International Conference, LPAR 2006, Phnom Penh, Cambodia, November 13-17, 2006,

China Threat - Perceptions, Myths and Reality

Tasks and Methods in Applied Artificial Intelligence: 11th International Conference on Industrial and Engineering Applications of Artificial ... International Conference on Industrial an)

Artificial Intelligence and Symbolic Computation: 8th International Conference, AISC 2006, Beijing, China, September 20-22, 2006, Proceedings

Fantasy and Reality in History

MICAI 2000: Advances in Artificial Intelligence: Mexican International Conference on Artificial Intelligence Acapulco, Mexico, April 11-14, 2000

Artificial Neural Networks - ICANN 2006: 16th International Conference, Athens, Greece, September 10-14, 2006, Proceedings, Part I

Advances in Logic, Artificial Intelligence and Robotics: Laptec 2002 (Frontiers in Artificial Intelligence and Applications, 85)

Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms (Frontiers in Artificial Intelligence and Applications)

Information Security Practice and Experience: Second International Conference, ISPEC 2006, Hangzhou, China, April 11-14, 2006, Proceedings

Artificial Intelligence in Education (Frontiers in Artificial Intelligence and Applications)

Reality and Dreams

Value, Reality, and Desire

Meaning, Knowledge, and Reality

New Trends in Applied Artificial Intelligence: 20th International Conference on Industrial and Engineering Applications of Artificial Intelligence and ... Lecture Notes in Artificial Intelligence)

Advances in Artificial Reality and Tele-Existence: 16th International Conference on Artificial Reality and Telexistence, ICAT 2006, Hangzhou, China, November

Artificial Life and Virtual Reality

Artificial Life and Virtual Reality

MICAI 2006: Advances in Artificial Intelligence: 5th Mexican International Conference on Artificial Intelligence, Apizaco, Mexico, November 13-17,

AI 2006; Advances in Artificial Intelligence

MICAI 2005: Advances in Artificial Intelligence: 4th Mexican International Conference on Artificial Intelligence, Monterrey, Mexico, November 14-18, 2005,

Advances in Artificial Intelligence

Innovations in Applied Artificial Intelligence: 18th International Conference on Industrial and Engineering Applications of Artificial Intelligence

AI 2006: Advances in Artificial Intelligence: 19th Australian Joint Conference on Artificial Intelligence, Hobart, Australia, December 4-8, 2006: Proceedings

New Frontiers in Artificial Intelligence, JSAI 2006 Conference and Workshops

Advances in Artificial Intelligence: 9th Mexican International Conference on Artificial Intelligence, MICAI 2010, Pachuca, Mexico, November 8-13, ... Lecture Notes in Artificial Intelligence)

Advances in Artificial Intelligence SBIA

GWAI-92: Advances in Artificial Intelligence: 16th German Conference on Artificial Intelligence, Bonn, Germany, August 31 - September 3, 1992. Proceedings ... Lecture Notes in Artificial Intelligence)

Reality

Advances in Computer Graphics: 24th Computer Graphics International Conference, CGI 2006, Hangzhou, China, June 26-28, 2006, Proceedings

Advances in Applied Artificial Intelligence

Logic for Programming, Artificial Intelligence, and Reasoning: 13th International Conference, LPAR 2006, Phnom Penh, Cambodia, November 13-17, 2006,

China Threat - Perceptions, Myths and Reality

Tasks and Methods in Applied Artificial Intelligence: 11th International Conference on Industrial and Engineering Applications of Artificial ... International Conference on Industrial an)

Artificial Intelligence and Symbolic Computation: 8th International Conference, AISC 2006, Beijing, China, September 20-22, 2006, Proceedings

Fantasy and Reality in History

MICAI 2000: Advances in Artificial Intelligence: Mexican International Conference on Artificial Intelligence Acapulco, Mexico, April 11-14, 2000

Artificial Neural Networks - ICANN 2006: 16th International Conference, Athens, Greece, September 10-14, 2006, Proceedings, Part I

Advances in Logic, Artificial Intelligence and Robotics: Laptec 2002 (Frontiers in Artificial Intelligence and Applications, 85)

Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms (Frontiers in Artificial Intelligence and Applications)

Information Security Practice and Experience: Second International Conference, ISPEC 2006, Hangzhou, China, April 11-14, 2006, Proceedings

Artificial Intelligence in Education (Frontiers in Artificial Intelligence and Applications)

Reality and Dreams

Value, Reality, and Desire

Meaning, Knowledge, and Reality

New Trends in Applied Artificial Intelligence: 20th International Conference on Industrial and Engineering Applications of Artificial Intelligence and ... Lecture Notes in Artificial Intelligence)

Recommend Documents