Ubiquitous Intelligence and Computing: 5th International Conference, UIC 2008, Oslo, Norway, June 23-25, 2008 Proceedings

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris ...

Author: Frode Eika Sandnes | Yan Zhang | Chunming Rong | Laurence Tianruo Yang | Jianhua Ma

4 downloads 1127 Views 28MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

5061

Frode Eika Sandnes Yan Zhang Chunming Rong Laurence T. Yang Jianhua Ma (Eds.)

Ubiquitous Intelligence and Computing 5th International Conference, UIC 2008 Oslo, Norway, June 23-25, 2008 Proceedings

13

Volume Editors Frode Eika Sandnes Oslo University College, Oslo, Norway E-mail: [email protected] Yan Zhang Simula Research Laboratory, Lysaker, Norway E-mail: [email protected] Chunming Rong University of Stavanger, Stavanger, Norway E-mail: [email protected] Laurence T. Yang St. Francis Xavier University, Antigonish, NS, Canada E-mail: [email protected] Jianhua Ma Hosei University, Tokyo 184-8584, Japan E-mail: [email protected]

Library of Congress Control Number: 2008929350 CR Subject Classification (1998): H.4, C.2, D.4.6, H.5, I.2, K.4 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/Web and HCI ISSN ISBN-10 ISBN-13

0302-9743 3-540-69292-4 Springer Berlin Heidelberg New York 978-3-540-69292-8 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springer.com © Springer-Verlag Berlin Heidelberg 2008 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12277919 06/3180 543210

Preface

This volume contains the proceedings of UIC 2008, the 5th International Conference on Ubiquitous Intelligence and Computing: Building Smart Worlds in Real and Cyber Spaces. The conference was held in Oslo, Norway, during June 23–25, 2008. The event was the fifth meeting of this conference series. USW 2005 (First International Workshop on Ubiquitous Smart World), held in March 2005 in Taiwan, was the first event in the series. This event was followed by UISW 2005 (Second International Symposium on Ubiquitous Intelligence and Smart Worlds) held in December 2005 in Japan, by UIC 2006 (Third International Conference on Ubiquitous Intelligence and Computing: Building Smart Worlds in Real and Cyber Spaces) held in September 2006 in Wuhan and Three Gorges, China, and by UIC 2007 held in July 2007 in Hong Kong. Ubiquitous computers, networks and information are paving the road to a smart world in which computational intelligence is distributed throughout the physical environment to provide trustworthy and relevant services to people. This ubiquitous intelligence will change the computing landscape because it will enable new breeds of applications and systems to be developed; the realm of computing possibilities will be significantly extended. By embedding digital intelligence in everyday objects, our workplaces, our homes and even ourselves, many tasks and processes could be simplified, made more efficient, safer and more enjoyable. Ubiquitous computing, or pervasive computing, composes these numerous “smart things/u-things” to create the environments that underpin the smart world. A smart thing can be endowed with different levels of intelligence, and may be context-aware, active, interactive, reactive, proactive, assistive, adaptive, automated, sentient, perceptual, cognitive, autonomic and/or thinking. The field of intelligent/smart things is an emerging research field that covers many disciplines. A series of grand challenges exist to move from the world of ubiquitous computing with universal services of any means/place/time to the smart world of trustworthy services with the right means/place/time. The UIC 2008 conference offered a forum for researchers to exchange ideas and experiences in developing intelligent/smart objects, environments, and systems. This year, the technical program of UIC drew from a very large number of submissions: 102 papers submitted from 26 countries representing four regions – Asia Pacific, Europe, North and South America. Each accepted paper was reviewed (as a full paper) by at least three reviewers, coordinated by the international Program Committee. In order to allocate as many papers as possible and keep the high quality of the conference, we finally decided to accept 27 regular papers for presentation, which reflected a 26% acceptance rate. In addition, there were 28 special session papers included into the proceedings. The accepted papers provide research contributions

VI

Preface

in a wide range of research topics that were grouped into 14 conference tracks: smart objects and embedded systems, smart spaces/environments/services, intelligent computing, wireless sensor networks, context-aware applications and systems, and wireless networks. In addition to the refereed papers, the proceedings include Jadwiga Indulska’s keynote address on “Challenges in the Design and Development of Context-Aware Applications,” Petter Øyan’s keynote address on “The Importance of Including the Haptics Factor in Interaction Design,” and an invited paper from Stephen S. Yau on “Security Policy Integration and Conflict Reconciliation for Collaborations Among Organizations in Ubiquitous Computing Environment.” We believe that the conference not only presented novel and interesting ideas but also stimulated future research in the area of ubiquitous intelligence and computing. Organization of conferences with a large number of submissions requires a lot of hard work and dedication from many people. We would like to take this opportunity to thank the numerous people whose work made this conference possible and ensured its high quality. We wish to thank the authors of submitted papers, as they contributed to the conference technical program. We wish to express our deepest gratitude to the Program (Vice) Chairs for their hard work and commitment to quality when helping with paper selection. We would also like to thank all Program Committee members and external reviewers for their excellent job in the paper review process, the Steering Committee and Advisory Committee for their continuous advice, and Jadwiga Indulska and Daqing Zhang for organizing a panel on the important question: “What do we expect from pervasive/intelligent computing and how far are we from achieving it?” A special thanks to Yo-Ping Huang for organizing a special session on “Object Identification: Techniques and Applications.” We are also in debt to the Publicity Chairs for advertising the conference, to the Local Organizing Committee for managing the registration and other conference organization-related tasks, and to Oslo University College for hosting the conference. We are also grateful to Kyrre Begnum for the hard work on managing the conference website and the conference management system.

June 2008

Frode Eika Sandnes, Yan Zhang, Chunming Rong, Laurence Tianruo Yang and Jianhua Ma UIC 2008 Editors

Organization

Executive Committee General Chairs

Program Chairs

Program Vice Chairs

Honorary Chairs Steering Committee

International Advisory Committee

Frode Eika Sandnes, Oslo University College, Norway Mark Burgess, Oslo University College, Norway Chunming Rong, University of Stavanger, Norway Yan Zhang, Simula Research Laboratory, Norway D. Manivannan, University of Kentucky, USA Michael Beigl, University of Hannover, Germany Yo-Ping Huang, National Taipei University of Technology, Taiwan Oliver Sinnen, University of Auckland, New Zealand Graham Megson, University of Reading, UK Stephen S. Yau, Arizona State University, USA Norio Shiratori, Tohoku University, Japan Jianhua Ma (Chair), Hosei University, Japan Laurence T. Yang (Chair), St. Francis Xavier University, Canada Hai Jin, Huazhong University of Science and Technology, China Jeffrey J.P. Tsai, University of Illinois at Chicago, USA Theo Ungerer, University of Augsburg, Germany Leonard Barolli, Fukuoka Institute of Technology, Japan Victor Callaghan, University of Essex, UK Yookun Cho, Seoul National University, Korea Sumi Helal, University of Florida, USA Frank Hsu, Fordham University, USA Ali R. Hurson, Pennsylvania State University, USA Qun Jin, Waseda University, Japan Beniamino Di Martino, Second University of Naples, Italy Christian Muller-Schloer, University of Hannover, Germany Timothy K. Shih, Tamkang University, Taiwan Ivan Stojmenovic, Ottawa University, Canada Makoto Takizawa, Tokyo Denki University, Japan David Taniar, Monash University, Australia Jhing-Fa Wang, National Cheng Kung University, Taiwan

VIII

Organization

Executive Committee(continued) International Advisory Committee

Publicity Chairs

International Liaison Chairs

Award Chairs

Panel Chairs

Special Session Organizer Publication Chairs Web Chair Local Organizing Chair

Guangyou Xu, Tsinghua University, China Yaoxue Zhang, Tsinghua University, China Albert Zomaya, University of Sydney, Australia Yueh-Min Huang, National Cheng Kung University, Taiwan Jinhua Guo, University of Michigan - Dearborn, USA Mieso Denko, University of Guelph, Canada Hakan Duman, British Telecom, UK Ohbyung Kwon, Kyunghee University, Korea Wen Jing Hsu, National University of Singapore, Singapore Benxiong Huang, Huazhong University of Science and Technology, China Jong Hyuk Park, Kyungnam University, Korea Leonel Sousa, INESC-ID, Portugal Si Qing Zheng, University of Texas at Dallas, USA David Nelson, University of Sunderland, UK Jiannong Cao, Hong Kong Polytechnic University, Hong Kong Demissie Aredo, Oslo University College, Norway Jadwiga Indulska, University of Queensland, Australia Daqing Zhang, National Institute of Telecommunications, France Yo-Ping Huang, National Taipei University of Technology, Taiwan Tony Li Xu, St. Francis Xavier University, Canada Liu Yang, St. Francis Xavier University, Canada Ismail Hassan, Oslo University College, Norway Siri Fagernes, Oslo University College, Norway Simen Hagen, Oslo University College, Norway Kirsten Ribu, Oslo University College, Norway Kyrre Begnum, Oslo University College, Norway Jie Xiang, Simula Research Laboratory, Norway Qin Xin, Simula Research Laboratory, Norway Hai Ngoc Pham, University of Oslo, Norway

Program Committee Waleed Abdulla Sheikh Iqbal Ahamed

The University of Auckland, New Zealand Marquette University, USA

Organization

IX

Program Committee(continued) Alexandra Branzan Albu Najah Abu Ali Noureddine Boudriga Phillip G. Bradford Tsun-Wei Chang Jiming Chen Ruey-Maw Chen Shu-Chen Cheng Wei Chen Wen Chen Peter H.J. Chong Hung-Chi Chu Alva Couch Alma Leora Culn Waltenegus Dargie Reza Dilmaghani Panayotis E. Fouliras Alois Ferscha Edgar Gabriel Antonio Mana Gomez Zonghua Gu Khoon Guan Winston Seah Arobinda Gupta Laurence Habib Hrek Haugerud Dag Haugland Jianhua He Shang-Lin Hsieh Bin Hu Lin-Huang Chang Yu Hua Tao Jiang Wenbin Jiang Tore Møller Jonassen Audun Josang James B.D. Joshi Akimitsu Kanzaki Helen Karatza Andreas J. Kassler

University of Victoria, Canada United Arab Emirates University, UAE University of Carthage, Tunisia University of Alabama, USA De Ling Institute of Technology, Taiwan Zhejiang University, China National Chin-Yi University of Technology, Taiwan Southern Taiwan University, Taiwan Nanjing University of Posts and Telecommunications, China Shanghai Jiaotong University, China Nanyang Technological University, Singapore Chaoyang University of Technology, Taiwan Tufts University, USA University of Oslo, Norway Technical University of Dresden, Germany King’s College London, UK University of Macedonia, Greece Johannes Kepler Universit¨ at Linz, Austria University of Houston, USA University of Malaga, Spain Hong Kong University of Science and Technology, Hong Kong Institute for Infocomm Research, Singapore Indian Institute of Technology, India Oslo University College, Norway Oslo University College, Norway Bergen University, Norway Swansea University, UK Tatung University, Taiwan University of Central England, UK National Taichung University, Taiwan Huazhong University of Science and Technology, China University of Michigan, USA Huazhong University of Science and Technology, China Oslo University College, Norway Queensland University of Technology, Australia University of Pittsburgh, USA Osaka University, Japan Aristotle Univeristy of Thessaloniki, Greece Karlstad University, Sweden

X

Organization

Program Committee(continued) Paris Kitsos Gabriele Kotsis Lars Kristiansen Stefan Lankes Matti Latva-aho Cai Li Frank Li Jie Li Shiguo Lian Daw-Tung Lin Giuseppe Lo Re Arne Lkketangen Sathiamoorthy Manoharan Hassnaa Moustafa Josef Noll Lukowicz Paul

Hellenic Open University, Greece Johannes Kepler University of Linz, Austria University of Oslo, Norway RWTH Aachen University, Germany Oulu University, Finland Communications University of China, China Agder University, Norway University of Tsukuba, Japan France Telecom R&D Beijing, China National Taipei University, Taiwan University of Palermo, Italy Molde University College, Norway

University of Auckland, New Zealand France Telecom R&D , France UniK/Movation, Norway Lehrstuhl f¨ ur Informatik, Universit¨ at Passau, Germany Marius Portmann University of Queensland, Australia Ravi Prakash University of Texas at Dallas, USA Antonio Puliafito University of Messina, Italy Ragnihild Kobro Runde University of Oslo, Norway Hamid Sharif University of Nebraska, USA Albrecht Schmidt University of Munich, Germany Rose Shumba Indiana University of Pennsylvania, USA Isabelle Simplot-Ryl University of Lille, France Lingyang Song University of Oslo, Norway Min Song Old Dominion University, USA Lambert Spaanenburg Lund University, Sweden Zhou Su Waseda University, Japan Evi Syukur Monash University, Australia Chik How Tan Gjvik University College, Norway Hallvard Trtteberg Norwegian University of Science and Technology, Norway Shiao-Li Tsao National Chiao Tung University, Taiwan Thanos Vasilakos University of Western Macedonia, Greece Javier Garcia Villalba Complutense University of Madrid, Spain Agustinus Borgy Waluyo Institute for Infocomm Research (I2R), Singapore Guojun Wang Central South University, China Tzone-I Wang National Cheng-Kung University, Taiwan Xinbing Wang Shanghai Jiaotong University, China Zhijun Wang Hong Kong Polytechnic University, Hong Kong Jiang (Linda)Xie University of North Carolina at Charlotte, USA Qin Xin Simula Research Laboratory, Norway

Organization

Program Committee(continued) Stephen Yang Wei Yen Muhammad Younas Rong Yu Hongwei Zhang Chi Zhou Yuefeng Zhou

National Central University, Taiwan Tatung University, Taiwan Oxford Brookes University, UK South China University of Technology, China Wayne State University, USA Illinois Institute of Technology, USA NEC Laboratories Europe, UK

XI

Table of Contents

Keynote Speech Challenges in the Design and Development of Context-Aware Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jadwiga Indulska

1

The Importance of Including the Haptics Factor in Interaction Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Petter Øyan and Anne Liseth Schøyen

2

Regular Papers Ubiquitous Computing Security Policy Integration and Conflict Reconciliation for Collaborations among Organizations in Ubiquitous Computing Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stephen S. Yau and Zhaoji Chen Improved Weighted Centroid Localization in Smart Ubiquitous Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stephan Schuhmann, Klaus Herrmann, Kurt Rothermel, Jan Blumenthal, and Dirk Timmermann A Formal Framework for Expressing Trust Negotiation in the Ubiquitous Computing Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Deqing Zou, Jong Hyuk Park, Laurence Tianruo Yang, Zhensong Liao, and Tai-hoon Kim Pervasive Services on the Move: Smart Service Diffusion on the OSGi Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Davy Preuveneers and Yolande Berbers

3

20

35

46

Smart Spaces/Environments/Services Robots in Smart Spaces - A Case Study of a u-Object Finder Prototype - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tomomi Kawashima, Jianhua Ma, Bernady O. Apduhan, Runhe Huang, and Qun Jin Biometrics Driven Smart Environments: Abstract Framework and Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Vivek Menon, Bharat Jayaraman, and Venu Govindaraju

61

75

XIV

Table of Contents

A Structured Methodology of Scenario Generation and System Analysis for Ubiquitous Smart Space Development . . . . . . . . . . . . . . . . . . . . . . . . . . . Ohbyung Kwon and Yonnim Lee Capturing Semantics for Information Security and Privacy Assurance . . . Mohammad M.R. Chowdhury, Javier Chamizo, Josef Noll, and Juan Miguel G´ omez

90

105

Context-Aware Services and Applications A Framework for Context-Aware Home-Health Monitoring . . . . . . . . . . . . Alessandra Esposito, Luciano Tarricone, Marco Zappatore, Luca Catarinucci, Riccardo Colella, and Angelo DiBari Semantic Learning Space: An Infrastructure for Context-Aware Ubiquitous Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zhiwen Yu, Xingshe Zhou, and Yuichi Nakamura A Comprehensive Approach for Situation-Awareness Based on Sensing and Reasoning about Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Thomas Springer, Patrick Wustmann, Iris Braun, Waltenegus Dargie, and Michael Berger

119

131

143

Context-Adaptive User Interface in Ubiquitous Home Generated by Bayesian and Action Selection Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . Han-Saem Park, In-Jee Song, and Sung-Bae Cho

158

Use Semantic Decision Tables to Improve Meaning Evolution Support Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yan Tang and Robert Meersman

169

Combining User Profiles and Situation Contexts for Spontaneous Service Provision in Smart Assistive Environments . . . . . . . . . . . . . . . . . . . Weijun Qin, Daqing Zhang, Yuanchun Shi, and Kejun Du

187

Ubiquitous Phone System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shan-Yi Tsai, Chiung-Ying Wang, and Ren-Hung Hwang

201

Utilizing RFIDs for Location Aware Computing . . . . . . . . . . . . . . . . . . . . . Benjamin Becker, Manuel Huber, and Gudrun Klinker

216

Intelligent Computing: Middleware, Models and Services A Component-Based Ambient Agent Model for Assessment of Driving Behaviour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tibor Bosse, Mark Hoogendoorn, Michel C.A. Klein, and Jan Treur

229

Table of Contents

XV

A Cartesian Robot for RFID Signal Distribution Model Verification . . . . Aliasgar Kutiyanawala and Vladimir Kulyukin

244

Self-Localization in a Low Cost Bluetooth Environment . . . . . . . . . . . . . . . Julio Oliveira Filho, Ana Bunoza, J¨ urgen Sommer, and Wolfgang Rosenstiel

258

Penetration Testing of OPC as Part of Process Control Systems . . . . . . . Maria B. Line, Martin Gilje Jaatun, Zi Bin Cheah, A.B.M. Omar Faruk, H˚ avard Husev˚ ag Garnes, and Petter Wedum

271

Intersection Location Service for Vehicular Ad Hoc Networks with Cars in Manhattan Style Movement Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yao-Jen Chang and Shang-Yao Wu

284

Ubiquitous and Robust Text-Independent Speaker Recognition for Home Automation Digital Life . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jhing-Fa Wang, Ta-Wen Kuan, Jia-chang Wang, and Gaung-Hui Gu

297

Energy Efficient In-Network Phase RFID Data Filtering Scheme . . . . . . . Dong-Sub Kim, Ali Kashif, Xue Ming, Jung-Hwan Kim, and Myong-Soon Park Energy-Efficient Tracking of Continuous Objects in Wireless Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jung-Hwan Kim, Kee-Bum Kim, Chauhdary Sajjad Hussain, Min-Woo Cui, and Myong-Soon Park

311

323

Wireless Sensor Networks Data Randomization for Lightweight Secure Data Aggregation in Sensor Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Abedelaziz Mohaisen, Ik Rae Jeong, Dowon Hong, Nam-Su Jho, and DaeHun Nyang Mobile Sink Routing Protocol with Registering in Cluster-Based Wireless Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ying-Hong Wang, Kuo-Feng Huang, Ping-Fang Fu, and Jun-Xuan Wang

338

352

Towards the Implementation of Reliable Data Transmission for 802.15.4-Based Wireless Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . Taeshik Shon and Hyohyun Choi

363

An Energy-Efficient Query Processing Algorithm for Wireless Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jun-Zhao Sun

373

XVI

Table of Contents

Special Session Papers Smart Objects and Embedded Computing Rule Selection for Collaborative Ubiquitous Smart Device Development: Rough Set Based Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kyoung-Yun Kim, Keunho Choi and Ohbyung Kwon

386

An Object-Oriented Framework for Common Abstraction and the Comet-Based Interaction of Physical u-Objects and Digital Services . . . . Kei Nakanishi, Jianhua Ma, Bernady O. Apduhan, and Runhe Huang

397

Personalizing Threshold Values on Behavior Detection with Collaborative Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hiroyuki Yamahara, Fumiko Harada, Hideyuki Takada, and Hiromitsu Shimakawa IP Traceback Using Digital Watermark and Honeypot . . . . . . . . . . . . . . . . Zaiyao Yi, Liuqing Pan, Xinmei Wang, Chen Huang, and Benxiong Huang

411

426

Wireless Networks: Routing, Mobility and Security Multi-priority Multi-path Selection for Video Streaming in Wireless Multimedia Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lin Zhang, Manfred Hauswirth, Lei Shu, Zhangbing Zhou, Vinny Reynolds, and Guangjie Han

439

Energy Constrained Multipath Routing in Wireless Sensor Networks . . . Antoine B. Bagula and Kuzamunu G. Mazandu

453

Controlling Uncertainty in Personal Positioning at Minimal Measurement Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hui Fang, Wen-Jing Hsu, and Larry Rudolph

468

RFID System Security Using Identity-Based Cryptography . . . . . . . . . . . . Yan Liang and Chunming Rong

482

Ubiquitous Computing RFID: An Ideal Technology for Ubiquitous Computing? . . . . . . . . . . . . . . . Ciaran O’Driscoll, Daniel MacCormac, Mark Deegan, Fred Mtenzi, and Brendan O’Shea An Experimental Analysis of Undo in Ubiquitous Computing Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marco Loregian and Marco P. Locatelli

490

505

Table of Contents

XVII

Towards a Collaborative Reputation Based Service Provider Selection in Ubiquitous Computing Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . Malamati Louta

520

Petri Net-Based Episode Detection and Story Generation from Ubiquitous Life Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Young-Seol Lee and Sung-Bae Cho

535

Smart Spaces/Environments/Services Protection Techniques of Secret Information in Non-tamper Proof Devices of Smart Home Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Abedelaziz Mohaisen, YoungJae Maeng, Joenil Kang, DaeHun Nyang, KyungHee Lee, Dowon Hong, and Jong-Wook Han

548

Universal Remote Control for the Smart World . . . . . . . . . . . . . . . . . . . . . . Jukka Riekki, Ivan Sanchez, and Mikko Pyykk¨ onen

563

Mobile Navigation System for the Elderly – Preliminary Experiment and Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Takahiro Kawamura, Keisuke Umezu, and Akihiko Ohsuga

578

Time Stamp Protocol for Smart Environment Services . . . . . . . . . . . . . . . . Deok-Gyu Lee, Jong-Wook Han, Jong Hyuk Park, Sang Soo Yeo, and Young-Sik Jeong

591

Intelligent Computing: Middleware, Models and Services An Analysis of the Manufacturing Messaging Specification Protocol . . . . Jan Tore Sørensen and Martin Gilje Jaatun

602

A Long-Distance Time Domain Sound Localization . . . . . . . . . . . . . . . . . . . Jhing-Fa Wang, Jia-chang Wang, Bo-Wei Chen, and Zheng-Wei Sun

616

Towards Dataintegration from WITSML to ISO 15926 . . . . . . . . . . . . . . . . Kari Anne Haaland Thorsen and Chunming Rong

626

A SIP-Based Session Mobility Management Framework for Ubiquitous Multimedia Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chung-Ming Huang and Chang-Zhou Tsai

636

Context-Aware Services and Applications AwarePen - Classification Probability and Fuzziness in a Context Aware Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Martin Berchtold, Till Riedel, Michael Beigl, and Christian Decker

647

XVIII

Table of Contents

A Model Driven Development Method for Developing Context-Aware Pervasive Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Estefan´ıa Serral, Pedro Valderas, and Vicente Pelechano

662

Intelligent System Architecture for Context-Awareness in Ubiquitous Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jae-Woo Chang and Seung-Tae Hong

677

User-Based Constraint Strategy in Ontology Matching . . . . . . . . . . . . . . . . Feiyu Lin and Kurt Sandkuhl

687

Object Identification: Techniques and Applications RFID-Based Interactive Learning in Science Museums . . . . . . . . . . . . . . . . Yo-Ping Huang, Yueh-Tsun Chang, and Frode Eika Sandnes

697

Real-time Detection of Passing Objects Using Virtual Gate and Motion Vector Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Daw-Tung Lin and Li-Wei Liu

710

A Ubiquitous Interactive Museum Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yo-Ping Huang, Tsun-Wei Chang, and Frode Eika Sandnes

720

Dynamic Probabilistic Packet Marking with Partial Non-Preemption . . . Wei Yen and Jeng-Shian Sung

732

Fractal Model Based Face Recognition for Ubiquitous Environments . . . . Shuenn-Shyang Wang, Su-Wei Lin, and Cheng-Ming Cho

746

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

761

Challenges in the Design and Development of Context-Aware Applications Jadwiga Indulska School of Information Technology and Electrical Engineering, The University of Queensland and NICTA [email protected]

Abstract. Context-awareness plays an important role in pervasive computing as adaptations of applications to context changes (changes in computing environment and in user activities/tasks) help to achieve the goal of computing services available everywhere and at any time. There is a growing body of research on context-aware applications that are adaptable and capable of acting autonomously on behalf of users. However, there are still many open research issues that challenge the pervasive computing community. In this talk I discuss several of these research challenges. First, I outline the state of the art in context information modelling, management and reasoning as well as possible future research directions in this area. This is followed by a discussion of context information management that allows development of fault-tolerant and autonomic context-aware applications. As one of the challenges inhibiting the development of context-aware applications is their complexity, I discuss software engineering approaches that ease the task of developing such applications. Context-aware applications adapt to context changes using context information. However this context information may be imprecise or erroneous and therefore can lead to incorrect adaptation decisions creating usability problems and affecting acceptance of context-aware applications. This creates a need for some balance between autonomy of context-aware applications and the user control of the applications. I describe some early approaches my team is working on to tackle this problem. Finally, I discuss research issues related to privacy of context information and how context can be used to enhance security mechanisms within pervasive computing environments.

NICTA is funded by the Australian Government as represented by the Department of Broadband, Communications and the Digital Economy and the Australian Research Council through the ICT Centre of Excellence program; and the Queensland Government.

F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, p. 1, 2008. c Springer-Verlag Berlin Heidelberg 2008

The Importance of Including the Haptics Factor in Interaction Design Petter Øyan and Anne Liseth Schøyen

Abstract. One aspect of interaction design is communication of information through the use of a screen. Another aspect, the physical or haptic interaction with the device itself, is another important issue, especially to reduce errors when the device is used in critical situations. Ostfold university college has cooperated with the Institute for Energy Technology in Halden and the Norwegian Defence Research Establishment in Kjeller, and several student groups have been given the opportunity to work on research data and in collaboration with researchers from these two institutions within their final year studies. The focus of the projects has been realistic; emphasizing the importance of minimizing operating errors to ensure safe operation in critical situations, where the ability to give correct feedback through haptic interaction is as important as the correct understanding of visual communication. The cases are demonstrating the user centered approach to problem solving used by industrial designers and the analogy between the design- and the resarch process, especially focusing on the use of physical designs to test and review and thereby exploring form as an interaction parameter.

F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, p. 2, 2008. © Springer-Verlag Berlin Heidelberg 2008

Security Policy Integration and Conflict Reconciliation for Collaborations among Organizations in Ubiquitous Computing Environments Stephen S. Yau and Zhaoji Chen Department of Computer Science and Engineering School of Computing and Informatics Arizona State University Tempe, AZ 85287-8809, USA {yau,zhaoji.chen}@asu.edu

Abstracts. The investigative capabilities and productivity of researchers and practitioners of various organizations can be greatly expanded with collaborations among these organizations in ubiquitous computing environments. However, in order to provide effective collaborations among organizations in ubiquitous computing environments, due to the heterogeneity and dynamic nature of the ubiquitous computing environments it is critical to generate an integrated security policy set to govern the interactions among collaborating organizations during the collaboration. Existing approaches in security policy integration and conflict analysis have focused on static policy analysis which cannot address the dynamic formation of collaborative groups in ubiquitous computing environments. In this paper, an approach is presented to security policy integration, including a similarity-based policy adaptation algorithm for changing collaborative groups and a negotiation-based policy generation protocol for the new resources generated by the collaboration as well as for conflict reconciliation. Our approach can be automated with user defined control variables, which can greatly improve the efficiency and make the system scalable with little human intervention. Keywords: Security policy integration, conflict reconciliation, organization collaboration, similarity-based policy adaptation, negotiation-based policy reconciliation, ontology, ubiquitous computing environments.

1 Introduction Due to rapid development of mobile devices and embedded systems, computing resources are embedded in various devices, appliances, such as cell phones, PDAs, televisions, TiVo, refrigerator and cars. All these devices can provide certain information processing capability. With the technologies, such as Bluetooth and Wi-Fi, these devices can easily communicate and share data and/or resources with each other. These advances in ubiquitous computing can greatly improve the investigative capabilities and productivity of researchers and practitioners in many fields, such as health care, e-business, e-government, telecommunication, and homeland security, F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 3–19, 2008. © Springer-Verlag Berlin Heidelberg 2008

4

S.S. Yau and Z. Chen

where the sharing of data, equipment and other resources among collaborating organizations may be extensive. In such an environment, computing resources are distributed throughout the physical environment, and are usually owned and managed by various organizations. An organization can be viewed as a set of nodes which has a set of users, certain resources and is capable of performing certain functions autonomously. To facilitate collaborations among participating organizations, we need to make their resources available across organizations seamlessly. By seamlessly, we mean if a user from one of the participating organizations enters the environment controlled by another organization, the user can access and use the appropriate resources through direct interactions with the resource owners. Otherwise, the user will need to send a request to his/her affiliated organization, and then the affiliated organization will forward the request to the organizations which own the resources. The resource owners would then make the decisions on whether to allow the user to access their resources, and then inform the affiliated organization, which will inform the user of their decisions. It is obvious that seamless resource sharing will greatly improve the effectiveness and efficiency of collaborations among various organizations. However, ubiquitous computing devices in such ubiquitous computing environments are more vulnerable to attacks than other distributed computing systems, especially when malicious users pretend to be legitimate users and try to sneak in the collaboration groups. With the growing concerns on security and privacy, a major obstacle for this kind of collaborations to be widely adopted, especially for critical applications, is how to govern the access to sensitive data and limited resources during the collaborations. Each organization may specify its own set of security policies to govern how its users to access its resources. But, in collaborations among organizations, we need to integrate all the relevant security policies from the participating organizations to govern the interactions throughout the collaborations. The heterogeneity and dynamic nature of ubiquitous computing environments poses certain unique challenges for generating an integrated congruous security policy set. In this paper, we will present an approach to integrating security policies of collaborating organizations in ubiquitous computing environments, including a similarity-based policy adaptation algorithm for security policy integration and a negotiation protocol for conflict reconciliation. An example will be given to illustrate our approach.

2 Challenges and Current State of the Art Collaborations among organizations in ubiquitous computing environments have the following characteristics. First, each organization is a group of nodes which manages its own resources and can function autonomously. Second, a group of collaborating organizations can be formed and changed at runtime. Each organization can choose to join or leave the group based on its needs. Third, organizations are peers to each other. Normally, there is no organization to serve as an authority to coordinate or manage the interactions among various collaborating organizations. Due to these characteristics, we have the following three major challenges for providing security in such environments.

Security Policy Integration and Conflict Reconciliation

5

1) Ambiguous security policy specifications: Each participating organization specifies its security policies describing how its resources should be used, including various limits and access control, and how these policies are enforced at runtime. Because each organization acts autonomously, the organizations can choose different vocabularies to specify their own security policies, which may cause confusion and difficulty when they try to understand each other’s policies. 2) Dynamic set of users: Individual organization’s security policies normally do not address the expanded user sets for collaborations. An organization’s security policies only address its own users and resources. Because there is no authority in the coalition to manage the requests from other organizations, and because the group of collaborating organizations may be changing, security policies specified by participating organizations need to be integrated together to govern the interactions during the collaboration, and the owner of a resource needs to know how to handle the requests from other organizations at runtime without prior knowledge of the collaborators. The collaboration may generate some resources that cannot be provided by any single participating organization. To govern future accesses to these new resources, the integrated security policy set should include policies for all these resources and the policies need to be acceptable by all participating organizations. 3) Conflicting security policies: Participating organizations may have inconsistent security policies among them. Because the group of collaborating organizations may continuously change at runtime, the organizations cannot anticipate all possible collaborative partners a priori and specify their security policies accordingly. The fact that each participating organization has little knowledge about what has or has not been specified by the other organizations may lead to redundancy and conflict when trying to integrating their security policies together. To avoid inconsistency in security policies integration and enforcement, redundant policies should be removed, and conflicts must be resolved. To address the first challenge, several approaches [1, 2, 4] have been developed to provide unambiguous interoperable security policy specifications using XML. These approaches provide syntactic tags for the security policy specifications and the tags show how the specifications should be interpreted. The general syntax of such approaches is rigid in the sense that specifications must adhere to the general rules of XML to ensure that all XML-aware software can at least read and understand the relative arrangement of information within them. This usually results in lengthy specifications as the user needs to define a lot of tags just to comply with XML standards. We have developed an approach based on ontology in [3], where the security policy has a fixed format, but users can derive individual components from an ontology to specify application-specific information needed in the security policy specifications. All these approaches can address Challenge 1) sufficiently and provide unambiguous security policy specifications across collaborating organizations. In this paper, we will present methods to address Challenges 2) and 3). To address Challenge 2), relevant security policies from various collaborating organizations need to be combined together. Bonatti et al. [5] dealt the policy composition problem through a multi-level security model. Bidan et al. [6] introduced a set of

6


combination operators to combine security policies in a controlled manner. Several algebraic operators for combining enterprise privacy policies specified by EPAL were introduced in [7]. SPL [8] has stratified security rules, and there is a special layer of rules, which is used to control policy integration. These approaches are common in that they all require prior knowledge about the other collaborating organizations. However, in ubiquitous computing environments, groups of collaborating organizations are continuously changing with participating organizations joining or leaving the groups/. Each organization has little prior knowledge on the security policies of other collaborating organizations and how they will impact its own security policies. To address Challenge 3), Ribeiro, Zuquete and Ferreira [8] specified special rules to decide which security policy should override the others when there is a conflict. Similarly, Lupu and Sloman [9] specified “policy precedence relationships” to determine which one of two conflicting policies should apply and which should be ignored. Agrawal et al. [10] presented an approach to eliminating conflicts by checking the new policies and prevent them from being added to the integrated policy set if there is a conflict. All these approaches rely on some policy hierarchy structure, where certain policies are more important than the others, and more important policies override less important ones in case of conflict. For [10], the security policies that are already in the integrated set are more important than those that are waiting to join the set. It is difficult to adopt these approaches in ubiquitous computing environments because there is no authority in the group of collaborating organizations to establish such hierarchy and each participating organization is an autonomous entity. Hence, it is not reasonable to choose an organization’s security policies over the others in case of conflict. So far, no comprehensive approach addresses both Challenges 2) and 3) for collaborations among organizations in ubiquitous computing environments, where groups of collaborating organizations are continuously changing with little prior knowledge about when other organizations will join and what kinds of security policies they have specified. This makes policy integration and conflict reconciliation more difficult compared to the case that the groups of collaborating organizations remain the same.

3 Our Overall Approach Our approach aims at generating an integrated, congruous security policy set based on the security policies specified by individual participating organizations. Our overall approach is shown in Fig.1 and consists of the following three major parts. Part1) uses a specification language similar to the one we presented in [3] to generate security policy specifications for each collaborating organization, where a security policy can be specified as a quadtuple with the components derived from the ontology containing the key concepts needed for specifying security policies. For the sake of completeness, we will summarize the important aspects of this specification language needed in our approach in the Appendix.


7

Part2) adapts existing security policies to address the expanded user set for the collaboration. When an access to a resource is requested by a user from other participating organizations, the resource owner should make the decision on whether to grant the request to the resource. However, since the requester is from another organization, the resource owner may not have much knowledge about this requester, which makes it difficult to make a decision on the request. In reality, it is natural to evaluate a

OrgA

OrgB

Natural Language Security Policies

Natural Language Security Policies

Security Policy Specification Language

Security Policy Specification Language

Part1

Security Policy Specifications

Security Policy Specifications

Address expanded user set

Address newly generated resources Integration Process Negotiation-base Policy Generation

Similarity-base Security Policy Adaptation

Part2 Integrated Security Policy Specifications

Part3

Yes Have conflicts?

No

Negotiation -base Conflict Reconciliation

Final Security Policy Set for Collaboration

Fig. 1. Our overall approach for security policy integration and conflict reconciliation

8


stranger’s trustworthiness based on where he comes from and how he is trusted in the organizations he belonged before. Thus, we will use a similarity-base security policy adaptation algorithm to help the resource owner make a decision based on evaluating whether the participating organizations that the user comes from have similar security policies, and whether the user request will be granted under those security policies. The details of this algorithm will be presented in Section 4. Part3) uses a negotiation-based approach to generating a congruous set of security policies which can be accepted by all participating organizations. Participating organizations will first provide their inputs towards the generation of the policies, and then make compromise in order to resolve possible conflicts and reach an agreement. This is used to address possible conflicts which arise from the integration, and to specify new security policies for the resources generated by the collaboration. In both cases, no single organization can claim the sole ownership of the generated resources, and thus no security policies of any participating organization should take priority over those of other organizations. The details of this approach will be presented in Section 5.

4 Similarity-Based Security Policy Integration Algorithm The specification approach presented in the Appendix provides an unambiguous, interoperable security policy specification that is understandable by other participating organizations. To enable collaborations among various collaborating organizations, security policies specified by these organizations need to be integrated to govern the interactions among them during collaboration. It is similar to corporation mergers, where company policies related to labor forces, equipments as well as intellectual properties need to be integrated before the merge can be completed. In such a case, the integration of the policies of different companies is done through negotiation talks between the management and legal departments of the merging companies. However, for dynamic collaborations in ubiquitous computing environments, it is simply not practical if we need to halt the collaboration and get human involvement every time there is an organization join or leave the group of collaborating organizations. In this paper, we present an alternative approach based on policy adaptation and negotiation to achieve dynamic security policy integration with minimum human intervention. During the collaboration, an organization may need to choose to make its resources available to other collaborating organizations so that users from those organizations can access the resources. Even though a collaboration may involve multiple organizations, it is rare that a user belong to two or more of these organizations. In case a user belongs to more than one organization during a collaboration, when that user requests to access a specific resource, we can require the user to specify which organization affiliation he wants to submit this request. This is similar to system administration that a person can be a normal user or an administrator. But, when the user wants to perform system administration tasks, he must sign in as an administrator. So we assume that when a user requests to access a resource, he belongs to only one organization. A general scenario of such pair-wise user-resource access scenario is shown in Figure 2.


9

Fig. 2. Pair-wise user-resource integration

There are two organizations OrgA and OrgB. OrgA has its user set UA= {ui, uj, …} and data set DatA ={datAp, datAq …}. A set of security policies PA has been specified to govern the data access behavior in OrgA. Similarly, OrgB has its user set UB, data set DatB, and security policy set PB. The collaboration will not only make existing data available to all the researchers, it may also generate some new data which appears in neither DatA nor DatB. For example, the collaboration can produce a new data datk through analysis of datAq and datBm. We use DatAB to denote the set of the newly generated data. The integrated security policy set should be able to handle all possible data access requests. Let u be the requester and dat be the requested data. There are three types of requests. Type1 consists of C1: dat ∈ DatA , u ∈ UA and C2: dat ∈ DatB , u ∈ UB, where u and dat are in the same organization. Type2 consists of C3: dat ∈ DatB , u ∈ UA and C4: dat ∈ DatA , u ∈ UB, where u and dat are in different organizations and the original security policies specified by the data owner cannot cover the new users from the other collaborating organization. Type3 is C5: dat ∈ DatAB, u ∈ UA ∪ UB, where dat is newly generated data and both collaborating organizations can claim ownership of the data. For Type1 requests, the corresponding original security policies in PA and PB apply. For Type3 requests, the policy integration needs to generate a new security policy to address the security concerns from both organizations. To generate such policies, similar to a conflict reconciliation process, we take the security concerns of the newly generated data from both organizations, and through a negotiation process, a common set of policies accepted by both organizations can be generated. The negotiation process will be discussed in Section 5.

10


For Type2 requests, we will present a similarity-based security policy adaptation algorithm for generating integrated policies. This type of requests does not have existing security policies because the request and the resource belong to different organizations. Original security policies specified by the data owner cannot cover the new users from the other collaborating organization, and the collaborating organization does not have policies to govern the access to data not belonging to the organization. However, because one organization is the only owner of the data, the organization should control the generation of new security policies to address the new users of the data. To deal with this type of requests, adapting the existing security policies of the organization, which owns the data, to handle the new user set, is better than negotiation with the other organization. Because the data owner may have little knowledge about a requester which is not part of its original user set, it is difficult for the data owner to know how to adapt its security policy accordingly. However, it is very difficult to adapt existing security policies to address the new users without heavy human intervention, Because an organization may have specified a list of hundreds or even thousands security policies for its shared resources, it will greatly reduce the efficiency of the collaboration and easily cause human errors due to the large number of requests, On the other hand, even though the organization having the new requestor does not have a security policy to govern the access to this resource, it is often that there are some security policies for this requestor to access some resources similar to the resource it currently requests for accessing. For example, a user u ∈ UB wants to access the raw medical data dat1 ∈ DatA, which contains patients’ SSN. OrgA does not have existing security policy to cover u. If OrgA knows that u can access dat2 ∈ DatB, which contains patients’ mental evaluation results, medication history, and other sensitive information, it is reasonable for OrgA to draw the conclusion that u is trusted to handle sensitive data in OrgB. It is less risky to extend the access of dat1 to u compared to some other user u’ from the same organization which does not have such kind of access rights. Based on this observation, we can develop the following policy adaption algorithm based on policy similarity evaluation. We first need to define a metric for the similarities among different entities. Because our security policy specification is based on a vocabulary organized in a hierarchical structure [3], relative relations between various elements in security policies specifications can be established. Let e1, e2, and e3 be any three elements, e4 be the lowest common ancestor of e1 and e2 in the hierarchical structure, and e5 be the lowest common ancestor of e1 and e3. Element e1 is said to be closer to e2 than to e3 if e4 appears at a lower level in the class hierarchy than e5. The similarity index SI(ei , ej) between two elements ei and ej, is defined as follows:

{

SI (ei , e j ) =

1 if ei = e j SI (ek , e j ) / n if e j is ei 's ancestor, where n is the number of siblings e1 has and ek is ei 's parent SI (ei , el ) × SI (e j , el ) for all other cases, where el is the lowest common ancester of ei and e j

For example, e1:“SHA-1”is a child of e4:“Hash Function”. e2:“MD5” is also a child of e4. e3:“Fingerprint checking” is a child of e5: “Biometric Authentication” mechanism. Both e4 and e5 are derived from e6: “SecurityMechanism”. If these are all the elements in this example, then the similarity between e1 and e2, and between e1 and e3 can be calculated as follows:


11

SI(e1 , e2)= SI(e1 , e4) × SI(e2 , e4)=(1/2)×(1/2)=0.25, SI(e1 , e3)= SI(e1 , e6) × SI(e3 , e6)=((1/2)/2)×(1/2)=0.125. SI(e1 , e2) > SI(e1 , e3), which is in line with the fact that “SHA-1” is closer to “MD5” compared to “Fingerprint checking”. The similarity between two atomic security policies, P= (S, A, O, C) and P’ = (S’, A’, O’, C’), is defined as follows: SI(P, P’) = SI(S, S’) × SI(A, A’) × SI(O, O’) × SI(C, C’). The similarity between a composite security policy P = (P1∧P2) ∨ P3∨ ~P4 and an atomic security policy P’ is defined as follows: SI(P, P’)= (SI(P1, P’)× SI(P2, P’))+ SI(P3, P’)+(- SI(P4, P’)). The candidate policy for adaption can be a composite security policy. When searching for similar policies for the request in the collaborating organization, we only search for the atomic policies related to the requester. If there is a composite security policy, we can treat it as several atomic policies during our similarity calculation. Thus, we do not define the similarity between two composite security policies. Based on these definitions, our similarity-based security policy integration algorithm allows the data owner to make the decisions on whether to grant access to a requester from a participating organization based on similarity analysis of the two organizations’ security policies. Let P be the security policy specified by the data owner for a specific data dat. Let TS be a similarity threshold the data owner specified. A security policy P’ in the collaborating organization is selected by the data owner to help it make security decision only if SI(P, P’) is greater than TS. A greater TS will guarantee the selected policies to be closer to P, but if the selection criteria is too tight, it may not be able to return any policy which is certainly not the purpose of this process. The similarity-based security policy adaptation algorithm can be presented as follows: Input: a request where dat DatA , u UB Process: 1) Based on the request action and current situation, find the suitable policy, PA, in OrgA which is used to handle this kind of request if the user is from OrgA. 2) Find PS(u), which is the set of atomic policies regarding the requester u in OrgB. 3) For every P in PS(u), calculate SI(PA, P) 4) Select a subset PS’(u) of PS(u), where for every P in PS’(u), SI(PA, P) > TS 5) Divide PS’(u) into two subsets, PSP(u) and PSD(u), where PSP(u) is the set of rules that permit access for u, and PSD(u) is the set of rules that deny access for u. SI ( PA , P) ¦ SI ( PA , P) P¦ PPSP ( u ) PS D ( u ) 6) Calculate R u W , where W is a confidence min( SI ( PA , P), P PS '(u )) weight OrgA assigned to OrgB. W is originally set to 1, and every time if OrgA grants access for a user in OrgB based on this calculation and the user does not cause any security problem, W is increased by 1. On the other hand, if the user failed to meet the expectation, W is decreased by 1. Output: If R > 1, the request is granted, otherwise it is denied.

12


The original similarity threshold can be generated based on sampling result of the collaboration with domain knowledge. If on the average, every node has N children, two policies with a similarity index in the range N-3 ~ N-4 are considered to be similar. At runtime, the threshold can be updated based on previous evaluation results to make it more suitable for a specific collaboration. The advantage of this approach is that this process can be automated and very efficient.

5 Negotiation-Based Conflict Reconciliation Protocol We consider two security policies P1 and P2 have conflict with each other if for their satisfactory sets Q1 and Q2, there exists an element e such that e ∈ Q1 and e ∈ Q2 , where Q1 and Q2 are satisfactory sets for P1 and P2 respectively, which contain all the elements that comply with their corresponding policies. As discussed in Section 1, due to the dynamic nature of collaborations among organizations in ubiquitous computing environments, when specifying its own security policies, an organization has difficulties in knowing what security policies have specified by other organizations which may collaborate with in the future. Thus, for a complex system, conflicts are hard to avoid. It is important that the collaborations do not undermine the security of all participating organizations. On the other hand, security should not deny participating organizations the possibility of collaboration simply because there exist some conflicting security policies. Our conflict reconciliation protocol can improve the collaboration chance. It is based on the following observations of real world scenario: it is difficult to find suitable collaborator and collaboration can generate extra synergies. Thus, instead of rejecting a possible collaborator because of some conflicting security policies, collaborating organizations often prefer to take certain risk and move forward with the collaboration by relaxing their security policies and adopting some weaker security policies to resolve the conflicts. Policy P1 is said to be weaker than P2 if P1 a Q1, P2 a Q2, and Q2 ⊂ Q1. Because each collaboration goal and collaborating participants may be different, the security compromise should not be done in a static manner. Our reconciliation protocol captures these dynamic factors as situation-aware compromise thresholds, which specify how much compromise an organization is willing to make for a specific collaboration. A compromise threshold, TC, is specified based on the following factors: (1) How much benefit can be generated if this collaboration goes through? (2) How much damage it may cause by adopting a weaker security policy? (3) What is the track record of this specific collaborating organization? How did it perform in pervious collaborations? The value of Tc should be updated when the above three factors change. Now, we will present the negotiation-based conflict reconciliation protocol as follows:


13

Step1 For a specific resource, each organization specifies a chain of security requirements instead of just one security requirement. The head of the chain is the most desirable security policy. Traveling through the chain from the head, the security requirements become weaker and weaker. Step2 Each organization specifies compromise thresholds under various situations as (TC,T), which means when situation T is true, the weakest security policy it is willing to take is indexed by TC along the chain. Step3 At runtime, the system obtains the compromise thresholds TC1 and TC2 for both organizations based on the current situation, and identifies the index for the two conflict policies in their chain. Test to see whether the current index is less than the compromise threshold. If not, go to Step6. Step4 One organization is randomly selected to make compromise first, and proceeds along the policy chain to select a weaker policy; Step5 Test whether there is still any conflict. If no conflict is detected, then reconciliation is done. If there is still conflict, repeat Step4 by proceeds along the other organization’s policy chain. Step6 Both organizations make compromise in turns, until one of the compromise thresholds is reached. If no reconciled policy is generated by then, failure is reported for human intervention.

For two conflict policies P1 and P2, the above steps can be summarized in the following pseudo-code. // sp1[] is the policy chain contains P1, and sp2[] is the policy chain contains P2 x1 = GetCompThreshold1(θ ); x2 = GetCompThreshold2(θ ); n1 = Sizeof(sp1); n2 = Sizeof(sp2); flag = 0; for (p=0, q=0; p≤n1, q ≤ n2; ) { if ((sp1[p] not conflict with sq2[q]) substitute P1 with sp1[p] and P2 with sp2[q], break; else{ if(flag == 0){ p++; flag = 1; }else{ q++; flag = 0; } } } if (p > n1 || q > n2) report the conflict cannot be automatically reconciled For the sake of simplicity, we only illustrate the reconciliation process between two collaborating organizations. The reconciliation process involving more than two organizations can be handled in a similar fashion: We will have multiple security policy chains and the compromise may be made in a loop circling all participating organizations, instead of going back and forth between two organizations.

14


6 An Illustrative Example In this section, we will give an example to illustrate our approach. Continuing with the scenario described in Section 4, a university OrgA and a pharmaceutical company OrgB collaborate on a joint vaccine development project. UA is made up of a set of professors (UP), a set of graduate research assistants (UG) and a set of undergraduate students (UU). DatA includes some research papers (datAp), unpublished experiment results and paper drafts (datAo), and composition data on several promising chemicals, which the users in UA intend to use (datAq). UB consists of a set of research scientists (US), a set of lab technicians (UT) and several directors for research (UD). DatB includes patents of existing drugs related to this disease (datBn), a list of potential clinical trial participants (datBr), and a set of secret methods for fast vaccine prototype development (datBm). These users and data can be organized in a hierarchical manner as shown in Figure 3. Entity Is a

Is a

Resource

User

Is a

Data

Is a

dat Ap

Is a

Confidential Info

Is a

dat Bn

Is a

dat Ao

UD

dat Aq

Is a

Participant Is a

UG

Is a Is a

Is a

Supervisor

Is a

Is a

Pub Info

Is a

Is a Is a

Volunteer

Investigator

Is a

Is a

UU

Is a

UT

dat Bm

UP

Is a

US

DatBr

Fig. 3. The hierarchy of concepts of the example

To follow our approach, Part1), we have the following security policies specified: OrgA – (1) All members can access the published papers at anytime. P1: (UP ∪ UG ∪ UU, Access, datAp, __ ) (2) Professors can always access and modify the unpublished paper draft. P2: (UP, Access∪Modify, datAo, __ ) (3) Graduate assistants can always access the unpublished paper draft. However, they can only update the draft if they are the co-authors. P3: (UG, Access, datAo, __ ) ∧ (UG, Modify, datAo, C3), C3: Up ⊂ Author(datAo) OrgB – (1) All members can access the drug patents at anytime. P4: (UD ∪ US ∪ UT, Access, datBn, __ ) (2) Scientists and directors can access the trial participant list. P5: (UD ∪ US, Access, datBr, __ ) (3) Only directors can update the trial participant list. P6: (UD, Modify, datAo, __ )


15

Part 2), if a professor u from OrgA wants to access the list of trial participants, OrgB’s current policies cannot handle this request. Follow our similarity-based algorithm, we have 1) Because the request is to access datBr, P5 is the suitable policy for similarity comparison. We can transform P5 into P5’:(UD, Access, datBr, __ ) ∨ P5’’:(US, Access, datBr, __ ) 2) Select policies for professors in OrgA. We only consider the atomic component for professors if the policy also applies to other users. Thus, PS(u) = { P1’:(UP, Access, datAp, __ ), P2’:(UP, Access, datAo, __ ), P2’’:(UP, Modify, datAo, __ )}. 3) SI(P5, P1’) = SI(P5’, P1’) + SI(P5’’, P1’). SI(P5’, P1’) = SI(UD, UP) × SI(Access, Access) × SI(datBr, datAp) × SI( __, __) = 1/32 × 1 × 1/32 × 1 = 1/1024. Similarly, SI(P5’’, P1’) = 1/16. Thus SI(P5, P1’) = 65/1024. We can also calculate SI(P5, P2’) = 9/128, and SI(P5, P2’’) = 9/3200. 4) For this example, assume that OrgB sets the threshold Ts = N-4. Because on the average every node has 3 children, Ts = 1/81, and hence both P1’ and P2’ are selected. 5) Both P1’ and P2’ grant access to u, thus PSP(u) = { P1’ , P2’ } and PSD(u) = φ 6) Assume that this is the first time the two organizations collaborate. Then, W =1. R = (SI(P5, P1’)+ SI(P5, P2’)) / min (SI(P5, P1’), SI(P5, P2’)) = (65/1024+9/128) / min(65/1024, 9/128) = 137/65. Because R > 1, the access is granted to u. The system can make this decision automatically without any human supervision at runtime. Because professors in OrgA are the main contributors (Investigators) for this project, and professors can access confidential information in OrgA, it is reasonable for a human supervisor in OrgB to grant a professor from OrgA the access to the participant list. This decision is in line with the real world situation. Part3), since negotiation is used for conflict reconciliation as well as security policy generation for newly generated resources, we only need to show an example of the latter to illustrate this negotiation process. Assume that based on datAq, the collaboration developed several vaccine prototypes using the methods listed in datBm, and performed several clinical trials. We will have a new set of data datk which is the evaluation results of these vaccine prototypes. Neither OrgA nor OrgB can claim the ownership of datk completely. Thus, a security policy for datk will be generated through negotiation. Following our negotiation protocol discussed in Section 5, for this example, Step 1. OrgA presents its list of input policies sp1[], with the most desirable policy for at front. OrgB also presents its inputs sp2[]. Step 2. Both OrgA and OrgB specify their compromise thresholds for various situations as (TC,θ) pairs. Step 3. After analyzing the collaboration history and the importance of the data, the compromise thresholds for both organizations are 2. In the following we only list the input policies up to the threshold.

16


OrgA: (1) Only investigators can access datk through a communication channel using 64-bit encryption. sp1[0]: (UI, Access, datk, “64-bit encryption” ) (2) Both investigators and participants can access datk through a communication channel using 64-bit encryption. sp1[1]: (UI ∪ UPa, Access, datk, “64-bit encryption” ) (3) Both investigators and participants can access datk. sp1[2]: (UI ∪ UI, Access, datk, __ ) OrgB: (1) Both investigators and participants can access datk through a communication channel using 128-bit encryption. sp2[0]: (UI ∪ UPa, Access, datk, “128-bit encryption” ) (2) Both investigators and participants can access datk through a communication channel using 64-bit encryption. sp1[1]: (UI ∪ UPa, Access, datk, “64-bit encryption” ) (3) Anyone in the collaboration can access datk. sp2[2]: (U, Access, datk, __ ) Both organizations start with their first choice, sp1[0] and sp2[0]. Obviously, there is a conflict since sp1[0] allows less investigators to access datk while sp2[0] grant access to both investigators and participants. Step 4. OrgA is randomly selected to initialize the negotiation. It will proceed along the input policy list and present the next policy sp1[1] which is weaker than sp1[0]. Step 5. After analyzing sp1[1] and sp2[0], there is still a conflict since sp2[0] requires access through 128-bit encryption while sp1[1] only requires 64-bit encryption. Hence, Step 4 is repeated. This time, OrgB needs to make its compromise in the negotiation process. The next policy in its input list, sp2[1], is presented. Step 6. After analysis, no conflict is found between the two input policies sp1[1] and sp2[1], and hence a security policy for datk is generated: (UI ∪ UPa, Access, datk, “64-bit encryption” ) Note that we have not reached the compromise threshold. Though there is no conflict between sp1[2] and sp2[2] either, the negotiation process stops as soon as a pair of non-conflict policies is found because they are stronger than the policies appearing later in the list. This will provide the best protection for the collaboration system without creating any conflict.

7 Conclusions and Future Work In this paper, we have presented an approach to security policy integration and conflict reconciliation for collaborations among organizations in ubiquitous computing environments. Based on the similarities between security policies, a resource owner can adapt its existing security policies to cover the new users from collaborating organizations using the similarity indices as a selection criterion for selecting similar policies in the collaborating organization and make a decision based on their inputs. Collaborating organizations with conflicting security policies can try to resolve the conflicts by gradually making security compromise and relaxing their security


17

policies until either the conflict is resolved or the conflict reconciliation process reaches certain compromise threshold associated with that particular collaboration. This approach can also be used for generating security policies for the new resources generated as a result of the collaboration, where all participating organizations specify their security concerns for the resource, and the negotiation process is trying to find a common set of policies that will satisfy all the organizations’ concerns. We plan to conduct case studies to establish some applicable guidelines for specifying the similarity threshold and updating the confidence weight in the decision making process. In addition, we will investigate the impact of extending the security policy set through logical reasoning and try to associate that impact with policy similarity so that we can update the resource owner’s security policies based on the similarity analysis, rather than relying on similar security policies specified by the collaborating organizations.

References [1] Xu, F., Lin, G., Huang, H., Xie, L.: Role-based Access Control System for Web Services. In: Proc. 4th Int’l Conf. on Computer and Information Technology, pp. 357–362 (2004) [2] Bhatt, R., Joshi, J.B.D., Bertino, E., Ghafoor, A.: Access Control in Dynamic XMLBased Web Services with XRBAC. In: Proc. 1st Int’l Conf. on Web Services (2003), http://www.sis.pitt.edu/~jjoshi/ICWS_XRBAC_Final_PDF.pdf [3] Yau, S.S., Chen, Z.: A Framework for Specifying and Managing Security Requirements in Collaborative Systems. In: Yang, L.T., Jin, H., Ma, J., Ungerer, T. (eds.) ATC 2006. LNCS, vol. 4158, pp. 500–510. Springer, Heidelberg (2006) [4] OASIS, eXtensible Access Control Markup Language (XACML) Version 2.0, OASIS standard (2005), http://docs.oasis-open.org/xacml/2.0/ access_control-xacml-2.0-core-spec-os.pdf [5] Bonatti, P.A., Sapino, M.L., Subrahmanian, V.S.: Merging Heterogeneous Security Orderings. In: Martella, G., Kurth, H., Montolivo, E., Bertino, E. (eds.) ESORICS 1996. LNCS, vol. 1146, pp. 183–197. Springer, Heidelberg (1996) [6] Bidan, C., Issarny, V.: Dealing with Multi-policy Security in Large Open Distributed Systems. In: Quisquater, J.-J., Deswarte, Y., Meadows, C., Gollmann, D. (eds.) ESORICS 1998. LNCS, vol. 1485, pp. 51–66. Springer, Heidelberg (1998) [7] Backes, M., Durmuth, M., Steinwandt, R.: An Algebra for Computing Enterprise Privacy Policies. In: Samarati, P., Ryan, P.Y.A., Gollmann, D., Molva, R. (eds.) ESORICS 2004. LNCS, vol. 3193, pp. 33–52. Springer, Heidelberg (2004) [8] Ribeiro, C., Zuquete, A., Ferreira, P.: SPL: An access control language for security policies with complex constraints. In: Proc. Network and Distributed System Security Symp (NDSS 2001), pp. 89–107 (2001) [9] Lupu, E., Sloman, M.: Conflicts in policy-based distributed systems management. IEEE Trans. on Software Engineering 25(6), 852–869 (1999) [10] Agrawal, D., Giles, J., Lee, K.W., Lobo, J.: Policy ratification. In: Proc. 6th IEEE Int’l Workshop on Policies for Distributed Systems and Networks (POLICY), pp. 223–232 (2005) [11] Yau, S.S., Huang, D., Gong, H., Yao, Y.: Support for Situation-Awareness in Trustworthy Ubiquitous Computing Application Software. J. Software Practice and Engineering 36(9), 893–921 (2006)

18


Appendix Ontology-Based Security Policy Specification Our approach uses ontology-based security policy specification language in [3]. For the sake of completeness, we summarize the important features of the specification language. Natural language security policy specifications can be expressed by a set of logic expressions with the key vocabulary derived from a hierarchical security policy ontology. The ontology is shown in Fig. 4, where the generic classes in the core ontology are defined as follows: • SecurityPolicy can be an AtomicSecurityPolicy or CompositeSecurityPolicy. • CompositeSecurityPolicy is composed of AtomicSecurityPolicy. • AtomicSecurityPolicy has four components: Subject, Object, Action and Condition. • Both Subject and Object are Entity. • Entity can be User or Resource, and Data is also considered as Resource. • Action specifies what action Subject can perform on Object, which includes Access, Share, Delegate, Delete and Modify. • Condition can be specified as a specific SecurityMechanism, which must be adopted, or as a certain environment or system Situation, which will trigger the security policy. The value of a Situation expression is determined by situationaware processors [11].

Security Policy

compose

Is a

Is a

Composite Security Policy

Has

Atomic Security Policy

Has

Subject Is a

Is a

Condition

Has Has Is a

Action

Is a

Resource

Is a

Is a Is a

User Is a

Data

Is a

Security Mechanism

Object

Entity

Is a

Access

Is a

Share

Situation

Is a

Modify

Delegate Delete

Fig. 4. An ontology for security policy specification

This ontology provides the fundamental vocabulary for specifying security policies. However, users can extend the basic concepts and introduce more applicationspecific terms when specifying their own security policies. The basic building block is the atomic security policy, which can be specified as a quadtuple: P = (S, A, O, C), where S ⊂ Subject, A ⊂ Action, O ⊂ Object, and C ⊂ Condition. The quadtuple is interpreted as follows: Security policy P specifies that a set of subjects S can perform


19

a set of actions A on a set of objects O when a set of conditions C is met. A request is granted if there exists a policy P = (S, A, O, C) such that the requester s ∈ S, the requested action a ∈ A, the requested target o ∈ O and the conditions listed in C are satisfied when the request is made. If C is omitted, it means the policy applies to all conditions. Atomic security policies can be composed through logic operators conjunction ‘∧’, disjunction ‘∨’ and negation ‘¬’ to form more complex security policy specifications. Based on these definitions, for a give set of security policies, we can derive a satisfactory set Q which contains all the elements that comply with the given set of policies. We use P a Q to denote this logic reasoning process. This specification approach has two advantages. First, because all four elements within a specification are specified based on the ontology, each element is clearly defined. There is only one interpretation for a quadtuple, and thus a specified security requirement is unambiguous. Second, the hierarchical structure of the ontology can be used to establish common understanding of different terms used by different organizations in their security policy specifications. For example, one policy specifies “administrator can only access the account information through a 128-bit encrypted channel”, and the other policy specifies “administrator must use a 128-bit encrypted channel to access the username/password information”. Based on the ontology we used in our specifications, both “account information” and “username/password information” are Objects. If the two collaborating organizations establish a common understanding that “username/password” used by OrgA is the “account information” used by OrgB, the system will be able to “understand” this fact and process the policies from both organizations based on this fact accordingly. In this example, the two security policies will be treated as the same policy. The algorithm to bridge different terms used by different organizations and form common understandings was presented in [3].

Improved Weighted Centroid Localization in Smart Ubiquitous Environments Stephan Schuhmann1 , Klaus Herrmann1 , Kurt Rothermel1 , Jan Blumenthal2 , and Dirk Timmermann2 1 University of Stuttgart, Germany Institute of Parallel and Distributed Systems (IPVS) {firstname.lastname}@ipvs.uni-stuttgart.de 2 University of Rostock, Germany Institute of Applied Microelectronics and CE {jan.blumenthal,dirk.timmermann}@uni-rostock.de

Abstract. Location-awareness is highly relevant subject in ubiquitous computing, as many applications exploit location information to provide adequate services or adapt to a changing physical environment. While GPS provides reliable outdoor localization, indoor positioning systems present a bigger challenge. Many indoor localization systems have been proposed. However, most of them rely on customized hardware or presume some dedicated infrastructure. In this paper, we focus on WLAN-based localization in smart ubiquitous environments. We propose an improved scheme of the Weighted Centroid Localization (WCL) algorithm that is robust and provides higher location accuracy than the original WCL algorithm. The improvements are based on the use of dynamic weighting factors that are solely dependent on the correlation of the Received Signal Strength Indicators of the received beacon signals. Compared to the original WCL scheme, our approach does not increase requirements to the environment. Real-world experiments in a typical environment that we report on in this paper confirm that the increased location accuracy determined in previous calculations is reproducible in a realistic noisy environment. This provides a simple, cost-efficient, and battery-conserving, but yet adequate technique for getting the accurate location information of mobile devices.

1

Introduction

Location-awareness is an essential service for many wireless ubiquitous computing scenarios, as many applications integrate location information to increase context knowledge. However, ubiquitous computing environments have specific characteristics which limit the approaches that are applicable to maintain location-awareness. For instance, pervasive applications have to deal with fluctuating availability of devices due to mobility and network failures. Thus, environmental contexts are highly dynamic. Furthermore, processor performance and available energy of mobile nodes in ubiquitous computing scenarios are often limited. Therefore, intensive communication and computation tasks are not F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 20–34, 2008. c Springer-Verlag Berlin Heidelberg 2008

Improved Weighted Centroid Localization

21

feasible. In this context, algorithms are subject to strict requirements covering reduced memory consumption, communication, and processing time. In Smart Environments, available infrastructure devices can provide additional support for mobile devices. Determining the position of nodes in wireless networks, particularly in noisy environments, represents a real challenge. To identify the exact coordinates of a device (also called Station Of Interest, or SOI ) requires measuring a distance, e.g., measuring Time of Arrival (ToA) or Time Difference of Arrival (TDoA). Difficulties concerning time measurements result from synchronization of involved devices as well as the high computational effort to calculate the position. Measuring the Received Signal Strength Indicator (RSSI) offers a possibility to realize distance determination with minimal effort. A good localization algorithm should calculate a position as fast as possible and should be resistant to environmental influences as well as imprecise distances. However, it is desirable to use standard off-the-shelf consumer products without the necessity for far-reaching customization as this reduces the effort necessary for setting up the positioning system. Thus, our approach of combining the Weighted Centroid Localization (WCL) [1] with standard Wireless LAN technology has high practical relevance in typical environments. The main contribution of this paper is an improved Weighted Centroid Localization scheme that uses weighting factors of dynamic degrees to increase location accuracy in smart ubiquitous environments. Therefore, we derived optimal dynamic weighting factors through theoretical calculations and approximated the obtained weighting factors by a power function using the least-squares method. Furthermore, we evaluated the improved localization scheme in an office room that represents a typical noisy ubiquitous computing environment. The measurements confirmed that our approach increases location accuracy almost to the best possible value for weighted centroid schemes. The paper at hand is divided into six sections. The second section discusses the requirements to be met by the localization scheme and the environment. Section 3 gives a broad survey of existing localization approaches. In Section 4, we at first describe the original WCL scheme. Following, we derive an improved WCL scheme that uses dynamic weighting factors and increases location accuracy. We present our theoretical and practical evaluation results in Section 5. This is followed by the conclusion and a brief outlook to future work in Section 6 which closes this paper.

2

Requirements

In this paper, we focus on indoor localization for resource-restricted devices in smart ubiquitous environments. This requires to avoid high battery consumption, as mobile devices like PDAs or smart phones are equipped with limited battery and computation power. Hence, we aim at minimizing the requirements for the localization devices as well as the infrastructure to allow applicability to a wide

22

S. Schuhmann et al.

range of scenarios. This means we want to provide a flexible solution for indoor localization. Thus, our system has to maintain the following properties: – Minor Environmental Requirements: The environment solely has to provide infrastructure nodes with fixed positions that send periodic beacon signals. Many realistic ubiquitous scenarios fulfill this property, as they include 802.11 WLAN Access Points (APs), for example. These devices have to know their exact position and notify other devices of their location, e.g. within the beacon signals which needs slight adaptations to the APs. If changes to the APs are undesired, AP positions can be received by queries to a location server that provides the clients with AP locations. – Minimal Costs: Additional hardware costs should be avoided by relying on standard components that are available in typical ubiquitous scenarios, both on the client and on the infrastructure side. For instance, the localization scheme ought to integrate existing infrastructure devices without the need of large changes in their configuration. To minimize costs, we rely on exactly four beacon nodes in this paper. These beacons are placed at the edges of a rectangular plain, as common rooms typically have a rectangular base. This AP positioning enables nodes to localize themselves in the whole region. – Minimal Communication Needs on the Client Devices: Since power supply is critical for small mobile devices, the clients should avoid any additional communication during the localization process to maximize battery lifetimes. So, only the infrastructure devices should send beacons needed for the localization process, while the mobile devices have to exploit the information provided by these beacons. As localization is supposed to be executed in various scenarios in real-time, a solution that demands prior configuration of the client devices is not feasible, as this limits the use of the system. – Applicability in Realistic Ubiquitous Scenarios: Contrary to laboratory environments, realistic scenarios often suffer from effects like multi-path propagation or interferences with other wireless networks operating on the same frequency spectrum. This can significantly influence the accuracy of the localization. A localization scheme must be able to cope with this interference in order to provide accurate localization nevertheless.

3

Related Work

In this section, we investigate several related localization approaches concerning the above requirements to confine WCL from these approaches, and we clarify why they prove inadequate in our scenarios. In Figure 1, we provide a classification of proposed schemes. At first, these schemes can be divided into those that are based on coordinates and those that are coordinate-free. Coordinate-free schemes, as presented by Fekete et al. [2], focus on an abstract way of location-awareness. These schemes aim at achieving consistent solutions by the use of geographic clustering. Unfortunately, they rely on rather dense sensor networks to achieve adequate accuracy and, therefore, are not usable here. Thus, we focus on coordinate-based approaches that can further be divided into


23

Fig. 1. Classification of proposed localization schemes

range-free and range-based schemes. Range-free schemes comprise implicit distance measurements, while range-based schemes use explicit distances for localization. Regarding range-free schemes, there exist three main approaches: – Anchor-based approach: This approach uses anchor beacons, containing two-dimensional location information (xi , yi ), to estimate node positions. After receiving these beacons, a node estimates its location using a specific formula. Besides a scheme proposed by Pandey et al. [3], another anchorbased approach is Centroid Localization (CL), proposed by Bulusu et al. [4]. In this approach, the SOI calculates its position as the centroid of the coordinates from the stations whose beacons are received. While CL performs only averaging the coordinates of beacons to localize SOIs, the Weighted Centroid Localization (WCL) algorithm [1] uses weights to ensure an improved localization and, hence, represents a direct advance over CL. – Hop-based approach: In this type of localization scheme, the SOI calculates its position based on the hop distance to other nodes [5],[6]. This scheme delivers an approximate position for all nodes in networks where only a limited fraction of nodes have self-positioning capability. Through this mechanism, all nodes in the network (including other anchors) get the shortest distance, in hops, to every anchor. However, this approach only works well in dense networks and relies on flooding of messages which produces high communication overhead that is undesired in our scenarios. – Area-based approach: Area-based schemes [7],[8] perform location estimation by isolating the environment into triangular regions between beaconing nodes. The presence inside or outside of these triangular regions allows a node to narrow down the area in which it can potentially reside. Area-based approaches normally perform particularly well in heterogeneous networks with high node density. Unfortunately, this condition does not hold in our case and, thus, area-based approaches cannot be used here.

24

S. Schuhmann et al.

Besides range-free schemes, a couple of approaches exist that are based on the transmission ranges of the nodes. Range-based approaches can be subdivided into the following approaches: – Signal Propagation Time: The Time of Arrival (ToA) technology is commonly used as a means of obtaining range information via signal propagation time. This approach is used by satellite navigation systems [9]. Though this approach guarantees high precision and can be globally used, it suffers from several major drawbacks as this scheme can only be used outdoors, needs expensive hardware, and the receivers consume much energy. Thus, ToA approaches are inapplicable in our case. – Localization by Measuring Signal Propagation Time Differences and Arrival Angles: This technology is based on measurements of the Time Difference of Arrival (TDoA) [10], or the Angle of Arrival (AoA) [11]. While TDoA estimates the distance between two communicating nodes (this is also called ranging), AoA allows nodes to estimate and map relative angles between neighbors. Like ToA technology, TDoA and also AoA rely on special hardware that is expensive and energy consuming. Thus, these approaches are unsuited here. – Simple Signal Strength Measurements: This approach uses the Received Signal Strength Indicator (RSSI) of incoming beacon signals and has been proposed for hardware-constrained systems. Contrary to techniques like ToA, TDoA or AoA, RSSI is the only feature that is measurable with reasonably priced current commercial hardware. RSSI techniques use either theoretical or empirical models to translate signal strengths into distance estimates where each point is mapped to a signal strength vector [12], or to a signal strength probability distribution [13]. This technology suffers from problems such as background interference or multi-path fading which make range estimates inaccurate, as shown in [14]. Furthermore, many signal strength based approaches rely on special hardware, like small infrared badges [15], magnetic trackers [16], or multiple cameras [17], which usually are not included in typical environments and mobile devices. Some approaches do not use client-based, but sniffer-based localization [18]. However, this technique cannot be used here as it assumes that the access points can localize other devices which induces major changes in the APs. Many approaches use location fingerprinting for localization in wireless networks [12],[13]. This method consists of a training phase and a positioning phase. Unfortunately, this approach needs additional preconditioning and the creation of a training database for several environments beforehand, which makes it inapplicable in our case. In summary, one can see that most approaches cannot be chosen since they rely on extensive special hardware, need additional preconditioning to build a training database, use sniffer-based localization, or depend on high node densities to perform well. This narrows the huge field of localization schemes down to few anchor-based approaches. Among those, Weighted Centroid Localization uses more fine-grained parameters than CL to weight the received beacon signals, as


25

CL simply uses binary weighting (weight 1 for those APs whose beacons were received, weight 0 for the others). This ensures higher location accuracy for WCL.

4


After a short introduction to the original WCL scheme with static weighting factors [1], we propose an advanced scheme that uses dynamic weighting factors which are based on the correlation of the received RSSI values. 4.1

Static Degree Weighted Centroid Localization (SWCL)

We assume that the network consists of SOIs that want to localize themselves, and n beacons. In our case, beacons are WLAN access points whose position is assumed to be known exactly. We depend on a small number of n = 4 beacons, placed at the edges of a rectangular plain. The SOIs are mobile devices that initially do not know their own position. Algorithms such as CL use centroid determination to calculate a device’s position [4]. In the first phase, each beacon Bj sends its position (xj , yj ) to all nodes within its transmission range, which can simply happen through the periodically sent beacon signals. In the second phase, the SOI Pi calculates an approximation (xi , yi ) of its real position (xi , yi ) by a centroid determination from all n positions of the beacons in range (1). 1 (xj , yj ) n j=1 n

(xi , yi ) =

(1)

The localization error fi for the SOI Pi is defined as the distance between the exact position (xi , yi ) and the approximated position (xi , yi ) of Pi (2). fi = (xi − xi )2 + (yi − yi )2 (2) While CL only averages the coordinates of beacon devices to localize SOIs, WCL uses weights to ensure an improved localization. In CL, all weights of received beacon signals are implicitly equal to 1. WCL represents a generalization of this scheme since it introduces variable weights wij for each device Pi and each beacon Bj . These weights depend on the distance between the two as it will be explained later in this section. In the more general WCL equation, the number of beacons n is replaced by the sum of weight factors wij , and each beacon position (xj , yj ) is multiplied by its weight factor wij . Hence, (1) is expanded to the WCL formula (3) yielding the new approximation (xi , yi ) for Pi ’s position. n j=1 wij · (xj , yj ) n (xi , yi ) = (3) i=1 wij The weight wij is a function depending on the distance and the characteristics of the SOI’s receivers. In WCL, shorter distances cause higher weights. Thus, wij

26

S. Schuhmann et al.

and dij (the distance between beacon Bj and SOI Pi ) are inversely proportional. As an approximation, the correlation is equivalent to the function 1/dij . To weight longer distances marginally lower, the distance is raised to a higher power hs . For a concentric wave expansion with a linear characteristic of the receiver and a uniform density of the beacons, we form (4). wij =

1 (dij )hs

(4)

The static degree hs has to ensure that remote beacons still impact the position determination. Otherwise in case of a very high hs , the approximated position moves to the closest beacon’s position and the positioning error fi increases. There exists a minimum of fi where hs is optimal [19]. However, as the next section clarifies, this use of static weight values leads to suboptimal accuracy if you compare it to the use of adapted dynamic weight factors that are based on the correlation between the received RSSIs from several APs. According to Friis’ free space transmission equation (5), the detected signal strength decreases quadratically with the distance to the sender. Prx = Ptx · Gtx · Grx · (

λ 2 ) 4πdij

(5)

Prx = Remaining power of wave at receiver Ptx = Transmission power of sender Gtx , Grx = Gain of transmitter and receiver λ = Wave length dij = Distance between sender and receiver In embedded devices, the received signal strength is converted to the RSSI which is defined as ratio of the received power to the reference power (Pref ). Typically, the reference power represents an absolute value of Pref = 1 mW . The actually received power Prx can be obtained by the RSSI and vice versa, as denoted in the following equations. Prx = Pref · 10

RSSI 20

⇐⇒

RSSI = 20 · log10

Prx Pref

(6)

An increasing received power results in a rising RSSI. Thus, distance dij is indirect proportional to Prx , and (7) with power gs ( = hs ) is formed. gs wij = (Prx )

(7)

So, it can be seen that the knowledge of dij is not necessary, as the algorithm can simply use the received power Prx or, alternatively, the RSSI. In this paper, we use the RSSI, i.e. we take (6) and (7), and form (8). wij = (Pref · 10

RSSI 20

)gs .

(8)


AP3

R1,1 AP1

R1,2

R2,1 l1

AP4 AP1

AP2

a)

l2

AP3

R1,1

R2,2

l1

R2,2

R1,2

R2,1 AP2

27

AP4

l2

b)

Fig. 2. a) Distribution of gopt at a rectangular plain (p = 43 ), b) Potential for improvement of location error at a rectangular plain (p = 43 ), relative to longer side l1

In practical scenarios, the ideal distribution of Prx does not hold, because the radio signal propagation is interfered with many influencing effects, like reflection on objects, diffraction at edges, or superposition of electro-magnetic fields. Thus, Prx and Prx represent different values which yields localization errors. 4.2

Dynamic Degree Weighted Centroid Localization (DWCL)

Using an optimally chosen static degree gs , as original WCL does, results in improved location accuracy, compared to CL [1]. However, the degree gopt that leads to minimal error can significantly differ from gs in some regions, as this section will clarify. Therefore, we used (5) and (6), and put Prx = Prx to calculate the expected RSSI in various positions within an idealized region under observation. Following, we used the WCL algorithm with different powers g to determine the optimal gopt which yields highest location accuracy at a particular place in the region. Figure 2a shows the distribution of the factors gopt in a rectangular region where the proportion p of the length l1 of the longer side divided by the length l2 (≤ l1 ) of the shorter side is p = ll12 = 43 . It can be seen that the distribution of gopt is very uneven. The values at the regions R1,1 and R1,2 exactly between two adjacent anchors with a distance of l1 to each other (e.g., AP1 and AP3 ) are relatively high (up to 5.0), as well as at those regions R2,1 and R2,2 between two anchors with a distance of l2 to each other (e.g., AP1 and AP2 ), where the maximum gopt is approximately 2.5. In the special case of a quadratic plain (l1 = l2 , i.e. p = 1), the distribution of gopt is symmetric to the center of the region. By selecting the determined optimal dynamic powers shown in Figure 2a, the mean error could be decreased by about 30 % (compared to WCL with an optimal static weight factor of gs = 0.87 in a region with p = 43 ) to the best possible value for WCL. Thus, it is obvious that there is huge potential for improvements by using dynamic weight factors. Figure 2b shows this potential which represents the difference between the error if gs is used, and the minimum possible error if

28

S. Schuhmann et al.

the optimal gopt is used. This figure clarifies that the potential for improvements is alike very unevenly distributed. Particularly the regions Ri,j (i, j ∈ {1, 2}) offer an enormous potential of more than 10 % of l1 for improvements, which corresponds to the noticeably increased optimal weight factor at these regions, as Figure 2a shows. It is obvious that the influence of the two farthest access points was too high there, as the value of gs was chosen too low. This shifted the calculated position near the centre of the region. The proceeding of DWCL to obtain the optimal dynamic weighting factor gd is the following: At first, the SOI has to identify the approximate subregion it is located in. Then, it has to lookup subregion-specific parameters. Now, the SOI can at first calculate the value of a balancing function and, based on this result, the approximated actual dynamic weight gd and its location within the whole region under observation. We will focus on these steps in the following. Subregion Determination. First of all, a SOI has to determine the approximate subregion in which it is located. This has to happen by solely regarding the RSSIs of incoming beacon signals. Figure 2a suggests to define four different subregions, as there exist four peaks for the optimal values of g. A simple way to divide the region in adequate subregions is using the perpendicular bisectors of the sides from AP1 to AP4 and from AP2 to AP3 , as illustrated in Figure 3. A SOI then can easily determine in which of the emerging subregions C1,1 , C1,2 , C2,1 , and C2,2 it is located. This is performed by simply comparing the RSSIs pairwise, as denoted in Table 1. Table 1. Determination of subregions RSSI constraints

Subregion

(RSSI(AP1 ) > RSSI(AP4 )) ∧ (RSSI(AP3 ) > RSSI(AP2 )) (RSSI(AP4 ) > RSSI(AP1 )) ∧ (RSSI(AP2 ) > RSSI(AP3 )) (RSSI(AP1 ) > RSSI(AP4 )) ∧ (RSSI(AP2 ) > RSSI(AP3 )) (RSSI(AP4 ) > RSSI(AP1 )) ∧ (RSSI(AP3 ) > RSSI(AP2 ))

C1,1 C1,2 C2,1 C2,2

Determination of Subregion-Specific Parameters. After determining the subregion it is located in, a SOI needs to obtain further parameters which we will derive in this paragraph. Regarding the beacon signals received within Ri,j , the RSSIs have several special characteristics there: – The RSSIs of the two strongest received signals are almost equal, as the distances from the SOI to the corresponding access points, respectively, are almost the same. The same holds for the two weakest signals. – At those points in Ri,j where gopt is maximal, the quotient Qi (i ∈ {1, 2}) of the distance to the far APs by the distance to the close APs can be calculated by Pythagoras’ formula, as depicted in Figure 3. l22 + ( 12 l1 )2 l12 + ( 12 l2 )2 x1 x2 Q1 = = ; Q = = (9) 2 1 1 y1 y2 2 l1 2 l2


29

Fig. 3. Calculation of parameters Q1 and Q2

Fig. 4. Typical values in common rooms a) for D1 and D2 , b) for a1 , a2 , b1 , and b2

According to Equation 6, the RSSI differences between the received signals from the corresponding APs are Di = 20 · log10 Qi , i ∈ {1, 2}.

(10)

Thus, a SOI only needs to know l1 and l2 to achieve the parameters Qi and Di . The values of l1 and l2 can easily be obtained by the fixed AP positions. The SOI has to use D1 if it is situated in region C1,1 or C1,2 . Otherwise, it has to choose D2 , as it is situated in region C2,1 or C2,2 then. Common values for Di are shown in Figure 4a. For example, in a region with p = 43 , the calculation results for Di are D1 = 9.0908 dB and D2 = 5.1188 dB. The differences between the received RSSIs can generally be expressed as follows: Δk = RSSIk − RSSIk+1 , k ∈ {1, 2, 3} where RSSIk represents the k-th strongest received RSSI. Within regions Ri,j , the values of Δ1 and Δ3 are typically very low (close to 0), while those of Δ2 are close to D1 (in R1,j ) or D2 (in R2,j ).

30

S. Schuhmann et al.

Fig. 5. Distribution of calculated gi (S) in theory (p = 1, i.e. g1 (S) = g2 (S))

Balancing Function. To differ if a SOI is located in Ri,j where gopt is considerably high, we decided to define the following balancing function S that is based on the special RSSI characteristics in Ri,j that were mentioned above: S = Δ1 + |Di − Δ2 | + Δ3 , i ∈ {1, 2}

(11)

As already mentioned, Δ1 and Δ3 are close to zero within Ri,j , while Δ2 depends on the parameter p and is around Di in Ri,j . Thus, (Di − Δ2 ) is around zero in Ri,j . Following, S is minimum within Ri,j and increases outside of these regions. Hence, the use of S helps to state more precisely the location of the SOI. Approximation. We calculated S for various positions in the whole region, and compared the values with those of the optimal gopt at this position. The distribution of S dependent on g is shown in Figure 5 and suggests to approximate gd by the power function that is denoted in (12). ai gd = gi (S) = bi , i ∈ {1, 2} (12) S with the unknown parameters a1 and b1 if the SOI is within C1,1 or C1,2 , and with a2 and b2 if the SOI is located in C2,1 or C2,2 . Now, we approximated the power functions with the least-squares method for various values of p and got the parameters ai and bi as denoted in Figure 4 that yield minimum root mean square (RMS) errors. For example, in case of p = 1, we got a1 = a2 ≈ 1.580 and b1 = b2 ≈ 0.248 which led to a very small RMS error of only 0.002 which is a small fraction of the RMS error of SWCL with optimal static weight.

5

Evaluation

First, we present the theoretical results of DWCL and compare them with those of SWCL. Then, we concentrate on the indoor tests with 802.11 WLAN access


31

Fig. 6. Evaluation results of theoretical calculations 3·108 m

points, operating at a wavelength of λ = fc = 2.44·109sHz ≈ 0.123 m, and compare our practical results with the calculated ones. 5.1

Theoretical Results

We compared the calculated location accuracies of DWCL and SWCL with the results we got by choosing the optimal gopt at each position within the observed region. The results are shown in Figure 6. The mean error of SWCL at a quadratic room (p = 1) is 7.6 %, while the mean error of DWCL is 5.3 % and, hence, very close to the best possible WCL value of 5.1 %. Regarding the maximum error within the region, DWCL and the WCL optimum are almost equal, while SWCL’s accuracy is worse by 2.5 percentage points. The results for a value of p = 2 are quite similar, even though DWCL’s mean gain decreases a bit. 5.2

Indoor WLAN Tests

We implemented the WCL scheme on a common smart phone1 which served as a client device. The infrastructure beacons were represented by standard 802.11 access points2 . The transmission power was put to the maximum possible value of 50 mW with a fixed bandwidth of 11 Mbit/s at 802.11b mode. According to [20], this provides a maximum range of about 67 meters in indoor scenarios. Figure 7a shows the setting for our indoor tests. We used four beacons, A1 to A4 , which were placed in the corners of a quadratic plain of 4.8 m x 4.8 m (i.e., p = 1) in a first test, and a rectangular plain of 5.8 m x 3.2 m (i.e., p ≈ 1.81) in a second test. The tests have been performed in a wall-enclosed office within a 1 2

T-MobileTM MDA Pro with PXA 270 CPU (520 MHz), 48 MB RAM, IEEE 802.11b. CiscoTM Aironet 1200, [20].

32

S. Schuhmann et al.

Fig. 7. a) Indoor test setup, b) Distribution of calculated gd at indoor tests (p = 1)

building where interferences with other WLANs occured. Furthermore, typical office furniture like chairs, desks, and a flip chart were available in the room. The SOI and the APs were placed on a plain table. We deactivated rate shifting on the APs to guarantee fixed transmission powers and performed 20 localization processes at various positions within the region, respectively. Then, we compared the averaged calculated positions with the real ones to determine the location errors. Figure 7b displays the values of the balance function S at the test positions and the approximated power function for p = 1. The mean square error (0.330) is

Fig. 8. Evaluation results of indoor tests


33

higher than in theory as the signals suffered from noise and reflections. However, this error is still about 33.7 % lower than that of SWCL with gs = 0.93. Figure 8 presents a comparison of the mean errors of DWCL, SWCL, and calculations with gopt in these tests. It states that for p = 1, DWCL performs about 0.7 percentage points worse than the theoretical WCL optimum, but at the same time about 2.3 percentage points better than SWCL regarding the mean error, which implies it uses most of the existing potential for improvements. Concerning worst cases, DWCL’s difference to the optimum in accuracy is close to zero, while SWCL performs about 4.2 percentage points worse than the optimum. These results confirm the increased location accuracy by dynamic weighting factors, particularly regarding worst case accuracies in rooms that are almost quadratic where DWCL chooses nearly optimal values for gd .

6

Conclusion and Outlook

In this paper, we proposed an improved Weighted Centroid Localization scheme called DWCL in combination with 802.11 access points as a simple, cost-efficient, applicable, and accurate localization scheme for ubiquitous computing scenarios. We introduced the problem domain, discussed the requirements, and presented the original WCL scheme with static weight factors (SWCL). Following, we developed an improved WCL scheme called DWCL that uses dynamic weighting factors which depend on the RSSIs to increase location accuracy. From our experiments, we conclude that DWCL represents a very effective and yet simple method for indoor localization, as it implies almost optimal location accuracy for weighted centroid schemes with rectangular anchor positioning. It does not pose additional requirements to the clients as well as to the infrastructure beacons. Standard WLAN access points suffice for achieving acceptable localization errors, and the computational burden put on mobile client devices is mild. Due to the fact that WLAN-based WCL can be used in environments with off-the-shelf WLAN hardware, it is appropriate for achieving ad hoc localization in many ubiquitous smart environments with minimal prior setup efforts. The research group at the University of Stuttgart currently investigates if this improved WCL scheme is applicable for localization across several rooms with larger distances between the access points, and alternative positionings of the access points. The research team at the University of Rostock concentrates on the reduction of the positioning error that is mainly caused by border effects. Their current focus is the adaptation of weight functions according to the number of known beacons and the adjusted transmission range of the beacons.

Acknowledgement This work is funded by the German Research Foundation (DFG) within Priority Programme 1140 - Middleware for Self-organizing Infrastructures in Networked Mobile Systems.

34

S. Schuhmann et al.

References 1. Blumenthal, J., Reichenbach, F., Timmermann, D.: Position Estimation in Ad hoc Wireless Sensor Networks with Low Complexity. In: Joint 2nd Workshop on Positioning, Navigation and Communication and 1st Ultra-Wideband Expert Talk, pp. 41–49 (March 2005) 2. Fekete, S.P., Kr¨ oller, A., Buschmann, C., Fischer, S., Pfisterer, D.: Koordinatenfreies Lokationsbewusstsein. it - Information Technology 47(2), 70–78 (2005) 3. Pandey, S., Anjum, F., Kim, B., Agrawal, P.: A low-cost robust localization scheme for WLAN. In: Proceedings of the 2nd Int.’l Workshop on Wireless Internet (2006) 4. Bulusu, N., Heidemann, J., Estrin, D.: GPS-less Low Cost Outdoor Localization For Very Small Devices. IEEE Personal Communications Magazine 7(5), 28–34 (2000) 5. Niculescu, D., Nath, B.: DV Based Positioning in Ad hoc Networks. Journal of Telecommunication Systems (2003) 6. Nagpal, R.: Organizing a global coordinate system from local information on an amorphous computer. Technical Report AI Memo No. 1666, MIT A.I. Laboratory (1999) 7. He, T., et al.: Range-free localization and its impact on large scale sensor networks. Transactions on Embedded Computing Systems 4(4), 877–906 (2005) 8. Elnahrawy, E., Li, X., Martin, R.P.: Using Area-Based Presentations and Metrics for Localization Systems in Wireless LANs. In: Proceedings of the 29th IEEE International Conference on Local Computer Networks (LCN 2004), Washington, USA (2004) 9. Parkinson, B.W., Spilker Jr., J.J.: Global Positioning System: Theory and Applications, Vol. 1. In: Progress in Astronautics and Aeronautics, American Institute of Aeronautics and Astronautics, vol. 163, pp. 29–56 (1996) 10. Harter, A., Hopper, A., Steggles, P., Ward, A., Webster, P.: The Anatomy of a Context-Aware Application. Mobile Computing and Networking, 59–68 (1999) 11. Niculescu, D., Nath, B.: Ad hoc positioning system (APS). In: Proceedings of GLOBECOM, San Antonio (November 2001) 12. Bahl, P., Padmanabhan, V.N.: RADAR: An In-Building RF-Based User Location and Tracking System. In: INFOCOM, vol. (2), pp. 775–784 (2000) 13. Youssef, M., Agrawala, A., Shankar, U.: WLAN Location Determination via Clustering and Probability Distributions. In: Proceedings of IEEE PerCom 2003 (March 2003) 14. Ganesan, D., Krishnamachari, B., Woo, A., Culler, D., Estrin, D., Wicker, S.: Complex Behavior at Scale: An Experimental Study of Low-Power Wireless Sensor Networks. Technical Report CSD-TR 02-0013, UCLA (February 2002) 15. Want, R., Hopper, A., Falc˜ ao, V., Gibbons, J.: The Active Badge Location System. ACM Transactions on Information Systems (TOIS) 10(1) (1992) 16. Paperno, E., Sasada, I., Leonovich, E.: A New Method for Magnetic Position and Orientation Tracking. IEEE Transactions on Magnetics 37(4) (July 2001) 17. Krumm, J., Harris, S., Meyers, B., Brumitt, B., Hale, M., Shafer, S.: Multi-Camera Multi-Person Tracking for EasyLiving. In: Proceedings of the 3rd IEEE International Workshop on Visual Surveillance (VS 2000), Washington, USA (2000) 18. Ganu, S., Krishnakumar, A., Krishnan, P.: Infrastructure-based location estimation in WLAN networks. In: Proceedings of IEEE WCNC 2004 (March 2004) 19. Blumenthal, J., Grossmann, R., Golatowski, F., Timmermann, D.: Weighted Centroid Localization in WLAN-based Sensor Networks. In: Proceedings of the 2007 IEEE International Symposium on Intelligent Signal Processing (WISP 2007) (2007) 20. Cisco Systems, Aironet 1200 Series Access Point Data Sheet, http://www.cisco.com/en/US/products/hw/wireless/ps430

A Formal Framework for Expressing Trust Negotiation in the Ubiquitous Computing Environment* Deqing Zou1, Jong Hyuk Park2, Laurence Tianruo Yang 3, Zhensong Liao1, and Tai-hoon Kim4 1 Services Computing Technology and System Lab, Cluster and Grid Computing Lab, School of Computer Science and Technology, Huazhong University of Science and Technology, China 2 Department of Computer Science and Engineering, Kyungnam University, Korea 3 St. Francis Xavier University, Canada 4 Division of Multimedia, Hannam University, Korea

Abstract. There are lots of entities in the ubiquitous computing environment. For the traditional public key Infrastructure (PKI), every entity should be signed a valid certificate by the certificate authentication center. However, it’s hard to construct a centralized trust management framework and assign a valid certificate for every entity in the ubiquitous computing environment because of large numbers of dynamic entities. Trust negotiation (TN) is an important means to establish trust between strangers in ubiquitous computing systems through the exchange of digital credentials and mobile access control policies specifying what combinations of credentials a stranger must submit. Current existing TN technologies, such as TrustBuilder and KeyNote, focused on how to solve a certain problem by using some special techniques. In this paper, we present a formal framework for expressing trust negotiation. The framework specifies the basic concepts, elements and the semantics of TN. By analyzing TN, we point out how to build a TN system in practice.

1 Introduction With the comprehensive popularity of computer knowledge and the great development of computer network, people have more and more requirements towards resource sharing over Internet. Common access control mechanisms, such as MAC [1], RBAC [2], could not work well because of their limitation in design and application. In the ubiquitous computing environment, it’s hard to construct a centralized trust management framework for large numbers of dynamic entities. Exchange of attribute credentials is a means to establish mutual trust relationship between strangers in the ubiquitous computing environment, who wish to share resources or conduct business transactions [3][4][5]. As a new access control mechanism, TN successfully provides a method to share resources for strangers from different security domains in the ubiquitous computing environment. *

The project is supported by National Natural Science Foundation of China under Grant No. 60503040.

F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 35–45, 2008. © Springer-Verlag Berlin Heidelberg 2008

36

D. Zou et al.

In TN, automated trust negotiation (ATN), as a new kind of TN, has a great improvement towards previous TN. Generally, ATN differs from TN as the following aspects: 1) ATN discloses the access control policy when the resource can be available. While in TN, the requestor would not know what credentials he must submit before the negotiation starts. From this, we can see that ATN saves much time, which helps to improve negotiation efficiency. 2) ATN has done much work in privacy protection. In TN, the requestor is required to submit his certificate to attain the access. In ATN, the trust relationship is established by the exchange of credentials and access control policies. The disclosure of a certificate may pose a threat towards the user private privacy, while the disclosure of a credential would not do. 3) ATN pays much attention on information protection, while TN does not. In ATN, the transmitting messages are encrypted, while in most TN systems, the exchanged data are plaintexts. Currently, there are still some problems unsolved in TN. First, there are no unified secure definitions for TN’s concepts. For example, what is a credential? What information a credential must include? Which credential is secure? Second, TN is difficult to be formalized by a certain language, which leads to a low security level. Third, since most information protection techniques are immature or heavyweight, how to protect privacy and sensitive information is still a serious issue. Finally, there is no guideline to instruct how to build a TN system. In the paper, we aim at expressing TN theoretically, specifying the internal relationship between TN’s concepts or components. Generally, our work has the following contributions: 1) A common framework of general-purpose for TN is proposed in this paper. Previous work focused on how to solve a certain problem, such as giving a model, using some special techniques and so on. Our work aims at the unification of all kinds of models and techniques. 2) We propose a lot of useful suggestions on how to build a TN system. Usually, before developing a TN system, some preparations should be made. For example, which policy language will be selected? How should we improve negotiation efficiency? Previous work did not deal with such problems. 3) We present a new thought to improve TN’s security level. According to the specification of TCSEC (Trusted Computer System Evaluation Criteria) [6], TN has a low security level. If TN is formalized by a certain policy language, TN’s security level would be improved or enhanced. The rest of this paper is organized as follows. Section 2 describes the related work, including TN project survey and information protection techniques. Section 3 is the main part of the paper. It describes the framework in detail. Section 4 purposes useful suggestions on how to build/develop a TN system. Section 5 concludes the paper and specifies the future work.

A Formal Framework for Expressing Trust Negotiation

37

2 Related Work Our work is originally motivated from the existing automated trust negotiation research [3][4][5] whose goal is to enable trust establishment between strangers in a decentralized or open environment, such as Internet. Our work focuses on analyzing TN, including the concepts, the components and the relationship between the concepts. To make it clear, existing TN projects and systems should be collected. Meanwhile, how to protect sensitive information or personal privacy is still a hot issue. Based on these, in this part, we mainly investigate TN projects/systems and information protection. In the existing TN systems, TrustBuilder [7][8], KeyNote [7][9][10] are much well-known. Some similar systems include PolicyMaker [11], SD3 [12], PRUNES [13] and Trust-X [14]. Here, we mainly discuss TrustBuilder and KeyNote. TrustBuilder represents one of the most signification proposals in the negotiation research area. It provides a broad class of negotiation strategies, as well as a strategyand language-independent negotiation protocol that ensures the interoperability of the defined strategies within the its architecture. Each participant in a negotiation has an associated security agent that manages the negotiation. During a negotiation, the security agent uses a local negotiation strategy to determine which local resources to disclose next and to accept new disclosures from other parties. In KeyNote, the predicates are written in a simple notation based on C-like expressions and regular expressions, and the assertions always return a Boolean (accepted or denied) answer. Moreover, the credential signature verification is built in to KeyNote system. Assertion syntax is based on a human-readable “RFC-822”-style syntax. And the trusted actions in KeyNote are described by simple attribute/value pairs. As far as the information protection is concerned, some typical and influencing techniques are involved as the following. W.H. Winsborough and N. Li [4] presented ACK policy to control the disclosure of credentials and policies, and developed TTG protocol to construct policy tree so that it was easy to detect whether the credentials matched the policies or not. ACK policy is useful to protect sensitive information, and TTG protocol enables two parties to do joint chain discovery in an interactive manner as well as the parties to use policies to protect sensitive credentials and the attribute information contained in the credentials. However, ACK policy and TTG are limited in applying because of difficulty in constructing them. N. Li et al [15] proposed OSBE protocol to prevent important information from leaking and void the negotiation being attacked. OSBE bases its idea on digital signature and checks the message’s integrity so as to detect whether the negotiator has the right signature. OSBE protects message from unauthorized access, but it is heavyweight to build and the signature computing has a great cost. Holt et al introduced hidden credentials in [16]. They gave a formal description for hidden credentials, including the concepts of credential and policy indistinguishability, and showed how to build them using IBE. Their work also gave some compelling examples of the utility of hidden credentials. In short, they provided a good model for trust negotiation to implement hidden credentials. Based on this, Robert W et al [17] used hidden credentials to conceal complex policies, and Keith F et al [18] used them

38

D. Zou et al.

to hide access control policies. Since hidden credential system cannot prevent from invalid inference, its implementation is limited in some degree. J. Li and N. Li proposed the notions of OACerts and OCBE in [19]. OCBE protocol adopted the idea of zero-knowledge and ensured that if and only if the recipient was the specified one, he could get the resource, otherwise he got nothing. However, the method has not discussed how to guarantee the security of message during the transmission over the insecure Internet.

3 Framework for Expressing Trust Negotiation In this section, we will describe the TN framework in detail. First, the basic concepts and elements are listed. Then, the framework is introduced. Finally, the corresponding expressions are given. 3.1 Basic Concepts and Elements TN is performed by the exchange of credentials and access control policy. In this part, we give the basic concepts of TN and put forward their corresponding formalized description as follows: • Access control policy (ACP): the constraint in accessing the resource, which indicates what credentials the accessor should prepare and submit. In TN, access control policy has two types: compound policy and meta policy. Meta policy is the minimal element to compose a compound policy. To realize it, a set of conjunctive predicates are required, such as , < ∧,∨,¬> and so on. • Access request (AR): a structural message to tell the resource provider that the resource is expected to be accessed. An access request should at least include: 1) accessor, 2) requested resource, 3) requested operation. • Certificate: to identify a person’s identity or attribute information, such as principal, public key. In previous TN, certificate is the main tool to carry a person’s information. Currently, as a person’s privacy is considered, certificate is hardly disclosed and transferred any more. • Credential: a kind of temporary certificate, whose content comes from the user’s certificate. In other words, a certificate contains many credentials. When the credential is introduced in TN, user’s certificate would not be released. • Negotiation strategy (NS): a mechanism to specify how to disclose access control policy and credentials. There are two extreme negotiation strategies: eager strategy and parsimonious strategy. The former discloses zealously without being requested, while the latter would not release any until a request is received. • Negotiation protocol (NP): a rule to specify what actions would be taken during the negotiation process, including authentication, delegation, authorization communication mode (TCP/UDP), encryption mode (symmetrical or asymmetrical), and encryption function and so on. For example, is transmitting message transferred via plaintext or ciphertext? How to select the encryption functions?


39

Accordingly, the basic elements of TN are as follows: • ACP: access control policy set. ACP= {acp1,acp2,…,acpn}, acpi (1≤i≤n) is a meta policy. Usually, an access control policy ACP can be depicted as ACP=f(acp1,acp2,…,acpn), while f is compound function by using the specified conjunctive predicates. • AR: access request set. AR={ar1,ar2,…,arn}, ari (1≤i≤n) is a specific access request towards a certain resource. • C: credential set, C={c1,c2,…,cn}, ci (1≤i≤n) is a credential. Every credential carries only one kind of attribute information. • R: resource which can be accessed. In TN, R stands for the available resource. It can be the resource itself, or the access entry. When R is disclosed, the trust relationship is established, and the negotiation process ends up. • SEQ: disclosure sequence set, which is used to store the disclosed access requests, access control policies and credentials. SEQ=SEQAR∪SEQACP∪ SEQC∪R, while SEQAR (SEQACP, SEQC) denotes the disclosed access request (resp. access control policy, credential) sequence. When R∈SEQ, the negotiation succeeds. • NS: negotiation strategy set. NS={ns1,ns2,…,nsn}, nsi (1≤i≤n) is the selected negotiation strategy. In practice, new negotiation strategies may be developed according to the application. • NP: negotiation protocol set. NP={np1, np2,…,npn}, npi (1≤i≤n) is a certain negotiation protocol to specify how to exchange information during the negotiation process.

Fig. 1. The relationship between AR and ACP

3.2 Framework for Expressing Trust Negotiation In our framework, TN can be described as follows: TNparameter=

(1)

In (1), TNparameter denotes the parameter to deploy a TN system. When TNparameter is confirmed, the work mode of TN is established. ATN/TN stands for the negotiation

40

D. Zou et al.

type. When ATN is selected, it is an ATN system. AR (ACP, C, NS, NP) is the corresponding access request (resp. access control policy, credential, negotiation strategy, negotiation protocol) set. There are many internal relationships and constraints among TN’s parameters. 1) The relationship between AR and ACP In TN, when the resource provider receives an access request ar, he will analyze it and check whether it is valid and effect. If ar passes the examination, the resource provider will return a relative access control policy set Policy (including one or more meta policies) as a response, otherwise, ar is denied. Generally, every request should have a corresponding policy set, i.e., ∀ar∈AR,∃Policy⊆ACP⇒Correlate(ar,Policy), in which the function Correlate() is a binary logic predicate. Meanwhile, it may happen multiple requests correspond to a same policy set. This is caused by the access granularity. In short, the relationship between AR and ACP can be shown as Fig. 1.

Fig. 2. The relationship between ACP and C

2) The relationship between ACP and C In TN, credentials are submitted to meet the requirements of a compound policy. To avoid the different meanings, each credential can only satisfy a meta policy. That is to say, ∀acp∈ACP, ∃c∈C⇒Match(acp,c) and ∀c∈C,∃acp∈ACP⇒ Match(acp,c), in which the function Match() is also a binary logic predicate. The relationship between ACP and C can be shown by Fig. 2. 3) The relationship between NS and NP In TN, negotiation strategy decides the disclosure mode. Different negotiation strategy has different requirements towards system, network, user, resource provider and negotiation protocol. For example, if the parsimonious strategy is adopted, i.e., the negotiation needs a tight coupling communication, which causes a great network burden, and has a strict requirement towards the negotiation protocol. Accordingly, negotiation protocol decides the negotiation process. Usually, negotiation protocol and negotiation strategy should cooperate with each other; otherwise, the negotiation is easy to fail. The relationship between NS and NP is shown as Fig. 3.


41

Fig. 3. The relationship between NS and NP

3.3 Trust Negotiation Expression From above, we can see that the state of TN can be depicted as: STN = SX + SY

(2)

SX = NS × NP

(3)

SY = AR × ACP × C

(4)

SX denotes a relatively static state of TN. When the negotiation starts, NS and NP would be ascertained. When SX is fixed, the disclosure sequence SEQ would be somehow clear. In short, SX decides the qualitative change of SEQ. SY stands for a relatively dynamic state of TN. SY records the total negotiation process, which can be used to track both negotiators’ actions. Meanwhile, it is the important information to compose the log file. If the negotiation fails, SY can be useful to produce feedback. In short, SY decides the quantitative change of SEQ. As negotiation protocol is taken into account, NP can be depicted as: NP=

(5)

In (5), A1 is authentication. In TN, authentication is an important means to make certain a user’s identity. Usually, authenticating process is to ensure that the acceesor holds the private key of his certificate. Suppose p and l to be a user’s principal (also called public key) and license (also called private key). The authenticating process can be: 1) the server randomly picks a string str, and encrypts it with the user’s principal p and passes it to the user; 2) the user decrypts it and returns it through a secure VPN (virtual private network) channel; 3) the server compares the original string with the received string, and decides the authentication result. Since it would take a great cost to set up a VPN channel, authentication can be realized by mutual encryption via insecure channel such as Internet without VPN. Based on this, we present a mutual authentication shown as Fig. 4.

42

D. Zou et al.

Fig. 4. Mutual authentication for TN

In short, authentication can be depicted as: A1= Principal × License

(6)

In (5), D is delegation. In TN, delegation is to endow others with one’s privileges. To finish a delegation, two users (suppose A and B) should do: 1) the authentication of A and B; 2) the delegation of B towards A, to let A own B’s privileges. Based on these, the delegation can be depicted as: DBA = A1A × A1B × PrivilegeB

(7)

In (5), A2 is authorization. In TN, authorization is to let the accessor have the specified privileges to operate on resource. In short, authorization can be depicted as: A2= Principal × License × PrivilegeR

(8)

In (5), PL/CI denotes the message encapsulation means. PL is plaintext, while CI is ciphertext. TCP/UDP denotes the communication means. TCP stands for tight coupling communication. When NS is parsimonious strategy, TCP must be selected. UDP stands for loose coupling communication. S/A denotes the encryption means. S stands for symmetric encryption, while A stands for asymmetric encryption. When S is selected, the relative algorithms could be AES [20], Rijndael [21] and so on. Corresponding, when A is selected, the relative algorithms could be RSA [22], ECC [23] etc.

4 Guideline for Building a TN System The research in theory is useful to guide teams to build practical systems. Before building a TN system, some questions should be answered.


43

1) Application type The team should make clear what services the TN system can provide. How about the security requirements? Would the online service be high available or have high computing power? How to ensure QoS? When these problems are clear, the team can make good preparations for the development. 2) Policy language Not every language can serve for TN. TN has a lot of requirements [3][7] towards policy languages. The selection of policy language should be adapted to application type. 3) Specification of AR, ACP and C To attain the goal of high efficiency and least artificial participation, TN should be much intelligent as possible. To realize it, TN should improve the readability of access request, access control policy and credential. So before developing TN, the formats of AR, ACP and C should be ascertained. This helps to perform all kinds of operations. An example for the formats of AR, ACP and C can be: AR=(owner):(resource):(operation): (period)

(9)

ACP=(owner):(recipient):(item):(predicate): (value):(period)

(10)

C=(owner):(recipient):(item):(value):(period)

(11)

In (9), owner denotes the holder of AR, usually the principal of the user; resource denotes requested resource or service; operation denotes the requested operation towards the resource; while period denotes the valid life time of AR. In (10), owner denotes the holder of ACP; recipient denotes the receiver of ACP; item denotes the attribute content, such as role, age, name etc; predicate denotes the comparing symbol, predicate∈ {>, <provide interface=”communication.ChatClient” /> <deployment> <require name=”resources1” class=”descriptor.owl#MemoryDependency” /> <require name=”software1” class=”descriptor.owl#JavaVMDependency” /> <require name=”hardware1” class=”descriptor.owl#DisplayDependency” /> <require name=”hardware2” class=”descriptor.owl#KeyboardDependency” />

It models the above dependencies as class restrictions on concepts defined in our context ontologies1. A device should comply with these constraints if the service is to be deployed successfully. For example: = 98304” /> 1

See http://www.cs.kuleuven.be/∼ davy/ontologies/2008/01/ for the latest revision of our context ontologies.

Pervasive Services on the Move

51

The previous constraint declares that a device should have at least one memory resource instance (of class RAM or one of its subclasses) with a property currAvailable that has a value larger than or equal to 98304. The type of the property is specified in the Hardware.owl ontology, and for this value it is bytes.

The above constraint declares that a device should have at least one Java virtual machine instance with a GUI rendering engine instance that belongs to the JavaAWT class. The main advantage of our context ontologies [9] is that complex semantic relationships only need to be specified once and can be reused by every service descriptor. Moreover, each device profile can specify their characteristics with the same ontologies. If a device then would specify that it runs OSGi on top of JDK 1.6, then an AWT rendering engine dependency would semantically match, but a Java MIDP 2.0 LCDUI rendering engine would not. Resource and hardware constraints can be expressed in a similar way. The matching itself to verify that a service can move to a particular host is carried out by a context enabling service that is described in the following section. 3.2

Context-Awareness as an OSGi Distributed Declarative Service

In order to diffuse OSGi services in a way that takes into account the heterogeneity and dynamism of a ubiquitous computing environment, we need to be aware of what the characteristics and capabilities of these devices are, where they are located, what kind of services they offer and what resources are currently available. The COGITO service will provide this information on each host. It gathers and utilizes context information to positively affect the provisioning of services, such as the personalization and redeployment of services tailored to the customer’s needs. The core functions of the enabling service are provided as a set of OSGi bundles that can be subdivided into the following categories: – Context Acquisition: These bundles monitor for context that is changing and gather information from resource sensors, device profiles and other repositories within the system itself or on the network. – Context Storage: A context repository ensures persistence of context information. It collects relevant local information and context that remote entities have published in a way that processing can be handled efficiently without losing the semantics of the data.

52

D. Preuveneers and Y. Berbers

Fig. 2. Building blocks of the COGITO enabling service

– Context Manipulation: These bundles reason on the context information to verify that context constraints are met. Besides a rule and matching engine that exploits semantic relationships between concepts [11], it also provides adapters that transform context information into more suitable formats. A high-level overview of the building blocks of the context-awareness enabling service is given Figure 2. The advantage of decomposing the context-awareness framework into multiple OSGi bundles is that the modular approach saves resources when certain bundles are not required. COGITO is implemented as a distributed Declarative Service. As the Declarative Service specification does not cover the registration of remote services, we have each device announcing its presence and that of its context enabling service with a service discovery protocol like UPnP, SLP or ZeroConf. Upon joining a network, each other node in the network creates a remote proxy to the enabling service of the joining node, and the Declarative Services bundle will ensure lazy registration and activation of the proxy whenever remote context is acquired. When a node leaves the network, the proxies on the other nodes are deactivated. This approach for distributed Declarative Services simplifies the collection of context information on the network, but more importantly it also enables transparent sharing of intensive context processing services (such as a rule or inference engine bundle) on resource-constrained devices. 3.3

Service State Representation, Extraction and Synchronization

In order to replicate stateful services on multiple hosts with support for state synchronization, we need to explicitly model the state variables of the service, extract them at runtime and send them to the replicated peers. In order to keep


53

state representation and exchange straightforward and lightweight, we encapsulate the state of a service in a JavaBean. JavaBeans have the benefit of (1) being able to encapsulate several objects in one bean that can be passed around to other nodes, (2) being serializable and (3) providing get and set methods to automate the inspection and updating of the service state. The approach is quite similar to the stateful session beans in the Enterprise JavaBeans specification [12]. Stateful session beans maintain the internal state of web applications and ensure that during a session a client will invoke method calls against the same bean in the remote container. In our approach however, a JavaBean is situated locally within each replicated service and does not involve remote method calls. Instead, the contents of the JavaBean is synchronized with those of the replicated services. Our current implementation tries to synchronize as soon as the state of an application has been changed. When network failures arise, the state updates are put in a queue and another attempt is carried out later on. The queue holds the revision number of the last state that was successfully exchanged and this for the three replicated applications. If, however, two or more replicated services independently continue without state synchronization, a state transfer conflict will occur when the involved hosts connect again. In that case, a warning will be shown and the user can choose to treat the services as separate applications, or have one service push its state to the others. This approach is rather crude, but acceptable as long as we are mainly dealing with single-user applications.

! " # $ !!

$

" %

!!

Fig. 3. New service states in the life-cycle of a migrating or diffusing service. Some of the transitions of the original service are shown in green, the ones of the replicated service(s) in red.

54

3.4


Extending the Life-Cycle Management to Relocate Services

In a ubiquitous computing environment where pervasive services are replicated, a user may switch from one replicated service to another. Therefore, we add extra information to the service state that helps to coordinate the synchronization of the state changes. For example, we add a revision number that is increased after each write operation on the bean and use Vector Clocks [13] among all peers to order the state updating events. Fig. 3 shows that an active service can not only be stopped, but that in another transition the state can be extracted from the service and synchronized, or that the service can continue to either migrate or replicate to a new host. We use a lease timeout algorithm to garbage collect stale replicated services in order to recycle resources. The timeouts are application dependent, but can be reconfigured by the user.

4

Experimental Evaluation

The usefulness and feasibility of dealing with context-aware service migration and diffusion will be illustrated with three applications: (1) a grocery list application, (2) a Jabber instant messaging client, and (3) a Sudoku game. The devices used in the experiment include two PDAs, a laptop and a desktop, all connected to a local WiFi network. This setup will simulate the real-life scenario in the next section. 4.1

Scenario

The scenario goes as follows: It is Saturday morning and mom and dad go shopping. Everybody at home uses the grocery list application to make up a shared grocery list. In the past, mom sometimes forgot her grocery list at home, but now she only needs to take along her smartphone. The application just diffuses along. Dad also has his PDA with him and runs the same replicated application. They decided to split up the work and go shopping separately. At the mall, each time mom or dad finds a product that they need, it is removed from the shared sticky note application. They get each others updates when the state of the application is synchronized. Mom is still checking out while dad is already at the car. As he is waiting, he has another go at the Sudoku puzzle that he was trying to solve on the home computer that morning. His daughter is online and starts a conversation. She wants to send him a picture, but the display of his PDA is too small. He takes his briefcase, turns on his laptop, the application migrates and he continues the conversation there. Unfortunately, he forgot to charge his battery so after a while his laptop gives up on him. “Back to the PDA but then without the picture”, dad decides. In the meantime, mom has arrived and dad tells her about the picture. She will have a look at it at home once the application synchronizes with the desktop computer.


55

Fig. 4. The grocery list is automatically adapted to fit the new screen size if needed. After redeployment the states of all the replicated grocery lists are synchronized.

4.2

Experiments

In the experiment all the applications are launched on the desktop PC. The Jabber instant messaging client and Sudoku puzzle diffuse to one handheld, while the grocery list application diffuses to both PDAs. The application states are continuously synchronized between all hosts. Later on, the instant messaging application migrates from one PDA to the laptop and back (while synchronizing with the desktop PC). The mobility of the grocery list is illustrated in Figure 4. Additionally, these three applications all make use of two extra general-purpose services: (i) a stateless audio-based notification service, and (ii) a stateful logging service. By only deploying them on the desktop computer and synchronizing them on the laptop, we enforce that these two services can only be reached from a PDA through a remote connection. The applications will invoke them when (1) the grocery list changes, (2) the online status of somebody changes or (3) when the puzzle is solved. The bindings to the remote services are used to test handover to a replicated service after network disruptions. We let the setup run for 30 minutes, in which the following actions are simulated: – Each 30 seconds the grocery list is modified and synchronized. This event also triggers an invocation to both the remote services. – Each 5 seconds the state of the Jabber client is synchronized. Each minute the online status of somebody in the contact list changes, and this triggers an invocation to the remote services. – The Sudoku puzzle changes each 15 seconds and the 81 digits of the puzzle are synchronized. A puzzle is solved after 10 minutes, and this initiates an invocation to the remote services.

56

D. Preuveneers and Y. Berbers Table 1. Overhead results of state transfer, synchronization and handover Grocery List

Jabber Client

Sudoku Puzzle

Bundle size

23023 bytes

288235 bytes

54326 bytes

State size

7024 bytes

18235 bytes

1856 bytes

State synchronization

53 Kbytes / min 112 Kbytes / min 37 Kbytes / min

Relocation time

9 seconds

13 seconds

7 seconds

Synchronization delay

350 msec

518 msec

152 msec

Handover delay (audio service) 47 msec

61 msec

54 msec

Handover delay (log service)

213 msec

78 msec

167 msec

Each 3 minutes we shut down a random network connection for 1 minute to verify if the replicated applications can recover from the missed state updates and that handover to one of the remote replicated services. We measured the overhead of state transfer and synchronization and compare that to the resources that the applications required. We also measured the service availability and responsiveness by measuring the delay of handing over to another remote replicated service after a network disconnection. The experiment was repeated 3 times and the averaged results are shown in Table 1. 4.3

Discussion of the Results

The test results show there is only a small overhead for the state transfer. This is mainly a consequence of the size of the applications. They are rather small as they would otherwise not run on mobile devices. For other data intensive applications, such as multimedia applications, we expect that the overhead will be a lot bigger if data files have to be moved along as part of the state of the service. As the overhead was limited, we did not implement incremental state transfers, but for media applications this could be an improvement. The delays in state synchronization are low because the experiments were carried out on a local WiFi network. Other tests in multi-hop networks showed longer but still acceptable state synchronization delays (at most 5 times longer). More interesting though is the difference in service relocation time and state exchange time. By proactively moving the service in advance, the time to get an up-to-date replica is a lot smaller for state synchronization compared to service relocation. As such, service diffusion with state synchronization provides a much better usability to the user compared to plain service migration. Of course, this assumes the required bandwidth is available. The handover to the remote services worked in all cases because we avoided the case where all network connections to the remote replicated services are disrupted. In theory, our framework can handle this particular case if the remote service calls can be buffered and invoked asynchronously. However, our current


57

implementation does not support this. Interesting to note for the handover to the replicated services (in our experiment, the stateless audio service and stateful log service), is that handover to another log service takes a bit longer on average. This result is due to the fact that for the audio service we do not need to wait until the replicated service has acquired the latest revision of the service state. If state synchronization is taking place during handover, the handover delay can become higher. In our experiment, the remote services were only available on two hosts (the laptop and the desktop). If more devices would host a replicated service, the decision to handover to a particular device could depend on which replica is already completely synchronized.

5

Related Work

In recent years, many researchers have addressed the issue of state transfer, code mobility and service oriented computing in ubiquitous computing environments. This research has greatly improved our knowledge of how context-aware service oriented architectures can accommodate to new functionality requirements in changing environments. Providing a detailed review of the state of the art on all of these domains is beyond the scope of this paper. Instead, we focus on those contributions that are most related to the work presented in this paper. In [14], the OSGi platform was used to perform context discovery, acquisition, and interpretation and use an OWL ontology-based context model that leverages Semantic Web technology. This paper has inspired us to use the OSGi platform for context-aware service orientation, and more specifically service mobility. In [15], the authors present an OSGi infrastructure for context-aware multimedia services in a smart home environment. Although the application domain that we target goes beyond multimedia and smart home environments, we envision that more data driven services like multimedia applications are good candidates to further evaluate our service mobility strategies. Other context-awareness systems leveraging the OSGi framework include [16,17]. Optimistic replication is a technique to enable a higher availability and performance in distributed data sharing systems. The advantage of optimistic replication is that it allows replicas to diverge. Although users can observe this divergence, the copies will eventually converge during a reconciliation phase. In [18], the authors provide a comprehensive survey of techniques developed to address divergence. Our current service replication and state synchronization is more related to traditional pessimistic replication, because user perceived divergence might cause confusion. As discussed in [19], we will further investigate how optimistic replication can improve the quality of service in mobile computing, while keeping the user perceived divergence at an acceptable level. Remote service invocation on the OSGi framework is an issue that has been addressed by several researchers and projects. The Eclipse Communication Framework [20] provides a flexible approach to deal with remote services and service discovery on top of the Equinox OSGi implementation. Remote OSGi

58


services can be called synchronously and asynchronously on top of various communication protocols. The R-OSGI [21] framework also deals with service distribution. While both projects address network transparency for OSGi services, neither of them deals with explicit service mobility. The Fluid Computing [22] and flowSGI [23] projects explicitly deal with state transfer and synchronization in a similar way to ours. They also discuss state synchronization and provide a more advanced state conflict resolution mechanism. Our contribution builds upon this approach and specifically focuses on service oriented applications. Our method also enhances the peer selection for service replication by using context information of the service and the device. This approach provides better guarantees that the application will actually work on the device.

6

Conclusions and Future Work

This paper presents a context-driven approach to live service mobility in pervasive computing environments. It focuses on service migration and diffusion to multiple hosts to increase accessibility and expedite human interaction with the service. We summarized the basic requirements for service mobility, including enhanced service descriptions, context-awareness to select appropriate targets for service migration, and life-cycle management support to carry out service relocation with state transfer and state synchronization. We have discussed how we implemented these requirements on top of the OSGi framework and validated our prototype by means of several applications that are replicated in a small scale network with enforced network failures. We studied the effects of service migration and service diffusion. Service migration moves an application from one host to another including its state, while service diffusion replicates the service and the state on multiple hosts. State synchronization ensures that each service can be used interchangeably. Experiments have shown that the overhead of state transfer and synchronization is limited for relatively small applications. The results illustrate that, if the available network bandwidth permits, the time to keep the state in sync is a lot smaller than to migrate a service from one host to another. This means that if a user wants to use another device because it provides a better quality of service (e.g. the bigger screen in the scenario), that proactive service diffusion provides a much better usability (i.e. shorter delays). Future work will focus on a better handling of state synchronization conflicts and improving the state encapsulation approach to deal with incremental state transfers for data intensive applications. However, our main concern will always be to keep the supporting infrastructure flexible and lightweight enough for lowend devices with room left to actually run applications. The outcome of this future work could result in a better policy with respect to on how many remote devices a service should be replicated and under which circumstances service migration would be a better approach compared to service diffusion.


59

References 1. Weiser, M.: The computer for the 21st century. Scientific American 265, 94–104 (1991) 2. Papazoglou, M.P., Georgakopoulos, D.: Service oriented computing. Commun. ACM 46, 24–28 (2003) 3. Shaw, M., Garlan, D.: Software architecture: perspectives on an emerging discipline. Prentice-Hall, Inc., Upper Saddle River (1996) 4. Dey, A.K.: Understanding and using context. Personal Ubiquitous Comput. 5, 4–7 (2001) 5. Open Services Gateway Initiative: OSGi Service Gateway Specification, Release 4.1 (2007) 6. Helal, S.: Standards for service discovery and delivery. Pervasive Computing, IEEE 1, 95–100 (2002) 7. Kramer, J., Magee, J.: The evolving philosophers problem: Dynamic change management. IEEE Trans. Softw. Eng. 16, 1293–1306 (1990) 8. Vandewoude, Y., Ebraert, P., Berbers, Y., D’Hondt, T.: An alternative to quiescence: Tranquility. In: ICSM 2006: Proceedings of the 22nd IEEE International Conference on Software Maintenance, Washington, DC, USA, pp. 73–82. IEEE Computer Society, Los Alamitos (2006) 9. Preuveneers, D., den Bergh, J.V., Wagelaar, D., Georges, A., Rigole, P., Clerckx, T., Berbers, Y., Coninx, K., Jonckers, V., Bosschere, K.D.: Towards an extensible context ontology for Ambient Intelligence. In: Markopoulos, P., Eggen, B., Aarts, E., Crowley, J.L. (eds.) EUSAI 2004. LNCS, vol. 3295, pp. 148–159. Springer, Heidelberg (2004) 10. Mokhtar, S.B., Preuveneers, D., Georgantas, N., Issarny, V., Berbers, Y.: EASY: Efficient SemAntic Service DiscoverY in Pervasive Computing Environments with QoS and Context Support. Journal Of System and Software 81, 785–808 (2008) 11. Preuveneers, D., Berbers, Y.: Encoding semantic awareness in resource-constrained devices. IEEE Intelligent Systems 23, 26–33 (2008) 12. Burke, B., Monson-Haefel, R.: Enterprise JavaBeans 3.0, 5th edn. O’Reilly Media, Inc, Sebastopol (2006) 13. Mattern, F.: Virtual time and global states of distributed systems. In: Parallel and Distributed Algorithms: proceedings of the International Workshop on Parallel and Distributed Algorithms 14. Gu, T., Pung, H.K., Zhang, D.Q.: Toward an osgi-based infrastructure for contextaware applications. Pervasive Computing, IEEE 3, 66–74 (2004) 15. Yu, Z., Zhou, X., Yu, Z., Zhang, D., Chin, C.Y.: An osgi-based infrastructure for context-aware multimedia services. Communications Magazine, IEEE 44, 136–142 (2006) 16. Lee, H., Park, J., Ko, E., Lee, J.: An agent-based context-aware system on handheld computers, 229–230 (2006) 17. Kim, J.H., Yae, S.S., Ramakrishna, R.S.: Context-aware application framework based on open service gateway 5, 200–204 (2001) 18. Saito, Y., Shapiro, M.: Optimistic replication. ACM Comput. Surv. 37, 42–81 (2005) 19. Kuenning, G.H., Bagrodia, R., Guy, R.G., Popek, G.J., Reiher, P.L., Wang, A.I.: Measuring the quality of service of optimistic replication. In: ECOOP Workshops, pp. 319–320 (1998)

60


20. The Eclipse Foundation: Eclipse Communication Framework (2007), http://www.eclipse.org/ecf/ 21. Rellermeyer, J.S., Alonso, G., Roscoe, T.: Building, deploying, and monitoring distributed applications with eclipse and r-osgi. In: Eclipse 2007: Proceedings of the 2007 OOPSLA workshop on eclipse technology eXchange, pp. 50–54. ACM, New York (2007) 22. Bourges-Waldegg, D., Duponchel, Y., Graf, M., Moser, M.: The fluid computing middleware: Bringing application fluidity to the mobile internet. In: SAINT 2005: Proceedings of the The 2005 Symposium on Applications and the Internet (SAINT 2005), Washington, DC, USA, pp. 54–63. IEEE Computer Society, Los Alamitos (2005) 23. Rellermeyer, J.S.: flowsgi: A framework for dynamic fluid applications. Master’s thesis, ETH Zurich (2006)

Robots in Smart Spaces - A Case Study of a u-Object Finder Prototype Tomomi Kawashima1, Jianhua Ma1, Bernady O. Apduhan2, Runhe Huang1, and Qun Jin3 1 Hosei University, Tokyo 184-8584, Japan [email protected], {jianhua, rhuang}@hosei.ac.jp 2 Kyushu Sangyo University, Fukuoka 813-8503, Japan [email protected] 3 Waseda University, Saitama 359-1192, Japan [email protected]

Abstract. A smart space is a physical spatial environment such as a room that can provide automatic responses according to the contextual information occurring in an environment. The context data is usually acquired via various devices which are installed somewhere in the environment and/or embedded into some special physical objects, called u-objects, which are capable of executing computation and/or communication. However, the devices used in current researches on smart space are either fixed or residing in real objects, which are incapable of moving by themselves. Likewise, more and more robots have been produced, but most of them are designed and developed based on the assumption that the space surrounding a robot is an ordinary one, i.e., a non-smart space. Therefore, it is necessary to study what additional services can be offered and what specific technical issues will be faced when adding robots to smart spaces. To identify the potential novel services and technology problems, this research is focused on a case study of a u-object finding service performed by a robot in a smart space. This paper presents the design and development of the system prototype robot which can communicate with other devices and can find a u-object with attached RFID tag in a smart room. Some preliminary experiments were conducted and the results verified the functionalities of the system.

1 Introduction The essential of ubiquitous computing (ubicomp), as emphasized by Mark Weiser, is “enhancing computer use by making many computers available throughout the physical environment, but making them effectively invisible to the user” [1, 2]. The synergy of ubiquitous computing and communications is the natural result of two fundamental technology trends: (1) the continuing miniaturization of electronic chips and electro-mechanical devices following Moore’s law, and (2) their interconnection via networks especially using wireless communications. Aside from the widespread use and popularity of wireless technology such as IEEE802.11 a/b/g [3] and Bluetooth [4], the WPAN (Wireless Personal Area Network) [5, 6] and high-speed wireless F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 61–74, 2008. © Springer-Verlag Berlin Heidelberg 2008

62

T. Kawashima et al.

networks using UWB (Ultra Wide Band) [7] will soon be available and will play important roles in short range ubiquitous communications. Due to the continuing miniaturization and availability of high-speed communications, many devices can be distributed to surrounding ambient environments and make the ordinary environments capable of computing. That is, a real spatial environment such as a room may become smart to take automatic responses according to the contextual information [8, 9] occurring in this environment. The context data is usually acquired via various devices which are installed somewhere in the environment and/or embedded into some special physical objects. Real physical objects are called u-objects [10], as opposed to virtual e-things, if they are attached, embedded or blended with computers, networks, sensors, actors, IC-tags and/or other devices [11]. The smart space/environment, as one of the major ubicomp areas, has recently received much attention, and various smart space systems have been developed to make smart homes, smart offices, smart classrooms, etc. [12, 13]. However, the devices used in the current smart space researches are either fixed or residing in real objects that cannot move on their own. On the other hand, robotics is an exciting field that has been getting a great deal of attention for decades. Lots of robots have been produced and deployed to various applications such as in games, entertainment, household chores, rescue, military, and manufacturing [14]. The robots can also be seen as some special kinds of smart u-objects with partial human-like or animal-like behaviors. The most special characteristic of robots are that they can move by themselves according to the received instructions or perceptual information. However, most of them are designed and developed based on the assumption that the space surrounding a robot is an ordinary one, i.e., a non-smart space [15, 16]. It is then interesting and worth investigating to know what new services can be offered by smart spaces in which robots exist, and what new technical issues arise when combining smart spaces and robots to make smart systems. Flipcast [17] showed a design of an agent-based platform for communications between robots and ubiquitous devices, but it was not based on a smart space and did not give any concrete examples. Generally, there is almost no systematic research on integrated systems with some combination of robots and smart spaces. One way to approach the systematic research would be to start making concrete and representative sample applications. Therefore, this research is focused on a case study of a u-object finding service done by a robot in a smart space. This paper presents the design and development of a system prototype to manage robots and u-objects attached with RFID tags in a smart room space, and for a robot to communicate with other devices and find a tagged uobject in the smart space. We have conducted some preliminary experiments and the result verifies and evaluates the aforementioned functionalities. In what follows, we first check the possible roles of robots in a smart space as well as the relevant technical problems in general, and then present the u-object finding robot prototype in detail in the remaining sections. After showing the prototype overview in Section 3, we explain how to manage robots and u-objects in Section 4, and how to control and communicate with the robots in Section 5. The experimental

Robots in Smart Spaces

63

environment and results in finding u-objects are described in Section 6. Finally, the conclusion and future work are addressed in the last section.

2 General Services and Issues of Smart Spaces with Robots This section will discuss in general the special roles of robots in a smart space and the related basic issues to be solved in order to provide the desired services of robots. A smart space that can offer automatic services mainly relies on the sensing devices to acquire contextual information, and actuation devices to take responsive actions. As mentioned earlier, these devices are usually installed in some parts of the physical environment; such as walls, or embedded in some physical objects such as books. Because such devices are fixed and don’t move by themselves, they can sense information or act only in specific locations/directions. Different from these devices, robots are capable of moving from one location to another in a space. In addition, many robots can carry sensors and/or actuators. Therefore, a robot can move to a place to sense information or take an action in that place. For example, a movable robot can monitor a place from any direction using its camera to record the condition of a house, office, classroom, shop, street, and so on. Another example is that a robot, under instruction by a virtual host/manager of the smart environment, may bring an object from somewhere in the environment. In general, additional service functions which can be done by robots in smart spaces fall into the following categories. Location-Varied Information Acquisition: When carrying sensors, a robot can get the requested information not only at a specified location, but also from some data distributed in a space. For example, a robot with a microphone can move around to locate a sound source, which is usually hard to precisely detect by fixed microphones. The temperature distribution in a room can also be easily gathered by a movable robot by carrying only a single temperature sensor. Otherwise, many temperature sensors have to be installed in different parts of a relatively large room. That is, robots can function as mobile sensors to collect information at any point in smart space where they can possibly go. Location-Dependent Responsive Action: According to the sensed information, a smart space can take responsive actions. Some reactions may be conducted best at some specific locations. For example, the sound-aware smart room we developed [18] can sense ambient sounds using a microphone array, and output a speech to one of the speakers installed in a fixed location in the room. It sends a be-quiet reminder speech to a speaker near the person who is talking/singing loudly who may disturb others in the same location. Having a robot carrying a speaker in the room, the robot can move near the person and then give the reminder. In general, a robot may conduct some action by moving to a place requested by a smart space. Location-Aware Interaction with Devices/u-Objects: When a robot works in a nonsmart environment, it has to sense and interact with the environment using multimedia data from cameras, microphones, and so on. Therefore, the sophisticated robots rely on complex intelligent technologies for multimedia processing and understanding,

64

T. Kawashima et al.

which are not yet perfect and are very costly. In a smart environment, however, there are various electronic devices which are either independent or embedded in some objects, i.e., being u-objects. Thus, robots will be able to interact with these devices or u-objects via wireless electronic communications, instead of using audiovisual communications. Through communications, a robot may interact with nearby u-objects to identify their locations, know what they are doing, etc. By means of interaction, the robots and devices/u-objects in a smart room can work collaboratively to carry out some tasks. In summary, robots can be applied to add flexible location-based service functions to smart spaces because of their capability to move. To realize the new functions, a robot must collaboratively work with a virtual space host/manager and other devices/u-objects in the space. Therefore, the two fundamental issues for robots to conduct collaborative work in a smart space are how to manage the robots, and how to enable robots to communicate with the space host and various devices/u-objects. A long-term solution to completely solve the management and communication issues would be a scalable framework or middleware capable of supporting many applications in different smart spaces with multiple robots and heterogeneous devices/u-objects. This preliminary study is focused on the necessary management and communications in a specific application, i.e., a prototype called u-object finder, a robot for finding u-object(s).

3 The u-Object Finder Overview It is a very common experience that sometimes we forget and could not immediately find some common items which we need badly, such as keys or books. This is more common to small children and elder people since they often forget what they have just seen or done. The u-object finder (UOF) is an application of the smart space in which a robot moves around to search for a lost object. Figure 1 shows the UOF conceptual environment used in our study and its development.

㫌㪄㪦㪹㫁㪼㪺㫋㩷㪝㫀㫅㪻㪼㫉

㪬㪦㪝㪄㪤㪸㫅㪸㪾㪼㫉㪩㪝㪠㪛㩷㪩㪼㪸㪻㪼㫉

㫌㪄㪦㪹㫁㪼㪺㫋㪩㪝㪠㪛㩷㪫㪸㪾㫊㪬㪦㪝㪄㪩㫆㪹㫆㫋

Fig. 1. The u-object finder environment


65

The environment is a typical room space equipped with computers, devices, and robots which are connected by wireless networks. Basic assumptions to the environment are as follows.

・・・・

・・

Every object to be searched must be attached or embedded with at least one device which is used to uniquely identify the object and communicate with other devices. In the present prototype, we attached a RFID tag to every searchable uobject, also called a tagged object. A single object may have multiple attached tags, which can be used to either identify different parts of the object, or to improve search efficiency. A RFID reader is installed somewhere at the room entrance to read the RFID tag of a u-object when the object passes through the entrance. It is used to detect what and when a u-object has been brought in/out of the room. There is one or more movable robots in the room which moves according to the commands and instructions from a room manager. Each robot may carry multisensors for various purposes. To find a tagged object, a RFID reader is carried by the robot in our system. A number of RFID tags are set at fixed positions on the room floor. These tags are used for the robot to precisely locate its position since all the position data has been measured and kept in some room database. Usually a robot can remember the route it has passed, which can be used to estimate the robot’s present position. The electric-magnetic signal strength received by the robot from wireless routers can be also used to estimate its position, but this exhibits some errors in estimating the robot’s position. Thus, the position-fixed RFID tags on the floor can be read when the robot moves, and exploited to reduce or correct the errors. All machines and devices including robots are able to conduct wireless communications via either some specific communication scheme or a common wireless LAN router. There exists a virtual manager, i.e. an application residing on a PC, to manage all devices, u-objects, RFID tags and robots. It also provides an interface for a user to request a robot to find an object and receive the search result via our present UOF prototype.

The u-Object Finder system, as shown in Fig. 2, is mainly composed of the uObject Finder Manager (UOF-Manager) that functions as a central administrator of all devices including robots, and the u-Object Finder Robot (UOF-Robot) that is mainly in charge of searching u-objects in the room. The UOF-Manager has two main components: the u-Object Manager (O-Manager) to manage u-objects, and the Robot Manager (R-Manager) to manage robots. On the other hand, the UOF-Robot has two main components: the Robot Controller (RController) and the Robot Machine (R-Machine). The R-Machine is some kind of robot that has a motor, wheels, an ultrasonic sensor, etc., and thus movable by itself. When the robot moves near a barricade or blocks, it can turn and change direction using the data from the ultrasonic sensor. The movement direction, destination and route will be determined according to the robot’s task, current position, surrounding

66

T. Kawashima et al.

UOF- Manager R- Manager O- Manager

UOF- Robot IEEE802.11b

Bluet oot h

Radio Frequency

u- Object

R- Controller (PDA)

Radio Frequency

R- Machine (Robot)

Fig. 2. The u-object finder components

situations, and so on. The R-Controller functions as a mediator between R-Manager and R-Machine, and runs on a PDA that is carried by the robot. The PDA provides a popular programming environment and adopts the general TCP/IP communication protocol on top of common wireless LANs. A RFID reader and other wired sensors can be also connected directly to this PDA. Therefore, the PDA can greatly extend and enhance the functions of a low-cost robot. The PC, PDA, robot and devices are connected via several networks including IEEE 802.11b LAN, Bluetooth, and radio frequency communication between an RFID reader and an RFID tag. The general procedure to search a u-object is conducted in the following steps. At first, a user informs the UOF-Manager what u-object to search. The UOF-Manager will find out if the object is in the room by checking the object’s existence status kept in the u-object database. If it is in the room, the UOF manager will check if any UOFRobot is available in the room to do the search job. When a robot is available, the UOF-Manager will send an object finding command and the RFID tag number(s) of the u-object to the UOF-Robot. Then, the robot moves around and reads the nearby RFID tags. Once the tag number of the searched object is detected, the UOF-Robot will stop and send the UOF-Manager a search success message. At the same time, the robot will give a speech or a beeping sound to notify the user of its location, which is near to the found u-object. If the u-object cannot be found after a certain time, the UOF-Robot will send an unsuccessful message to the UOF-Manager. The next two sections will explain the UOF-Manager and UOF-Robot in details, respectively.

4 UOF-Manager The UOF-Manager is an application working on a PC. A user can operate the UOFRobot and manage the u-Object respectively using the UOF-Manager. The UOFManager consists of a UOF-GUI, an R-Manager (Robot-Manager) and an O-Manager (u-Object Manager), as shown in Fig. 3. The user can register and delete a u-Object by simply holding it over an RFID reader. The data of the registered u-Object is saved in the u-Object database by the O-Manager.


67

UOF- GUI

R- Manager

Operation Retrieval Connection Module Module Module

UFCP Stack

O- Manager

Registration Status Module Module

u- Object Database

Fig. 3. The UOF-Manager component

4.1 UFCP (U-object Finder Control Protocol) To manage robots and let them work in a smart space, the R-Manager and robots must be first enabled to communicate with each other. One objective of this study is to develop the UFCP stack based on TCP/IP, which will run the R-Manager, RController and R-Machine. It is a protocol for the UOF-Manager to retrieve and operate the UOF-Robot via message exchanges. The UFCP messages are in two forms: UFCP-TCP and UFCP-UDP. The UFCP-UDP is used to retrieve the UOF-Robots in the same network segment, while the UFCP-TCP is used to operate the UOF-robots. The format of the UFCP packet takes on ASCII. The UFCP packet has two fields: command and data, separated by a comma. Table1 shows the message format of UFCP-TCP and UFCP-UDP. 4.2 R-Manager The R-Manager has the UFCP Client which can search a UOF-Robot, make a connection, start and stop the UOF-Robot, change the threshold value of devices (ultrasonic sensor, motor), and get the status of the UOF-Robot. Through the UOF-GUI, a user can conduct the following management functions.

・ Retrieving the UOF-Robot ・

When receiving a robot retrieval demand from a user, the U-Manager will broadcast a UFCP robot search message over connected wireless LANs. The UOFRobot receiving the message will reply an acknowledgment message with its name and current position. The list of retrieved UOF-Robot is shown in UOFGUI. Connection of the UOF-Robot After the user selected one UOF-Robot from the retrieved UOF-Robot list, the user can make a connection using UFCP to the selected UOF-Robot. After the connection is established, the user can start to operate the robot.

68

T. Kawashima et al. Table 1. UFCP message format

UFCP- TCP DATA DESCRIPTION COMMAND UOF- ROBOT START RFID Starting u- Object finding UOF- ROBOT STOP Stopping u- Object finding ACK VALUE, POSITION, etc. Acknowledgement NAK ERROR MESSAGE Negative acknowledgement GET STATUS Getting UOF- Robot status GET VALUE DEVICE NAME Getting specified device’ s value GET POSITION Getting UOF- Robot position SET VALUE DEVICE NAME, VALUE Setting specified device’ s value FOUND RFID Informing found RFID tag SOUND Request for playing sound UFCP- UDP DATA DESCRIPTION COMMAND SEARCH Searching UOF- Robot ACK UOF- ROBOT NAME Acknowledgement

・ Operation of the UOF-Robot

The UOF-Manager can specify which robot will search a u-object through its RFID, can start and stop the UOF-Robot, or get the status of the UOF-Robot. In addition, the UOF-Manager can ask the UOF-Robot to produce a beeping sound to inform the user of its current position in the smart room.

4.3 O-Manager The O-Manager handles the registration and deletion of a new u-Object. When the user holds the RFID attached to an u-Object over an RFID reader, the O-Manager obtains the RFID data through the RFID reader and check whether the RFID is already registered or not. If the RFID is not yet registered, the O-Manager shows a popup window for the user to input the name of the u-Object associated with the RFID. After the user input the name successfully, the user can select the u-Object to be searched in the smart room. The user can also register a u-Object by not using an RFID reader but by inputting the RFID data and name manually. Another function of the O-Manager is to manage the existence status of all registered u-objects. A registered object can be brought out/back to the room. The u-object’s in/out state and corresponding time can be detected by the RFID reader installed at the room entrance. When a UOF-Robot moves around the room, it may encounter many u-objects including the one to be searched and others which are not to be searched. Actually, the positions’ data of all u-objects encountered by the robot can be kept in the u-object database. These existence and position status information can be further used to speedup the searching and tracing of u-objects.


69

5 UOF-Robot As explained in the previous sections, the UOF-Robot is mainly in charge of searching a u-object in the room space. The search is based on two special functions. The first is that, the UOF-Robot can move around the space; and the second is that, the robot carries a RFID reader to detect a RFID tag within a certain distance, e.g. 10cm. The UOF-Robot consists of a R-Controller, a R-Machine and a R-Reader, as shown in Fig. 4.

R- Controller (PDA) R- Machine

R- Controller (PDA)

UFCP Stack ht oot eu lB

R- Machine

UFCP Stack Motor Ultrasonic Sensor

Device Driver B S U

R- Reader

Fig. 4. The UOF-Robot components

Ultrasonic Sensor

Motor R- Reader

Fig. 5. A snapshot of the UOF-Robot

The R-Controller resides in a PDA and is used to communicate and control both the R-Machine and the R-Reader. The R-Reader is a RFID reader that is connected to the PDA via a USB cable. A manufacturer-dependent device driver should be installed in the PDA for the R-Controller to communicate with the R-Reader. The RReader’s task is relatively simple, i.e., mainly reading the nearby tags and sending the tags’ codes to the R-Controller. The R-Machine is a self-movable robot that has a motor, wheels, an ultrasonic sensor, etc. Figure 5 shows a snapshot of the UOF-Robot used in our system. The R-Machine can start to move or stop by following the instructions from the RController. The R-Machine can move around by itself by sensing the surrounding situations with its ultrasonic sensor. The sensor can measure the distance between the R-Machine and the object in front. When the distance from an encountered barricade or block is below some threshold value, it may move away from the barricade/block, and change its moving direction. The changes of rules on the robot’s turn movement and the degree of direction are programmable, and can be set by the user on the UOFGUI. Although the R-Machine is operated by the R-Manager, their interaction messages during operations are mediated via the R-Controller. The R-Controller communicates with the R-Manager over a wireless LAN (IEEE802.11b), and also communicates with the R-Machine using a Bluetooth network, which is embedded in the robot. Like the R-Manager, the UFCP stack runs on both the R-Controller and R-Machine. Therefore, the three of them can exchange

70

T. Kawashima et al.

messages with each other. When receiving a robot retrieval message request from RManager, the R-Controller sends a UFCP-UDP response message to the R-Manager. The operation request, on the other hand, is sent as an UFCP-TCP message. When the robot moves, the R-Reader detects whether there is any RFID tag nearby. If it detects one, it sends the tag code to the R-Controller which will check whether the code is the same as the u-object’s code to be searched. When the RController gets the RFID code of the u-object to be searched from the R-Reader, it sends a found message in UFCP-TCP format to the UOF-Manager. In addition, when it detects a position tag which is set on the floor in a smart room, it also sends the UOF-Manager the tag code, which can be used to get the current robot position. An operation message from the UOF-Manager is resent to the R-Machine by the RController via the Bluetooth network.

6 System Implementation and Experiment A preliminary u-object finding prototype has been implemented with emphasis on communications between heterogeneous devices across different physical networks, and management of the robots and u-objects. The following devices and software were used in our current implementation.

・・・・

Robot: Mindstorm NXT including an Ultrasonic sensor and a Motor developed by LEGO [19]. It adopts lejOS [20] to control the robot movement and set some moving parameters. The ultrasonic sensor is installed in front of the robot. RFID tag and reader developed by Phidgets, Inc [21]. Java-based API which can run on Windows, Linux and Mac OS X platforms. The RFID tags were attached in all objects to be searched, and were also set on some locations on the room floor. The readable distance between a tag and a reader is about 5cm. PDA: AXIM series by Dell, Inc. It runs on Windows Mobile OS and supports .NET Compact Framework. Networks: wireless LAN (IEEE 802.11a/b/g) between the PC and PDA, and Bluetooth between PDA and NXT robot. The USB cable is used to connect a Phidgets tag reader to the PDA.

The u-object finding system is developed using Java programming language. Its software consists of three main parts. The first one is the UFCP, which is a TCP/IPbased common communication platform running on PC, PDA and robot. The UFCP is a very light protocol, only about 50Kbytes, and thus able to run low-spec devices. The second one is the R-Controller running on the PDA. It has multiple functions; such as interacting with the UOF-Manager to get commands and send back results, checking the tag codes from the R-Reader, controlling the R-Machine, and mediating messages between the R-Manager and the R-Machine. The third one is the UOF-Manager running on a PC, which manages both robots and u-objects, and also provides a GUI to interact with a user. To test our system, the experimental environment was set up in our laboratory room, and a partial setting is shown in Fig. 6. The space is 5mx5m on which there is one tagged object to be searched, a robot to search the object, some position tags fixed on the floor, etc.


71

Fig. 6. The UOF experiment environment

When a user wants to find an object in the space, he/she first starts the UOFManager, which is a Java application program running on a PC. Figure 7 shows the snapshot of the manager’s UOF-GUI. All u-objects registered in this room are listed on the upper-left of the GUI. From the u-object list, the user selects the desired object to be found. Next, the user can check what UOF-Robots are available in the room by clicking a robot retrieval button. As shown in the bottom-left of Fig. 7, one robot is retrieved through communications between the R-Manager on the PC and the R-Controller on the PDA carried by the robot. When the UOF-Manager makes a connection with the UOF-Robot successfully, a new tab window is added to the right side of GUI. At the same time, the robot appears at its present position in the middle-right map of the GUI. Some parameters related to the robot movement are displayed as well. The user can operate the UOF-Robot using the buttons and the text field on the tab window. Upon receiving the instruction to find a u-object, the UOF-Manager will send a “START” message command which sends the u-object tag code to the R-Controller of the selected UOF-Robot. Then, the R-Controller gives the “START” command to the R-Machine and asks the R-Reader to read the RFID tags. When the robot moves around, the R-Reader sends every tag code it has detected to the R-Controller. The R-Controller compares the tag code received from the R-Reader with the tag code of the searched u-object. If the two codes are the same, it means that the searched object is very near the robot. Thus, the R-Controller sends a “STOP” command to the R-Machine, and a “FOUND” message to the UOF-Manager. And then, the robot stops and generates a beeping sound to notify the user of its location. If the u-object is not yet found within the specified period of time, the search operation is considered a failure, and then the process is terminated. In our experimental environment, the u-object was found in many cases, but the search time varied greatly depending upon the positions of the robots and the uobject. This is mainly due to the short readable distance between a RIFD read and a

72

T. Kawashima et al.

Fig. 7. A snapshot of the UOF-Manager GUI

tag, and the robot’s moving course in our current implementation which is based on a rather simple algorithm that the UOF-Robot monitors the distance value from its ultrasonic sensor and turns left or back whenever the distance between the robot and barrier becomes less than 10cm. Although the searching performance is unsatisfactory, the whole system works well in terms of communications and management of the robots and devices in the smart space.

7 Conclusion and Future Work The smart space/environment and robotics are used to be studied as two separate research disciplines. It is therefore very significant to study their combination so as to form a new merged research area and generate some novel applications. Based on this idea, we first investigated the potential services and possible new technical issues when integrating the robots into smart spaces, and then focused on a case study of a uobject finding movable robot prototype to search a u-object, which is attached with RFID tags, in a smart room space. We have discussed the design and implementation as well as experiments of the developed system prototype. As mentioned in the previous section, the research presented in this paper is mainly for communications and management of robots and u-objects in a smart space. However, the object searching procedure took a long time and thus it’s inefficient. This is because of two problems: the short reading distance (about 10cm) between the RFID reader and a tag, and the course was simple for the robot’s movement. To overcome the former limitation, the RFID tag with longer reading distance such as 30~50cm will be used in our next study. Multiple tags can be attached on the different sides of a large object to improve the searching efficiency. In addition, a set of RFID readers which can read tags within 1~2m may be installed in fixed locations to first detect the possible region where the u-object is located and the robot can go directly to the


73

region to find the more precise position of the u-object. To overcome the second limitation, the robot must be able to remember the route passed and the obstacles encountered to avoid going to the same places repeatedly. Another interesting approach is to ask two or more robots to search a u-object together, and study how the robots collaborate with each other. Aside from the above, there are still many technical problems and challenging issues to tackle when combining robots and smart space which should be studied along with many kinds of applications in the future.

Acknowledgement This work was partially supported by Japan Society for the Promotion of Science Grants-in-Aid for Scientific Research, 19200005.

References 1. Weiser, M.: Ubiquitous Computing. IEEE Computer (October 1993) 2. Weiser, M., Brown, J.S.: The Coming Age of Calm Technology. In: Denning, P.J., Metcalfe, R.M. (eds.) Beyond Calculation: The Next Fifty Years of Computing (1996) 3. IEEE802.11, http://www.ieee802.org/11/ 4. Bluetooth, http://www.bluetooth.org/ 5. Callaway, E., Gorday, P., Hester, L., Guiterrez, J.A., Naeve, M.: Home Networking with IEEE802.15.4: A Developing Standard for Low-Rate Wireless Personal Area Networks. IEEE Communications Magazine, 70–77 (August 2002) 6. IEEE802.15 Working Group for WPAN, http://www.ieee802.org/15/ 7. Porcino, D., Hirt, W.: Ultra-Wideband Radio Technology: Potential and Challenges Ahead. IEEE Communications Magazine, 66–67 (July 2003) 8. Dey, A.K.: Understanding and Using Context. Personal and Ubiquitous Computing 5(1), 4–7 (2001) 9. Abowd, G.D., Mynatt, E.D., Rodden, T.: The Human Experience. IEEE Pervasive Computing, 48–57 (January-March 2002) 10. Ma, J.: Smart u-Things – Challenging Real World Complexity. In: IPSJ Symposium Series, vol. 2005(19), pp. 146–150 (2005) 11. Private, G.: Smart Devices: New Telecom Applications & Evolution of Human Interfaces. In: CASSIS Int’l Workshop, Nice (March 2005) 12. Cook, D.J., Das, S.K.: Smart Environments: Technologies, Protocols and Applications. Wiley-Interscience, Chichester (2005) 13. Ma, J., Yang, L.T., Apduhan, B.O., Huang, R., Barolli, L., Takizawa, M.: Towards a Smart World and Ubiquitous Intelligence: A Walkthrough from Smart Things to Smart Hyper-spaces and UbicKids. International Journal of Pervasive Comp. and Comm. 1(1) (March 2005) 14. Pinto, J.: Intelligent Robots Will Be Everywhere, in Automation.com (2003), http://www.jimpinto.com/writings/robots.html 15. Sakagami, Y., Watanabe, R., Aoyama, C., Matsunaga, S., Higaki, N., Fujimura, K.: The intelligent ASIMO: System Overview and Integration. Intelligent Robots and System (2002)

74

T. Kawashima et al.

16. Kaneko, K., Kanehiro, F., Kajita, S., Hirukawa, H., Kawasaki, T., Hirata, M., Akachi, K., Isozumi, T.: Humanoid Robot HRP-2 (ICRA 2004) (April 2004) 17. Ueno, K., Kawamura, T., Hasegawa, T., Ohsuga, A., Doi, M.: Cooperation between Robots and Ubiquitous Devices with Network Script Flipcast. In: Proc. Of Network Robot System: Toward Intelligent Robotic Systems Integrated with Environments (IROS) (2004) 18. Ma, J., Lee, J., Yamanouchi, K., Nishizono, A.: A Smart Ambient Sound Aware Environment for Be Quiet Reminding. In: Proc. of IEEE ICPADS/HWISE 2005, Int’l Workshop on Heterogeneous Wireless Sensor Networks, Fukuoka (July 2005) 19. Lego Mindstorm NXT, http://mindstorms.lego.com 20. lejOS, http://lejos.sourceforge.net/ 21. Phidgets, Inc., http://www.phidgets.com

Biometrics Driven Smart Environments: Abstract Framework and Evaluation Vivek Menon1, , Bharat Jayaraman2, and Venu Govindaraju2 1

Amrita University, Coimbatore 641 105, India vivek [email protected] 2 University at Buffalo, Buffalo, NY 14260, USA [email protected], [email protected] Abstract. We present an abstract framework for ‘smart indoor environments’ that are monitored unobtrusively by biometrics capture devices, such as video cameras, microphones, etc. Our interest is in developing smart environments that keep track of their occupants and are capable of answering questions about the whereabouts of the occupants. We abstract the smart environment by a state transition system: Each state records a set of individuals who are present in various zones of the environment. Since biometric recognition is inexact, state information is probabilistic in nature. An event abstracts a biometric recognition step, and the transition function abstracts the reasoning necessary to effect state transitions. In this manner, we are able to accommodate different types of biometric sensors and also different criteria for state transitions. We define the notions of ‘precision’ and ‘recall’ of a smart environment in terms of how well it is capable of identifying occupants. We have developed a prototype smart environment based upon our proposed concepts, and provide experimental results in this paper. Our conclusion is that the state transition model is an effective abstraction of a smart environment and serves as a basis for integrating various recognition and reasoning capabilities.

1

Introduction

The vision of pervasive computing [1] provides the inspiration for a smart environment saturated with sensors, computing, and communication devices that are gracefully interfaced with human users [2]. In this paper, we focus on smart indoor environments such as homes, offices, etc. Indoor environments do not suffer from the problems of power or battery-life that confront outdoor environments. The goal of our research is to develop smart indoor environments that can identify and track their occupants as unobtrusively as possible and be capable of answering queries about the occupants. Such ‘context-aware’ systems can identify and track people in environments ranging from homes for elderly or disabled, office workplace, department stores and shopping complexes to larger arenas such as airports, train stations, etc.

This work was done while the author was a Visiting Research Scientist at the Center for Unified Biometrics and Sensors, University at Buffalo.

F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 75–89, 2008. c Springer-Verlag Berlin Heidelberg 2008

76

V. Menon, B. Jayaraman, and V. Govindaraju

(a)

(b)

(c)

Fig. 1. Architecture of a Biometrics Driven Smart Environment

Identification approaches vary from tag-based approaches such as those involving RFID to those based on biometrics of the user. Tag-based methodologies tend to be obtrusive, requiring the individual to continuously retain them, however small the tag maybe. Some of the biometric techniques, such as fingerprint and iris scans, require a ‘pause-and-declare’ interaction with the human [3]. They are less natural than face, voice, height, and gait, which are less obtrusive and hence are better candidates for use in our smart environments. Figure 11 shows the overall architecture of a biometrics-driven smart environment. Although the figure illustrates a single biometric modality of face recognition, the architecture is also applicable to other biometric modalities. For example, zone 1 might use voice recognition, zone 2 might use face recognition, 1

Face images blurred to preserve anonymity.

Biometrics Driven Smart Environments

77

and zone 3 might use height estimation. However, in all cases the output of a biometric recognition is the set of person probability pairs as discussed in more detail below. The main contribution of this paper is a framework for abstracting the behavior of a biometrics-driven smart environment in terms of a state transition system. Our proposed framework supports multiple biometric modalities in a uniform manner and facilitates a precise statement of the performance aspects of a smart environment. The state of an environment is expressed in terms of the probabilities of the occupants being present in the different zones of the environment. The state information is probabilistic because a biometric recognizer typically provides a set of scores indicating the degree of match between the subject and the candidates in the database. Therefore, in our approach an event abstracts a biometric recognition step - whether it is face recognition, voice recognition, etc. - and is represented as a set of pairs o, p(o) where p(o) is the probability 2 that occupant o has been recognized at this event. The transition function abstracts the reasoning necessary to effect state transitions. Effectively, the transition function takes as input, a state and an event, and determines the next state by assigning revised probabilities to the occupants in the environment based upon the probabilities in the event. In this manner, we are able to accommodate different types of biometric sensors and also different criteria for state transitions, including those that incorporate declarative knowledge of the individuals and the environment. It is not necessary for us to consider non-deterministic transitions, since a state itself is represented as a set of occupants and their probabilities. We introduce the concepts of precision and recall in order to provide a quantitative measure of the performance of a smart environment. Precision captures how well an occupant is recognized, while recall captures whether an occupant is recognized at all. These are complementary concepts and together capture the overall performance of a smart environment. The concepts of precision and recall are standard performance measures in the information retrieval literature [6], but we have adapted the definitions to suit our context. The rest of this paper is organized as follows. The related work is presented in Section 2 while the details of our model are discussed in Section 3. The experimental prototype is described in Section 4 and the conclusions and the future work are presented in Section 5.

2

Related Work

There has been considerable interest in the subject of smart environments. A major difference between our proposed approach and several of the approaches surveyed below is our use of a framework in which the details of biometric recognition are abstracted simply as events in a state transition system. We also 2

Such scores can be mapped to probabilities by running the recognizer on a large number of samples, as shown in [4,5].

78


characterize the performance of the smart environment in terms of the concepts of precision and recall. We briefly survey closely related efforts below and highlight their main features. The survey paper by Cook and Das [7] provides a good account of the stateof-the-art in smart environments. There are several projects that focus on smart homes (MavHome [8], the Intelligent Home [9], the House n [10] and development of adaptive control of home environments by also anticipating the location, routes and activities of the inhabitants. 2.1

Location Estimation

An extensive survey and taxonomy of location systems for a ubiquitous computing application is discussed in [11] while [12] provides a more recent survey of position location techniques in mobile systems and draws a comparison between their parameters. Fox et al. [13] highlight the relevance of Bayesian filtering techniques in dealing with the uncertainty characteristic of sensors in pervasive computing environments and apply them in the context of estimation of an object’s location. Data Association problem [14] is a pertinent issue in scenarios which involve tracking of multiple people using anonymous sensors. The ambiguity in identity estimation arising due to the absence of ID sensors and the corresponding tags on people is difficult to resolve. Thus identity estimation, absolute or relative is a desirable and integral subsystem of any object tracking mechanism. 2.2

Tracking

Bui et al. [15] propose an Abstract Hidden Markov Model(AHMM) based approach for tracking human movement in an office like spatial layout and to predict the object trajectories at different layers of detail. For location tracking in a single inhabitant smart space, Roy et al. [16] propose an optimal algorithm based on compressed dictionary management and online learning of the inhabitant’s mobility profile. In a related work [17], they highlight the complexity of optimal location prediction across multiple inhabitants in a smart home and propose a stochastic game-theoretic approach, which learns and estimates the inhabitants’ most likely location profiles. This study uses RFID tags to track the inhabitants. A combination of particle filters with Kalman filters to track multiple objects with accuracy of anonymous sensors and identification certainty of id-sensors is discussed in [13,18]. Krumm et al. [19] investigate the nuances of visual person tracking in intelligent environments in the context of the EasyLiving project [20] by deploying multiple cameras for tracking multiple people. However this tracking experiment was within the confines of a single room and the identity estimation and maintenance for facilitating the tracking only dealt with non-absolute, internally system generated identity of tracked persons. 2.3

Biometrics in Smart Environments

Pentland and Choudhury [3] highlight the importance of deploying audio-andvideo based recognition systems in smart environments as these are modalities


79

similar to those used by humans for recognition. They summarize the face recognition efforts and discuss various commercial systems and applications as well as its novel applications in smart environments and wearable computing. Chen and Gellersen [19] propose a new method to support awareness based on fusion of context information from different sources in a work environment which included integration of audio and video sources with more specific environment sensors and with logical sensors that capture formal context. Reasoning over the context information generated by applying different perception techniques on the raw data collected is used to generate a refined context. Hamid Aghajan et al. [22] propose a vision-based technology coupled with AI-based algorithms for assisting vulnerable people and their care givers in a smart home monitoring scenario. However, users are expected to wear a wireless identification badge that broadcasts a packet upon sensing a significant signal by one of the accelerometers. Gao et al. [23] propose a new distance measure for authentication in their face recognition system for a ubiquitous computing environment which relies on a fusion of multiple views of each person. Their work focuses on optimizing a single modality to improve robustness rather than deploying a multimodal approach that fuses different biometrics. Hong et al. [24] discuss structure, operation and performance of a face verification system using Haar-like features and HMM algorithm in a ubiquitous network environment. Zhang et al. [25] propose a distributed and extensible architecture of a continuous verification system that verifies the presence of logged-in user. A Bayesian framework that combines temporal and modality information holistically is used to integrate multimodal passive biometrics which includes face and fingerprint. Driven by the proliferation of commercially available hand-held computers, Hazen et al. [26] research the improvements in identification performance by adopting a bimodal biometric approach to user identification, for use on mobile devices by integrating audio and visual biometric information in the form of voice and face. They also report significant improvements in identification performance that can stem from using dynamic video information instead of static image snapshots on a database of 35 different people. Highlighting the privacy concerns of video recording discussed in Bohn et al [27] and reliability issues of face recognition techniques for user authentication, Vildjiounaite et al. [28], deploy a fusion of accelerometer based gait recognition and speaker recognition as an unobtrusive and marginally privacy-threatening means of user authentication with personal mobile devices. They have reported improved performance in the combination mode as opposed to individual modalities for a user base of 31. More recently Bernardin Stiefelhagen [29] have implemented a system for the simultaneous tracking and incremental multimodal identification of multiple users in a smart environment which fuses person track information, localized speaker ID and high definition visual ID cues opportunistically to gradually refine the global scene model and thus increase the system’s confidence in the set of recognized identities. In terms of the focus on the use of non-obtrusive biometrics based recognition and location estimation, our work is similar to [29]. However, in our research,

80


we propose an abstract framework where in a variety of biometric modalities can be incorporated in a uniform manner. Our approach to identity estimation deals with the absolute identity of people across multiple zones of a facility. However, we attempt to highlight the inherent uncertainty of automated face recognition by recasting the eigen distances generated by eigenface algorithm into a probability distribution of the registered faces, instead of the conventional approach of assigning the value with the least eigen distance as the matching face. This probabilistic approach to biometric recognition is a key theme around which we construct our abstract framework for a biometrics driven smart environment. 2.4

State Space Representation

It might appear that a Hidden Markov Model(HMM) would serve as an elegant basis for representing the state space. From a HMM perspective, a smart environment with n occupants and m zones can have mn distinct possible states. Thus probabilities are not associated with the states but with the transitions between them; these transition probabilities are to be learnt from past behavior or by simulation [15]. Thus an HMM approach is computationally more complex due to a state space explosion and the requirement of a priori probabilities of trajectories. In our approach, the size of a state is m ∗ n, meaning that for each of the m zones we record the probabilities of each of the n occupants being present in that zone. Therefore, in Section 4 (Evaluation), we depict a typical state as a table with n rows and m columns. The transitions from one state to another are deterministic. Therefore, given any event in a zone, the next state is unambiguously determined. In contrast with the HMM approach, we do not need to learn the transition probabilities in order to determine the next state because biometric recognition (or event) provides a direct means for effecting state transitions. Our state transition model is discussed in the next section.

3

Framework

Definition (Smart Environment). An n-person smart environment is abstracted as a state transition system (S, E, Δ) where S is the set of states labeled s0 , s1 , . . . sx ; E is the set of events labeled e1 , e2 , . . . ex and Δ : S × E → S is a function that models the state transition on the occurrence of an event. The state transitions may be depicted as follows: e

e

e

1 2 x s0 → s1 → s2 . . . → sx

We shall consider a smart environment as being divided into a number of zones, each of which may be a region (or a set of rooms). We include two special zones, an external zone and a transit zone, for the sake of convenience. Definition (State). Given n occupants, o1 . . . on and m zones labeled 1 . . . m, a state sk of the environment is represented by an m-tuple Z1k . . . Zmk where for 1 ≤ j ≤ m, Zjk= {oi , pjk (oi ) : 1 ≤ i ≤ n}. Also, in each state sk and for m each occupant oi , i=1 pjk (oi ) = 1.


81

The state of an environment is expressed in terms of the probabilities of the occupants being present in the different zones of the environment. The constraint m i=1 pjk (oi ) = 1 indicates that sum of probabilities of any occupant being present across all the zones in any state equals one. In the initial state s0 , we may assume without loss of generality that all occupants are in the external zone with probability 1. Given a smart environment with n occupants, m zones, and x number of events, the total size of the state space is m ∗ n ∗ (x + 1). Thus, the size of the state space is quadratic in m and n rather than exponential, as in HMMs. In this paper we model all exit events as entry events into a transit zone. Hence it suffices in our model to only consider entry events. An event is essentially an abstraction of a biometric or feature recognition step performed in the environment. Definition (Event). Given n occupants o1 . . . on , an (entry) event ek occurring at zone j (1 ≤ j ≤ m) at time t is represented as t, j, P , where P = {oi , pjk (oi ) : 1 ≤ i ≤ n} and pjk (oi ) is the probability that an occupant oi was recognized at zone j in event ek . As noted earlier, an event is an abstraction of a recognition step. For simplicity, we assume that events happen sequentially in time, i.e., simultaneous events across different zones are ordered arbitrarily in time. That is, the entry of an occupant oi into zone zi and occupant oj to zone zj at the same time t can be modeled as oi before oj or oj before oi . Definition (Transition Function). Δ : S × E → S, maps state sk−1 into state sk upon an event ek = t, j, P occurring at time t in zone j, where P = {oi , pjk (oi ) : 1 ≤ i ≤ n}. Let sk−1 = Z1k−1 . . . Zjk−1 . . . Zmk−1 and Zjk−1 = {oi , pjk−1 (oi ) : 1 ≤ i ≤ n}. Then Δ determines state sk = Z1k . . . Zjk . . . Zmk as follows: Let xi = 1 − pjk (oi ). Then, Zjk = {oi , pjk (oi ) + xi ∗ pjk−1 (oi ) : 1 ≤ i ≤ n}

(1)

Zlk = {oi , xi ∗ plk−1 (oi ) : 1 ≤ i ≤ n}, for 1 ≤ l ≤ m and l = j

(2)

The transition function maps a state sk−1 to a state sk upon an event ek occurring at zone j. For zone j, we sum the new probability pjk (oi ) for an occupant generated by event ek with the complement of the new probability value 1 − pjk (oi ), apportioned by a factor of the existing probability pjk−1 (oi ). In the event of a revision, there might be a violation of the constraint m that the sum of probabilities for any occupant across all zones equals one ( i=1 p(oi ) = 1). To restore adherence to this constraint, for each occupant oi , we apportion to the probability of oi being present in each zone l = j by redistributing the complement of the new probability value, 1 − pjk (oi ), in the ratio of the probability value in existing state plk−1 (oi ). This transition function ensures that the probability values associated with a new event as well as the current state figure in the determination of the new state as in a Markov process. Since we are dealing with a closed environment with a fixed set of occupants, o1 . . . on , we can, in general, utilize a priori declarative knowledge regarding

82


the occupants, such as their schedules or knowledge of the environment, such as the distance between rooms and whether an occupant could move between a pair of rooms within a certain interval of time. However, the transition function presented in the above definition does not make any use of such declarative knowledge of the occupants. The nature of the declarative knowledge can also be fuzzy to factor in the probabilistic nature of the environment. The reasoning component can alleviate some of the weaknesses in the recognition component. However it should ensure that its results are not totally contradictory to the recognition system thereby generating inconsistencies. We now define the concepts of precision and recall for a smart environment. These are defined in terms of the ground truth, which, for a given input event sequence, is a sequence of states of the environment wherein the presence or absence of any occupant in any zone is known with certainty (0 or 1). Precision captures how well an occupant is recognized, while recall captures whether an occupant is recognized at all. Definition (Ground Truth). Given n occupants O={o1 . . . on } and an event sequence e1 . . . ex , then the ground truth is the sequence of states g1 . . . gx where each gk = T1k . . . Tjk . . . Tmk and Tjk = {oi , qjk (oi ) : 1 ≤ i ≤ n ∧ qjk (oi ) ∈ {0, 1}}. Also, oi , 1 ∈ Tjk → oi , 1 ∈ Tlk , for all l = j in state gk . Given a zone of a state, the precision for that zone of the state is defined as the average probability of those occupants that are present in that zone of the state as given in the ground truth. The average precision across all zones (where at least one occupant is present as per the ground truth) is the precision for the state, and the average precision across all states is the precision for a given ground truth. Finally, the average across multiple ground truths is the precision of the smart environment. Definition (Precision). Given an environment with m zones, n occupants O = {o1 . . . on }, an event sequence E = e1 . . . ex , a ground truth G = g0 , g1 , . . . gx , and state transitions S = s0 , s1 , . . . sx . We define the precision, π, with respect to G as follows: Let πjk = ajk /bjk , where ajk =

{pjk (oi ) : 1 ≤ i ≤ n ∧ qjk (oi ) = 1}

bjk = |

{oi : 1 ≤ i ≤ n ∧ qjk (oi ) = 1}|

Then πk =

m j=1

πjk /m, and we define π =

x k=1

πk /x.

Now, given a set of ground truths {G1 , G2 , . . . G2 } with the corresponding precit sions {π 1 , π 2 , . . . π t }, the precision of the smart environment, Π = l=1 π l /t. For a given ground truth, state and zone, we define recall with respect to a threshold θ as the ratio a/b, where a is the number of occupants of that zone with probabilities greater than θ and who are present in the ground truth, and


83

b is the number of occupants who are present in the ground truth for that zone. The recall for a state is the average of the probabilities across all zones where at least one occupant is present as per the ground truth. The average recall across all states is the recall for a given ground truth, and the average across multiple ground truths is the recall of the smart environment. Definition (Recall). Given an environment with m zones, n occupants O = {o1 . . . on }, an event sequence E = e1 . . . ex , a ground truth G = g0 , g1 . . . gx , and state transitions S = s0 , s1 , . . . sx . We define the recall, ρ, with respect to a threshold θ as follows: Let ρjk = ajk /bjk , where ajk = |

{oi : 1 ≤ i ≤ n ∧ qjk (oi ) = 1 ∧ pjk (oi ) > θ}|

bjk = |

{oi : 1 ≤ i ≤ n ∧ qjk (oi ) = 1}|

Then ρk =

m j=1

ρjk /m, and we define ρ =

x k=1

ρk /x.

Now, given a set of ground truths {G1 , G2 , . . . G2 } with the corresponding precit sions {ρ1 , ρ2 , . . . ρt }, the recall of the smart environment, R = l=1 ρl /t. As it is clear, the recall is inversely proportional to the threshold, θ, since lowering the threshold will result in more occupants being identified. This figure is generally arrived at experimentally for a smart environment. A reasonable choice of θ is 0.5, and this is also the value that we adopt in our experiments. In the above definition, the recall was defined zone-wise. An alternative approach is to disregard the zones while taking the ratio; doing so would increase the overall recall. Our definition gives due importance to zones, and hence is a relatively more conservative.

4

Evaluation

We have developed an experimental prototype embodying the ideas presented in this paper. Figure 1(b) illustrates a 4-zone smart environment (fourth zone representing the external zone) with 25 occupants who are monitored by video cameras installed in each of the zones. Our experimental prototype collects sample face images of the 25 occupants of an office facility and pre-registers them in a training database. The image sensors deployed in each zone detects the presence of occupants as they move through the zones and verify the face images extracted from the video against the database. The distance scores generated by eigenface is recast into a probability value [4,5] which denotes a posterior probability of the detected face matching the pre-registered occupants. This set of person-probability pairs generated essentially constitutes an event as defined in Section 3.

84


Fig. 2. Event Sequence

Although our abstract framework is independent of the details of any particular biometric modality, we illustrate our concepts in terms of face recognition. Automated face recognition is yet to attain any comparable levels of robustness as that of humans. Factors such as viewing angle, distance, background clutter, lighting spectrum, intensity, angle and diffuseness of lighting, differences between posed photographs and spontaneous expression can cause fluctuations in the performance of computer vision based on statistical classifiers [30]. Our prototype is based upon OpenCV’s [31] implementation of the eigenface algorithm [32], which provides a basis for a simple though not robust implementation of face recognition. This prototype can be extended to incorporate other biometric recognizers in a similar manner. For example, for voice recognition, the voice samples of occupants are pre-registered in a database instead of face image samples. The ambient voice that is picked up by the voice sensors installed in different zones can be


85

Fig. 3. Sample State Transition

(a)

(b) Fig. 4. Illustrating Precision and Recall

matched against the voice database to generate a set of person-probability pairs. In this manner the different biometric recognizers are interfaced in a uniform way with the rest of the system. We illustrate with figures 2, 3, and 4, the computation carried out by the state transition system that is embodied in the 4-zone smart environment. We

86


have presented the observations and results obtained in a tabular form for ease of understanding. Figure 2 presents 10 events involving four of the occupants of the smart environment. Each event is presented as a column of probability values - these form the output of the face recognition module as it matches a given face against the training database. Shown in boldface are the probability values corresponding to the actual occupants who were involved in the corresponding events, as per the ground truth. Italicized values indicate the probabilities corresponding to the occupants who were not involved in the corresponding events, but falsely assigned values by the recognizer. This ambiguity may arise due to any of the reasons already discussed above. Figure 3 illustrates a sample transition from state s8 to s9 upon the occurrence of event e9 in zone 1. The probabilities of the occupants in zone 1 in state s9 are obtained as per equation (1) defined under the transition function definition. For the remaining zones of s9 , the probability values are obtained as per equation (2) defined under the transition function. Figure 4 illustrates the results of the precision and recall for the ground truth corresponding to the event sequence of figure 2. The low values for precision at zone 3, corresponding to states s3 and s10 in particular, can be traced to the ambiguity arising in the face recognition step at events e3 and e10 , both occurring at zone 3 which results in a low probability of recognition of occupants o1 and o2 at these events respectively. For the same reason, the values for recall at zone 3 also suffers, thereby affecting the average recall of the states s3 . . . s6 .

5

Conclusions

We have presented a novel framework for non-obtrusive biometric based indoor smart environments for identification and tracking. 1. A state transition framework in which events abstract different biometric recognition steps and transitions abstract different reasoning steps. 2. A characterization of the performance of the smart environment in terms of the concepts of precision and recall. Our state transition system is fundamentally probabilistic because the biometric recognition that underlies events is inexact in nature. Our formulation of precision and recall succinctly characterizes the performance of a smart environment. We believe that our state transition model is an effective abstraction of a smart environment and serves as a basis for integrating different recognition and reasoning capabilities. In our model, a state provides location information at a zone level and a sequence of consecutive states implicitly contains zone level tracking information for all occupants. While it is possible to define precision and recall with respect to any query of interest, we have formulated them in a query independent manner which we believe is more general. Our model is capable of supporting spatial and temporal queries, such as: the location of an occupant in the facility; time of entry/exit of an occupant; the first/last person to enter/leave the facility;


87

the current occupants present in the facility; etc. A query of special interest is tracking of individuals over time and space. We plan to enhance our current prototype by incorporating a variety of biometric recognizers, such as for height, gait, voice, etc. It is possible to fuse two or more biometric modalities to enhance the overall performance of recognition. We also plan to incorporate spatio-temporal reasoning based upon declarative knowledge of the environment as well as the occupants. Through such enhancements in recognition and reasoning, we can improve the overall precision and recall of the smart environment. We plan to test this approach on larger environments and support speech-based information retrieval queries about the environment.

Acknowledgements Thanks to Philip Kilinskas for his help in developing the experimental prototype; Dr. Jason J. Corso for discussions on Markov models; and members of the Center for Unified Biometrics and Sensors for their comments and suggestions.

References 1. Weiser, M.: The Computer for the 21st Century. Scientific American 265(3), 66–75 (1991) 2. Satyanarayanan, M.: Pervasive Computing: Vision and Challenges. IEEE Personal Communications 8(4), 10–17 (2001) 3. Pentland, A., Choudhury, T.: Face Recognition for Smart Environments. IEEE Computer 33(2), 50–55 (2000) 4. Cao, H., Govindaraju, V.: Vector Model Based Indexing and Retrieval of Handwritten Medical Forms. In: Proc. of the Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), pp. 88–92. IEEE Computer Society, Washington (2007) 5. Bouchaffra, D., Govindaraju, V., Srihari, S.: A Methodology for Mapping Scores to Probabilities. IEEE Transactions on Pattern Analysis and Machine Intelligence 21(9), 923–927 (1999) 6. van Rijsbergen, C.J.: Information Retrieval. Butterworths, Boston (1979) 7. Cook, D., Das, S.: How smart are our environments? An updated look at the state of the art. Pervasive and Mobile Computing 3(2), 53–73 (2007) 8. Youngblood, M., Cook, D.J., Holder, L.B.: Managing adaptive versatile environments. In: Proc. of the Third IEEE Intl. Conf. on Pervasive Computing and Communications, pp. 351–360. IEEE Computer Society, Washington (2005) 9. Lesser, V., et al.: The Intelligent Home Testbed. In: Proc. of the Autonomous Agents 1999 Workshop on Autonomy Control Software. ACM, New York (1999) 10. House n Living Laboratory Introduction (2006) 11. Hightower, J., Borriello, G.: Location Systems for Ubiquitous Computing. IEEE Computer 34(8), 57–66 (2001) 12. Manesis, T., Avouris, N.: Survey of position location techniques in mobile systems. In: Proc. of the Seventh Intl. Conf. on Human Computer interaction with Mobile Devices and Services. MobileHCI 2005, pp. 291–294. ACM, New York (2005)

88


13. Fox, D., Hightower, J., Liao, L., Schulz, D., Borriello, G.: Bayesian filtering for location estimation. IEEE Pervasive Computing 2(3), 24–33 (2003) 14. Bar-Shalom, Y., Li, X.-R.: Multitarget-Multisensor Tracking: Principles and Techniques, Yaakov Bar-Shalom (1995) 15. Bui, H.H., Venkatesh, S., West, G.: Tracking and surveillance in wide-area spatial environments using the Abstract Hidden Markov Model. Intl. Journal of Pattern Recognition and Artificial Intelligence 15(1), 177–195 (2002) 16. Roy, A., Bhaumik, S., Bhattacharya, A., Basu, K., Cook, D.J., Das, S.K.: Location aware resource management in smart homes. In: Proc. of the First IEEE Intl. Conf. on Pervasive Computing and Communications, pp. 481–488. IEEE Computer Society, Washington (2003) 17. Das, S.K., Roy, N., Roy, A.: Context-aware resource management in multiinhabitant smart homes: A framework based on Nash H-learning. Pervasive and Mobile Computing 2(4), 372–404 (2006) 18. Schulz, D., Fox, D., Hightower, J.: People tracking with anonymous and id-sensors using Rao-Blackwellised particle filters. In: Proc. of the 18th Intl. Joint Conference on Artificial Intelligence (IJCAI), pp. 921–926 (2003) 19. Krumm, J., Harris, S., Meyers, B., Brumitt, B., Hale, M., Shafer, S.: Multi-Camera Multi-Person Tracking for EasyLiving. In: Proc. of the 3rd IEEE Intl. Workshop on Visual Surveillance (VS 2000). IEEE Computer Society, Washington (2000) 20. Brumitt, B., Meyers, B., Krumm, J., Kern, A., Shafer, S.A.: EasyLiving: Technologies for Intelligent Environments. In: Thomas, P., Gellersen, H.-W. (eds.) HUC 2000. LNCS, vol. 1927, pp. 12–29. Springer, Heidelberg (2000) 21. Chen, D., Gellersen, H.: Recognition and reasoning in an awareness support system for generation of storyboard-like views of recent activity. In: Proc. of the Intl. ACM SIGGROUP Conference on Supporting Group Work. GROUP 1999. ACM Press, New York (1999) 22. Aghajan, H., et al.: Distributed Vision-Based Accident Management for Assisted Living. In: Okadome, T., Yamazaki, T., Makhtari, M. (eds.) ICOST. LNCS, vol. 4541, pp. 196–205. Springer, Heidelberg (2007) 23. Gao, Y., Hui, S.C., Fong, A.C.: A Multi-View Facial Analysis Technique for Identity Authentication. IEEE Pervasive Computing 2(1), 38–45 (2003) 24. Hong, K., et al.: Real Time Face Detection and Recognition System Using Haar-Like Feature/HMM in Ubiquitous Network Environments. In: Gervasi, O., Gavrilova, M.L., Kumar, V., Lagan´ a, A., Lee, H.P., Mun, Y., Taniar, D., Tan, C.J.K. (eds.) ICCSA 2005. LNCS, vol. 3480, pp. 1154–1161. Springer, Heidelberg (2005) 25. Zhang, S., et al.: Continuous Verification Using Multimodal Biometrics. In: Zhang, D., Jain, A.K. (eds.) ICB 2005. LNCS, vol. 3832, pp. 562–570. Springer, Heidelberg (2005) 26. Hazen, T., Weinstein, E., Heisele, B., Park, A., Ming, J.: Multi-Modal Face and Speaker Identification for Mobile Devices. In: Hammoud, R.I., Abidi, B.R., Abidi, M.A. (eds.) Face Biometrics for Personal Identification: Multi-Sensory Multi-Modal Systems, pp. 123–138. Springer, Heidelberg (2006) 27. Bohn, J., Coroama, V., Langheinrich, M., Mattern, F., Rohs, M.: Social, Economic, and Ethical Implications of Ambient Intelligence and Ubiquitous Computing. In: Weber, W., Rabaey, J., Aarts, E. (eds.) Ambient Intelligence, pp. 5–29. Springer, Heidelberg (2005)


89

28. Vildjiounaite, E., et al.: Unobtrusive Multimodal Biometrics for Ensuring Privacy and Information security with Personal Devices. In: Fishkin, K.P., Schiele, B., Nixon, P., Quigley, A. (eds.) PERVASIVE 2006. LNCS, vol. 3968, pp. 186–201. Springer, Heidelberg (2006) 29. Bernardin, K., Stiefelhagen, R.: Audio-visual multi-person tracking and identification for smart environments. In: Proc. of the 15th International Conference on Multimedia, pp. 661–670. ACM, New York (2007) 30. Hewitt, R., Belongie, S.: Active Learning in Face Recognition: Using Tracking to Build a Face Model. In: Proc. of the 2006 Computer Vision and Pattern Recognition Workshop, pp. 157–157. IEEE Computer Society, Washington (2006) 31. OpenCV, http://www.intel.com/technology/computing/opencv/index.htm 32. Turk, M., Pentland, A.: Eigenfaces for Recognition. J. Cognitive Neuroscience 3(1), 71–86 (1991)

A Structured Methodology of Scenario Generation and System Analysis for Ubiquitous Smart Space Development Ohbyung Kwon1 and Yonnim Lee2 1

School of International Management Kyunghee University Seochun, Ghiheung, Yongin, Kyunggi-do, South Korea [email protected] 2 Research Center for Ubiquitous Business and Services, Kyunghee University Seochun, Ghiheung, Yongin, Kyunggi-do, South Korea [email protected]

Abstract. Ubiquitous smart space (USS) has been regarded as a promising extension of ubiquitous services, and it is currently the subject of world-wide development. In one USS development methodology, scenario development is performed before system analysis and design. However, even though many redundant elements can be found between scenarios and system analysis results, developers have not been taking any structural approaches to join them together for more consistency and eventually higher productivity. Hence, the aim of this paper is to propose a methodology to increase the consistency in the early steps of USS development. To do so, scenario and requirement analysis are integrated in a structured manner. Keywords: ubiquitous smart space, scenario analysis, requirement analysis, structured method.

1 Introduction Advances in ubiquitous technologies have led to the development of many new types of ubiquitous services. Unlike earlier approaches, ubiquitous computing services are defined and used as services that are provided not just in physical spaces but also in logical spaces. This extension of service concept has drawn interest to USS (Ubiquitous Smart Space). USS is an artificial space that is added to ubiquitous computing technology. Recently ubiquitous computing services have developed in USS environment all over the world, for example: in the United States, there are several systems including HP’s Cool Town, Microsoft’s Easy Living, Cisco’s MIE, Intel’s Digital Home, Xerox’s Smart Media Spaces, NIST’s Smart Space, Georgia Tech’s e-Class, and MIT’s Auto-ID Center. In Europe, systems include EU’s 2Wear, Belfast’s MIME, Orange at Home of Orange, and Siemens’ Smart Home. In Japan, systems include the Ubiquitous ID Center’s Ubiquitous ID Technology, Matsushita’s eHII system, OMRON & ODAKYU’s GOOPAS, NTT’s Power Conversion System, and NTTDocomo’s Cmode. In Korea, F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 90–104, 2008. © Springer-Verlag Berlin Heidelberg 2008

A Structured Methodology of Scenario Generation and System Analysis

91

systems include ETRI’s u-Post, SKT’s Moneta, Samsung Electronics’ Digital Home, and CVnet’s Cyber Village. Most of these USSes are proposed by feasible service scenarios that involve various ubiquitous computing elements, technologies and products. In the first stage of development, they write a service scenario that shows a possible future lifestyle, and carries out a requirement analysis through that scenario. From this, the developers design and implement the USS. USS is totally different from the previous physical space so these services also deploy very differently. Therefore, the scenario method that is very widely used in future research has been recognized as an effective tool in developing future services. The scenario method was proven very useful for eliciting USS service idea [1]. Moreover, the scenario method is useful for explaining new ideas to another developer or to capture requirements for system implementation. Hence, the scenario method is widely used as a method to develop USS service [2]. Carnegie Mellon University’s Aura project was started with the goal of achieving invisible computing by Peter Steenkiste in 1999; they first used the scenario method and designed the Aura system to realize that scenario. However, to date, no standardized methodology exists for writing scenarios. Most scenario methods suggest simple guidelines without using all USS development phases. Therefore many scenarios have been developed by brainstorming and the contents have been decided by personal taste or by their organization’s preferences. Brainstorming is a group creativity technique designed to generate a large number of ideas for the solution to a problem. Although brainstorming is very useful for generating creative ideas, it is of course limited if there aren’t sufficient resources. Potential brainstorming problems can also include distraction, social loafing, evaluation apprehension, and production blocking. Existing scenario methods are not related to requirement analyses for software development, which can cause communication problems between scenario writer and requirement analyst. Requirements analysis for developing services has made progress independent of efforts for scenario writing. As a result, one-sided and subjective scenarios that were written without concerning the requirement analysis phase have created problems: double writing, inconsistency between scenario and requirement specification, waste of time and cost. Moreover, these types of scenarios make it difficult to objectively analyze and evaluate scenarios. It is also difficult to validate the degree of scenario reflection on requirements specifications since finding objective standards against which one can evaluate a subjective scenario is difficult. Therefore, this paper proposes the standardized and structured methodology of writing scenarios, and develops an integration methodology for requirement analysis that proceeds automatically and continuously from the written scenario. In this methodology, we focus on a technical-push scenario that is validating the utilitarian of the technology which is developed in the short term. This paper is organized in five sections. First we review previous research, and then Section 3 describes the scenario writing methodology and integration methodology for requirement analysis. A case study of our methodology is proposed in Sections 4. Finally, we provide concluding remarks in Section 5.

92

O. Kwon and Y. Lee

2 Scenario-Based Requirement Engineering Scenario-based requirements analysis methods express system requirements by using scenarios that were elicited through examples of real world experience and invented experiences [3,4]. Scenario-based requirements analysis methods can be simply divided into two methods: ScenIC and SCRAM [5,6]. ScenIC (Scenario in the Inquiry Cycle) proposes a schema of scenario-related knowledge composed of goals, objectives, tasks, obstacles, and actors [7]. Scenarios are composed of episodes and action carried out by actors, who are usually people but may also be machines. Goals are classified into achieving, maintaining, or avoiding states, while obstacles prevent goals from being achieved, or inhibit successful completion of tasks. The method proceeds in a cycle of expressing scenarios in a semistructured format, criticizing and inspecting scenarios in walkthroughs, which lead to refining requirements and specifications and the next cycle. Guidelines are given for formatting scenario narratives and identifying goals, actions and obstacles. Scenario episodes are assessed with challenges to see if goals can be achieved by the system tasks, whether the actors can carry out the tasks, whether obstacles prevent the actors carrying out the tasks, etc. Thus, dependencies between goals, tasks, actors and resources can be checked to make sure the system meets its requirements. Dependency analysis and means-ends analysis, in which tasks and the capabilities of actors are examined to ensure goals can be achieved. In SCRAM (Scenario-based Requirement Analysis Method) [8,9,10], scenarios are used with early prototypes to elicit requirements in reaction to a preliminary design. This approach does not explicitly cover modeling and specification, as this is assumed to progress in parallel, following the software engineering method of the designer’s choice. According to Alistair [5], the method consists of four phases: (1) Initial requirements capture and domain familiarization. This is conducted by conventional interviewing and fact-finding techniques to gain sufficient information to develop a first concept demonstrator; (2) Storyboarding and design visioning; (3) Requirements exploration; (4) Prototyping and requirements validation. This phase develops more fully functional prototypes and continues refining requirements until a prototype is agreed to acceptable by all the users. However, in the legacy scenario-based requirement analysis methods, requirement analysis is actually separated from scenario writing; they support system analysis only after writing scenarios completely.

3 Methodology Overview 3.1 Overall Framework To write scenarios and analyze requirements in USS development environment, an overall framework is proposed as shown in Fig. 1. There are two ways to write a scenario briefly. The first one is a guided approach. In this approach, we first analyze the environment, and then write scenarios based on the results. The other is an open-ended approach that writes scenarios freely without any environment analysis [24]. As previously stated, our research focuses on a technical-push scenario that validates the usefulness of the technology developed in the short term. Hence, we apply a guided approach because we aim to make our scenario


DDefi ne efi ne the the Technol oogy Technol gy MMatri xx atri

DDefi ne efi ne the the SServi ce ervi ce MMatri xx atri

SSketch ketch oout ut the oo the SScenari cenari OOutl ine utl ine

93

GGenerate enerate the the SScenari oo cenari

DDraw raw the the SScenari oo cenari DDescri p oonn escriptiti DDiag iagram ram

GGenerate enerate the the RRequi rem equi rem ent ent SSppeci fifi cati oonn eci cati

Fig. 1. Proposed methodology for generating scenario and requirement specification from technology and service matrix

practical to implement. In this paper, we define technologies, devices, and services that we can use in the scenario at present—this is the environment analysis phase. We then write the scenario based on the results. The scenario writer defines the technology matrix after collecting usable technologies and devices. Technology matrix means a sort of tables which shows summary of available technologies and corresponding devices for making services available. Through a combination of technology and devices in the defined technology matrix, we extract services that we can use in scenarios. We then define a service matrix, which describes in detail the content of each service extracted from a technology matrix, and evaluate its service level to decide if it is available to make scenarios. Based on this evaluation result, we confirm the service that we will use in the scenario. The scenario outline is then sketched out by a final confirmed service. After that, we draw the scenario diagram using the scenario outline and confirmed services. The scenario diagram is a technique that represents scenarios by predefined notations and processes. This diagram consists of a context diagram, a level-0 diagram and a level-1 diagram. Scenarios are generated by transforming a completed scenario diagram according to automatic transformation logic. Requirement specifications are written by transforming a generated scenario according to automatic transformation logic. Meanwhile, we create a scenario dictionary to improve understanding of the scenario reader and requirement specification writer. 3.2 Technology Matrix Definition First, the scenario writer collects technologies and devices that can be used in the scenario and describes brief explanation for each. Then, the scenarios writer may draw up a table which indicates a summary of those technologies and devices as shown in Table 1:

94

O. Kwon and Y. Lee Table 1. Technology Matrix Template

Category Available Technology Possible Device

Name

Explanation

Technology name

Brief explanation for the technology

Device name

Brief explanation for the device

3.3 Service Matrix Definition (1) Extract services Through a combination of technology and devices in the previously defined Technology Matrix, we extract services that we can use in scenarios. Extracted services would take concrete shape by the Service Matrix and be confirmed in the phase of service evaluation. Table 2 is the template for Service Extraction. Among these methodologies, we will focus on the former six steps from need identification to validation on service selection in this paper. Table 2. Service extraction template

Technology and Device Tech. 1

Tech. 2

{

Device 1

Device 3

{

{

Service

Device 4

Service name

Mark ‘{’ Tech. or Device used in the Service.

(2) Define service matrix Service matrix is a table to display in detail the contents of each extracted service. At this time, the functionality of the service that is expressed as an adjective or an adverb in service must be explained as detailed as possible. Table 3 is the template for the Service Matrix. Table 3. Service matrix template

Service

Description

Functionality Enabler

Function


95

(3) Evaluate service level We evaluate the service level of each service based on the checklist. Then we review the result of evaluation with the people who request the scenario. 3.4 Scenario Outline Identifying the topic and background of scenario might proceed as follows. • • • •

Define the value that can be provided for end user through the services used in scenario. Identify stakeholders related to the scenario Define the main character of the scenario and his goal. If needed we can set up the obstacles and complication elements of the scenario. Describe the brief contents for scenario.

Table 4 is the template for identifying the theme and the background of scenario. Table 4. The template for identifying the theme and the background of scenario

Category Name Theme Stakeholder Abstract

Description The name of scenario The theme of scenario Stakeholders are people who affects, or can be affected by the scenario financially Abstract of the scenario

(2) Define the actors and assign their role We define the actors who carry out the service in the scenario and assign their role. An actor might be divided by its property (device, agent, ontology, u-Product). Table 5 is the template for defining the actors and assigning their role. 3.5 Scenario Definition Diagram We draw the scenario diagram based on the sketch of the scenario. This diagram is a technique that represents scenarios by predefined notations and processes. It consists Table 5. The template for defining the actors and assigning their role

Category Device Agent Ontology u-Product

Actor Actor name

Role Actor’s role

96

O. Kwon and Y. Lee Table 6. Components of the scenario definition diagram

Component

Notation

Sequence

Scene

Stakeholder

Data Flow

Data Store

Information Flow

Description It represents the service offered in scenario. One sequence has more than one service. It represents the actual service action that is performed by using the data. It has an actor who is the object of services and an action that is service contents. It represents the stakeholders are people who affects, or can be affected by, the scenario financially It represents the data flow among services or actor’s action. It represents the optional data flow among services or actor’s action. It represents the looped data flow among services or actor’s action. It represents the data store that saves the data. It could have a number of various physical shapes. It represents the information flow between scene and data store. When a scene requests specific information, the data store provides it.

of a context diagram, a level-0 diagram, and a level-1 diagram. Table 6 shows the components of the scenario diagram. (1) Draw the scenario context diagram The scenario context diagram represents the business model of scenario. This diagram explains the relationship among stakeholders who affect, or can be affected by, the scenario financially. Hence, the scope of service is decided through this diagram. The scenario context diagram is drawn like Fig. 2. (2) Draw the scenario level-0 diagram The level-0 diagram represents all services offered in a scenario, and it depicts information flow and the transforms that are applied as data moves from input to output. The scenario level-0 diagram is drawn as a series of steps and will be drawn as in Fig. 3.


Stakeholder 1

Scenario name

Financial relation

97

Stakeholder 2

Financial relation

Financial relation Stakeholder 3 Fig. 2. Scenario contxext diagram Data 1. Service 1

Data 2. Service 2 Data

Fig. 3. Scenario level-0 diagram

(3) Draw the scenario level-1 diagram The Scenario Level-1 Diagram represents the actor and his action in each service. All actors have an Event, Action, and Response. In here, Event is the cause action leads to actor’s action and Response is the result action of the actor’s action. Table. 7 is the template for Action Matrix to organize these actors and actions. Scenario Level-1 Diagram is drawn like Fig. 4. Table 7. The template for action matrix

Actor Actor name

Event

Action

Response

The cause action leads to actor’s action

The actor’s own action

The result action of the actor’s action

98

O. Kwon and Y. Lee

Data flow

1.1 Actor 1

Data flow

1.2 Actor 2

Action1

Data flow

1.3

Data flow

Actor 3

Action2

Action3

Data flow

Data flow

Data Store Fig. 4. Scenario level-1 diagram

3.6 Scenario Generation We generate the scenario by transforming completed Level-1 Diagram according to automatic transformation logic on a per-Sequence basis. Since the transformation logics are highly dependent of language to be used, Korean language has been considered so far. 3.7 Generate Requirement Specifications In this paper, we use the UML that is a modeling language widely used for requirement analysis. Software system can be visualized, described, developed and created a document by the UML. Especially, we choose the Sequence Diagram among 4 models and 8 diagrams of UML that are OMG’s public standard. Sequence Diagram is drawn as follow steps. Step 1. On the basis of Level-1 Diagram, collect all data flows in scenario and give them serial number in order. And then rearrange each data flow, input and output as Table 8. Step 2. Placed all actors appeared in scenario horizontally by using predefined symbols of Fig. 5. And draw a vertical line started from each actor. Step 3. Draw all data flows arranged in Step 1 from Input Actor to Output Actor in order. At this time, represent data flow as an arrow along the vertical line. Table 8. Template for rearranging the data flow in scenario

No 1

Action

Actor

Next Actor

The actor’s own action

Actor who acts

Actor who is affected by the action

2

…

…

…


99

Human

u-product

sensor

event handler agent

ontology

Fig. 5. Symbols for sequence diagram

4 Case Study This case study describes the ‘U-Shopping Aid’ scenario. We applied our methodology to this scenario. ‘U-Shopping Aid’ scenario was proposed to make customers Table 9. Technology matrix Category Technology

Device

Name Agent Technology Semantic Web

Description Offer the information of the product reputation and the payment through the negotiation among agents Provide semantic information through the data written by Ontology Language (OWL) Transfer data between Network AP of shop and u-PDA

uPAN Scalefree Ubiquitous Mobile Object (UMO) UMO

It is a mobile object or device that people have. It can be changed own functions depended on context and share computing resource with smart objects Smart object that customer has. It play an important role in receiving the service Smart display device established in shop or customer’s house Performing a recognizing role among devices

Smart Wall Or AR Table RFID Sensor

Table 10. Extract services Technology and Device uAgent USN PDA

Augmented Reality

Teleconference

Smart Wall

Smart Table

G

G

{G

{G

{G

G

G

{G

G

{G

G

G

{G

G

G

{G

{G

G

G

G

{G

G

G

{G

{G

G

G

G

G

G

G

{G

G

{G

G

Service Customer-aware and product recommendation Virtual put-on service Image sync service Payment service1 Payment service2

100

O. Kwon and Y. Lee Table 11. Service matrix Service name

Definition

Customer-aware and product recommendation

When customer enters, it is aware of that fact automatically. Recommend products that customer would like

Image sync service

Provide the virtual image that customer puts on the chosen clothes using the pictures saved in smart object

Payment service1

It enables to share images among remote spaces and to communicate with each other

Payment service1

When customer makes the decision about buying, it automatically transmits the payment information to the seller and delivery company simultaneously When customer makes the decision about buying, it automatically transmits the payment information to the seller simultaneously

Payment service2

Functionality Function Notify the user’s current location information Mall Agent Detect the embedded chip in UMO automatically Customer Store the user profile information Ontology Shop Agent Make recommendation lists according to the user preference Augmented Display virtual images reality device when the user sees the chosen clothes through AR device AR Table or Display made picture Smart Wall u-PDA Transmit shared images AR Table or Display shared images Smart Wall and transmit the selected information u-PDA Transmit payment information Enabler UMO

u-PDA

Transmit payment information

Table 12. Scenario actors and their roles

Category

Actor Shop Agent

Agent Mall Agent

Sensor

Chip sensor at the Mall entrance

Ontology

Customer info Ontology

Role It offers the optimized service through being aware of various customer need and manage the agent of shops. It is in the shopping mall. It is aware of customer’s context information and transmits it to Shop Agent. It is a RFID chip sensor at the shopping mall entrance. It detects the customer’s smart objects. It stores a customer profile.


101

convenient for their shopping more practically. ‘U-Shopping Aid’ scenario consists of customer-aware and product recommendation services, virtual put-on service, image sync service and payment service. Let’s suppose that the Technology Matrix could be defined as Table 9. Then five services in Table 10 are extracted and Service Matrix is defined as listed in Table 11, which is based on Table 9. Then the actors and assign their role for our scenario is defined as shown in Table 12. Lastly, Scenario Context Diagram and Scenario Level-0 Diagram are prepared as Appendix A.

5 Conclusion Recently, USS developers are under increasing pressure due to the increased importance of making scenario and requirement analyses; hence the increasing need for structured methodologies. However, so far there has been no standardized methodology for writing scenarios. Most scenario methods suggest simple guidelines that do not take into consideration all USS development phases. Therefore many scenarios have been developed by brainstorming and the contents have been decided by personal taste or by the organization’s favor. Moreover, existing scenario methods do not relate to requirement analysis for software development. This fact creates communication problems between scenario writer and requirement analyst. It creates problems like double writing, inconsistency between scenario and requirement specification, waste of time and cost, and so on. These kinds of scenarios make it difficult to analyze and evaluate scenario objectively. Therefore, we propose the standardized and structured methodology of writing scenario and develop the integration methodology for requirement analysis that is proceeding from written scenario automatically and continuously. We are applying our methodology to the ‘U-Shopping Aid’ service development supported by a Korea governmental project.

Acknowledgment This research is partially supported by the ubiquitous Computing and Network (UCN) Project the Ministry of Information and Communication (MIC) 21st Century Frontier R&D Program.

References 1. Ranta, M., Asplund, H.: Utilizing scenarios in service idea generation- a case of distant participation for a seminar. In: Proceedings cf COST269 Conference, pp. 379–386 (2003) 2. Anton, A.I., Potts, C.: A Representational Framework for Scenarios of System Use. Requirements Eng. 3, 219–241 (1998) 3. Sutcliffe, A.: Scenario based requirement analysis. Requirements Engineering Journal, 48– 65 (1998)

102

O. Kwon and Y. Lee

4. Plihon, V., Ralyte, J., Benjamen, A., Maiden, N.A.M., Sutcliffe, A., Dubois, E., Heymans, P.: A Reuse Oriented Approach for the Construction of Scenario Based methods. In: Proc. Int’l Software process Assoc. Fifth Int’l Conf. Software Process (ICSP 1998), Chicago, pp. 14–17 (1998) 5. Sutcliffe, A.: Scenario-Based Requirements Engineering. In: Proceedings of the 11th IEEE International Conference, Montery Bay, USA, pp. 320–329. IEEE Computer Society Press, Los Alamitos (2003) 6. Misra, S., Kumar, V., Kumar, U.: Goal-oriented or scenario-based requirements engineering technique- What should a practitioner select? In: IEEE CCECE/CCGEI, Saskatoon, pp. 2290–2291 (2005) 7. Potts, C.: ScenIC: A Strategy for Inquiry-Driven Requirements Determination. In: 4th IEEE International Symposium on Requirements Engineering, pp. 58–65. IEEE Computer Society Press, Los Alamitos (1999) 8. Sutcliffe, A.G.: Scenario-Based Requirements Analysis. Requirements Engineering 3, 48– 65 (1998) 9. Sutcliffe, A.G.: User-Centred Requirements Engineering. Springer, London (2002) 10. Sutcliffe, A.G., Ryan, M.: Experience with SCRAM: A Scenario Requirements Analysis Method. In: IEEE International Symposium on Requirements Engineering: RE 1998, pp. 164–171. IEEE Computer Society Press, Los Alamitos (1998)

Appendix A. Example Results (1) Context Diagram COEX Mall

Payment Context-aware info Info & payment User (A)

U-shop service & delivery Service

Info & payment U-shop

U-Shopping Aid Customized i Purchasing info & payment Delivery Info

Distribution and delivery Service Provider


(2) Scenario Level-0 Diagram [Scenario name] Aware & recommend <Scenario Diagram>

[Scenario name] Virtual Put On

[Scenario name] Image Sync

[Scenario name] Payment and delivery

103

104

O. Kwon and Y. Lee

(3) Data Flows in Scenario No. 1 2 3 4 5 6 7 8 9

18

19 20 21 22 23

Data flow A enters Send A’s context info.

Actor Chip sensor at Mall Entrance Mall Agent Customer Info. Ontology Mall Agent

Request for A’s info. Send A’s info. Send A’s Context info. Ask A’s info. Shop Agent Send A’s info. Customer Info. Ontology Provide product list to Shop Agent recommend Send a message to A’s UMO suggest the VPO service Scenes from 10 to 17 are omitted here. Display the created IS service device augmented reality to share with family The info. about the AR Table family choices Send the family A’s UMO choices Decision to purchase A and pay Info. about payment A’s UMO Info. about payment Shop Agent and delivery

Next_Actor Mall Agent Customer Info. Ontology Mall Agent Shop Agent Customer Info. Ontology Shop Agent A’s UMO A

AR Table

A’s UMO A A’s UMO Shop Agent A

Capturing Semantics for Information Security and Privacy Assurance Mohammad M.R. Chowdhury1 , Javier Chamizo2 , Josef Noll1 , and Juan Miguel G´ omez2 1

UniK-University Graduate Center Post Box 70, N-2027 Kjeller, Norway {mohammad, josef}@unik.no 2 Escuela Politécnica Superior Universidad Carlos III de Madrid Avda. de la Universidad 30, Leganés, Madrid, Spain [email protected], [email protected]

Abstract. Security and privacy assurance is indispensable for ubiquitous access to information and resources. This paper focuses on the security and privacy provisions in a restricted organizational environment through access control mechanism. It includes the representation of the semantics of an organization and its access control mechanism exploiting the Web Ontology Language. The system controls access to the resources of an organization through differential access privileges. These are formulated based on the roles of the individuals, and the projects and departments they belong to. Instead of explicit definitions, some additional facts of the mechanism are inferred by executing semantic rules using the Jess rule engine over the designed ontology. These information are then passed back to the ontology to enrich it. The ontology is designed to cope with the organization restructuring with minimal efforts.

1

Introduction

Ubiquitous computing and connectivity together with extensive diffusion of portable devices allow users to access information/resources/services anytime and anywhere even when they are on the move. However, these access scenarios demand security and privacy assurance which is not a trivial job in today’s increasingly connected but dynamic systems. In this regard, Professor Dr. Eugene Spafford said [1], The only truly secure system is one that is powered off, cast in a block of concrete and sealed in a lead-lined room with armed guards - and even then I have my doubts. This paper focuses on the security and privacy provisions in a restricted organizational environment through access control mechanisms. Access control in distributed and dynamic systems is crucial for secure service access. It is included F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 105–118, 2008. c Springer-Verlag Berlin Heidelberg 2008

106

M.M.R. Chowdhury et al.

as one of the main sections in ISO/IEC 27000 series1 , an information security standard published by the ISO and the IEC. We believe that the capabilities of semantic technologies can contribute to mitigate these problems. The impact of Semantic Web technology is wide ranging. The Project10X (a consulting firm)2 study found that more than 190 companies including Adobe, Google, HP, Oracle and Sony are involved in developing Semantic Web based tools. But making it easier to comb through online data carries security implications. Among the challenges of security issues, policy-awareness and access control to Web resources play a major role, particularly given that these are two of the most significant requirements of information access and exchange. Design and maintenance of access control constraints in organizations are challenging problems as company structure, roles, user pools, security requirements are always changing. Conceptual organizational semantics and its access control mechanisms are fomally represented in an ontology using Web Ontology Language. The company system controls access to the resources of the organization by providing differential access privileges. In this ontology, these privileges are formulated based on the roles of the individuals. In addition, it considers which projects or departments they belong to. The proposed solution is designed to cope with the dynamic nature of an organization. This work is an extension of our previous work [2], where the concepts were so static that it could not reflect the organizational changes. This paper deals with a complex situation where an employee plays multiple roles across different departments and projects. The paper is organized as follows. The next section discusses the problem statements and our use case scenario. Section 3 briefly describes the Semantic Web technologies. The ontology representing organizational semantics are described in section 4. In Section 5, we illustrate the access control mechanism, the processes and results of inference based on the proposed ontology. In the next section, proposed solution is evaluated in the context of organizational restructuring. Section 7 contains overview of the related works and the paper concludes summarizing the paper and briefly stating the future works.

2

Problem Statement: Information Security and Privacy Assurance

Nowadays people in business organizations increasingly work in project oriented environments. Some of the projects deal with company sensitive information which should not be leaked to unauthorized employees. The project members come from different departments. They don’t enjoy the same rights or privileges within a project environment. This is more prevalent while accessing resources owned by the departments or projects. There are situations where a person 1 2

ISO 27002 - The Information Security Standard, http://www.standardsdirect.org/iso17799.htm [accessed on Jan. 4, 2008] The Project10X Special Report, http://www.semantic conference.com/semanticwave.html

Capturing Semantics for Information Security and Privacy Assurance

107

holds multiple roles and privileges. Employees with different levels of privileges are expected to access resources through the Intranet or Internet. Fig. 1 illustrates the use case scenario which describes a specific organizational environment. It deals with the roles like employee (department in general), supervisor, project leader (PL) and project member (PM). Telenor R&I (Department A) and its planning department (Department B) both involve in project release 7 and 8. Release 9 only resides in planning department. Each of the departments and projects has its own resources. A person has multiple roles like Josef Noll is not only a supervisor (Department A) and project leader (Release 9) but also a project member (Release 7). He should have access to corresponding department and project resources where he involves in. So, access rights depend on one’s role into the respective departments and projects. Following are some of the examples of restricted access scenarios, 1. Supervisor is the head of a department. Department owns some resources (administrative resources, documents, deliverables). He can read and write the documents. He can edit its administrative resources and give final approval to the deliverables. Supervisor can also monitor status of department’s employees who work in different projects. 2. Department’s employees will have only read and write privileges to its documents. 3. Departments participate in different projects. Project leader leads a project. He can read and write project’s documents. He can edit its administrative resources and give final approval to the project deliverables. 4. Besides leader, projects have members. They can only read and write project’s documents. Therefore, the architecture manages access to resources not only based on the roles but also based on the involvement in organizational divisions (departments, projects). Access scenarios of the use case are described in table 1.

Fig. 1. The use case scenario

108

M.M.R. Chowdhury et al. Table 1. Roles and privileges to access corresponding resources Employee

Role

Josef Noll

Supervisor

Project Leader

Hans Christian

Project Member Supervisor

Project Leader

3

George Kalman

Employee Project Member

Erik Swansson

Employee Project Member

Privilege

Access to Resources

Administrator Admin. Dept.A Final Approval Deliverables Dept.A Read Write Documents Dept.A Administrator Admin. Rel9 Final Approval Deliverables Rel9 Read Write Document Rel9 Read Write Documents Rel7 Administrator Admin. Dept.B Final Approval Deliverables Dept.B Read Write Documents Dept.B Administrator Admin. Rel7&Rel8 Final Approval Deliverables Rel7 &Rel8 Read Write Documents Rel7 &Rel8 Read Write Documents Dept. A Read Write Documents Rel8 Documents Rel9 Read Write Documents Dept. A Read Write Documents Rel8

Representation of Organizational Semantics

We advocate that access control solutions should adopt semantic technologies as the key building blocks for supporting expressive representation of organizational semantics and reasoning. This section briefly describes the technologies and clarifies our claims. 3.1

Using Ontology

A common format of information and data representation will likely never be achieved. Efficient management of information and data is possible by capturing the common and understandable meaning of them formally and unambiguously [16]. Ontologies [15] are the cornerstone technology of Semantic Web, providing structured vocabularies that describe a formal specification of a shared conceptualization. It is used to capture the knowledge about a domain of interest in the form of concepts and their relationships. It permits the description of organizational structures, roles, privileges and resources at different levels of abstraction and support reasoning about both the structure and the properties of the elements that constitute the system. We believe that designing a consistent ontology based on a sound conceptual foundation is worthwhile because it can be reused in various organizations to control access to their resources.


3.2

109

Introduction to the Web Ontology Language

Among the different ontology languages, we are focusing on the Web Ontology Language (OWL3 ) suggested by the World Wide Web Consortium (W3C). It is a markup language that builds on RDF4 and RDF Schema. OWL is chosen because it provides more vocabularies for describing concepts and properties (e.g. relations between concepts, cardinality, equality, richer typing of properties, etc) than that supported by XML, RDF, and RDFS. There are three species of OWL: OWL Lite, OWL DL and OWL Full and these are designed to be layered according to their increasing expressiveness. 3.3

Description Logics for OWL

Between the three different sub-languages OWL offers, we decided to use OWL DL. It is based on Description Logics (hence the suffix DL). These are the decidable part of First Order Logic5 and are therefore amenable to automated reasoning. Though OWL DL lacks in expressivity power compared with OWL Full, it maintains decidability6 and regains computational efficiency. The computational efficiency is an important feature since it is expected to support scores of relations. The mechanism is supposed to evaluate and grant permissions to access resources, it seems necessary to add reasoning support with it. In order to achieve more expressivity and decidability, we use Semantic Web Rule Language (section 5.1) which is designed as an extension of OWL DL.

4

Descriptions of Organizational Semantics

We assumed that company employees are already authenticated to the system through some secure means. Ontology models the organizational structures described in section 2. 4.1

Defining Concepts through Classes

Fig. 2 illustrates the proposed ontology in conceptual level. OWL classes are the concrete representations of concepts. In the proposed ontology, Identity class defines the identities of the company employees. We specify Company, Department and Project as subclasses of Work Unit in order to avoid defining explicit relationships between department/project and roles. We follow the set theory (eq. 1) while defining the class hierarchy of Role considering the fact that supervisor of 3 4 5 6

OWL Overview: http://www.w3.org/TR/owl-features/ RDF builds on URI and XML technologies. The specifications provide a lightweight ontology system. First Order Logic (FOL), http://en.wikipedia.org/wiki/First-order logic Logics are decidable if computations/algorithms based on the logic will terminate in a finite time.

110


Fig. 2. The ontology: class, property, instance

a department is also an employee of it. The same is true for project leader and member. Class hierarchy is a critical issue in inheritance of properties. {Supervisor, P rojectLeader, P rojectM ember} ⊆ Employee ⊆ Role

(1)

We divide the resources into subclasses, administrator resources and web resources. Web resources are further divided into documents and deliverables. These resources are related to appropriate privileges. Privileges are designed in accordance with the individual’s roles in the organization. In this paper, role is the positional hierarchy of employees. Through this, individuals are restricted to access correct resources. As for example, administrative resources are related to the Admin Privilege to ensure that only the roles having administrative privilege can access these. 4.2

Realizing Concepts through Instances

OWL classes are interpreted as sets that contain instances or individuals. Fig. 2 illustrates the instances of classes in ellipses. Instances of the identities are defined here simply as their names. Admin, Final approval and Read write instances of privilege are added. But new instances can be added (section 6) whenever necessary. Instances of the resources are added which correspond to the resources owned by the departments or projects. Instances to the subclasses of Role are added in accordance with the individual roles to realize multiple roles of a person. As for example, Sup Josef instance of Supervisor corresponds to the supervisor role of Josef Noll. Similarly, PM Josef corresponds to the project member role of Josef Noll.


4.3

111

Defining Relations through Properties

Properties are the binary relations between the two things, more specifically between the instances of classes. A property relates instances from the domain with the instances from the range. Syntactically, domain links a property to a class and range links a property to either a class or a data range. Due to the class hierarchy (eq. 1) and domain and range specifications, subclasses inherit the relationships between the respective classes. Fig. 2 also provides the properties and their relationships with classes. rolePlaysIn property specifies the fact that in which specific Work Unit (departments/projects) a Role instance plays its role. Ontology is supposed to answer who can see/access which resources and hasVisibilityOf property defines this situation. isSupervisorOf explains who is the supervisor of whom in a department. The relationships of hasVisibilityOf, isSupervisorOf are not defined explicitly. These relationships of the properties are filled in through the inference process.

5

Access Control by Enhancing the Expressivity of OWL

Access control is achieved by enhancing the expressivity of OWL through the inference process. This section describes the access control logic and expressivity needs, making a review of the language used and its benefit to the goals pursued. 5.1

Introduction to the Semantic Web Rule Language

The expressivity provided by the OWL is limited by tree like structures [17]. This means that knowledge cannot be inferred from indirect relations between the entities, however the solution spends most part of his power in inferring indirect relationships that will determine whether a subject has access to a resource or not and which are its privileges over it. Hierarchical structures as defined before and inherent relationships between working units and hierarchies of resources are a perfect field where inference can extract these knowledge. We did inference through the rule support over the ontology. and used Semantic Web Rule Language (SWRL7 ) which pretends to be a complimentary feature of OWL. SWRL is roughly the union of Horn Logic and OWL. As any Horn Logic based language, rules are defined as a set of precedent and consequent states. 5.2

Inference Results

Objects of the properties, hasVisibilityOf and isSupervisorOf are filled in through the inferred knowledge from executing the rules. We use Jess rule engine to run the rules. First, OWL ontology and SWRL rules are transferred to Jess. Running the engine then initiates the inference process, generates knowledge as Jess facts. This inferred knowledge can be used by the external interface or can optionally be passed back to the ontology to enrich it. All these actions are user-driven. Rules are formulated using the SWRL as follows, 7

The Semantic Web Rule Language, http://www.w3.org/Submission/SWRL/

112


– Rule1: Over which resource a Role has Visibility/Access? Employee(?Em) ∧ roleP laysIn(?Em, ?X) ∧ hasP rivilege(?Em, ?Y ) ∧ belongsT o(?Z, ?X) ∧ needP rivilege(?Z, ?Y ) −→ hasV isibilityOf (?Em, ?Z)

– Rule2: Who is supervisor of whom? Dept Employee(?DepEm) ∧ hasRole(?Y, ?DepEm) ∧ Department(?Dep) ∧ roleP laysIn(?DepEm, ?Dep) ∧ Corporate Identity(?ID) ∧ Supervisor(?Sup) ∧ hasRole(?ID, ?Sup) ∧ roleP laysIn(?Sup, ?Dep) −→ isSupervisorOf (?ID, ?Y )

Fig. 3 and 4 illustrates the inference results of rules execution. All the relationships in the ontology have not been explicitly defined. As for example, hasVisibilityOf relationship of project leader Hans (PL Hans) has not been defined (circled in fig. 3). The figure displays that 25 relationships are inferred, which answers the resources over which the roles have the required visibility/access (Rule 1). Our investigation shows that the knowledge are inferred as expected. The inference results are exported back to the ontology to fill these empty relationships (fig. 3 shows that 25 relationships are transferred back to the OWL knowledge). From Tab. 1, it is evident that Josef Noll is the supervisor of George Kalman, which was not explicitly defined. Execution of Rule 2 shows the similar result (Fig. 4). This can be used in a special situation, when supervisor wants to check the status of an employee of his department to a project where he is not involved.

6

Evaluation

The proposed solution is more maintainable since there exist a general schema for any organization that can easily be adapted, thanks to its expressivity capacity nearer to human understanding. Expressivity and inference capacities avoid the inclusion of redundant information in the ontology. The proposed ontology can reflect the organization changes of a company with minimum efforts. Now, we are going to describe a situation of this. A new department, Audit has been created in the company. Roman Stanek and Peter Johansson has joined as the supervisor and employee (auditor) of the department respectively. The department has budgets, and Peter audits the budgets. Audit department prepares the audit reports of company. It is expected that auditor (Peter) as well as supervisors of the departments can only check department’s budget. Roman and Peter only have access to the company audit reports that belong to the Audit department only. As a supervisor of Audit department, Roman can give final approval to these. The corresponding actions to reflect these changes into the ontology are described in the following points. Through this, we are going to evaluate the strength of the proposed ontology. – Add new identity instances: Roman Stanek and Peter Johansson. – Add new department instance: Audit


Fig. 3. Inferred results executing Rule 1

Fig. 4. Inferred results executing Rule 2

113

114


Fig. 5. Inferred relationships when Rule 1 is executed

Fig. 6. Inferred relationships when Rule 2 is executed

– Add new subclass of Resource: Budget & Audit. It contains three instances: audit report of company (Audit Company), BudgetDept.A and BudgetDept.B. A new subclass of resource is created because it needs a different privilege to support its privacy requirements. Besides, the department contains an administration resource (AdminResAuditDept). – Add new subclass: Auditor within the class tree: Role-CompanyEmployee. Add Auditor Peter instance of it. – Add new instance of Supervisor role: Sup Stanek.


115

– Add a privilege instance Admin Budget to ensure only auditor and supervisor have access to budget. This instance is related to Auditor Peter and Budget Dept.A & Budget Dept.B. – Fill up all the corresponding relationships. – If we execute Rule 1, new relationships are inferred with hasVisibilityOf property. These are exported back to the ontology to fill in (circled in the figure) the empty relationships. Fig. 5 shows these new relationships for instance of auditor role Auditor Peter. As expected, Peter can only have access to company audit reports and Budgets of Dept.A and B. Similarly, fig. 6 shows the fact that Roman Stanek is the supervisor of Peter Johansson. It requires only few changes in the ontology. Among these changes, only two new subclasses have to be created. Otherwise, all the remaining additions are in the instances which is quite apparent. Conceptually, proposed ontology can follow the organizational changes. One of the ways of simplifying access rights management is to store the access control matrix using access control list (ACL). Though ACLs are simple to implement, its management steps are quite tedious especially when it is required to revoke somebody’s privilege. The proposed ontology provides a simple mechanism to revoke somebody’s role or privilege. One can simply delete the relationship between the role instance and the privilege instance to withdraw the privilege and afterward can delete the corresponding role instance to repeal the subject’s role entirely. Among the few disadvantages, it is computationally expensive and decidability not guaranteed by SWRL. In addition, XML processing is slower than that of database. But ontologies can be stored in the relational databases or even be mapped from them.

7

Related Works

The significance of adding privacy-enhancing technologies (PET) in virtual community networks is overwhelming [5], [6]. Information security and privacy requirements are more evident in a business environment. These issues are handled in this paper through access control mechanisms in the context of project oriented corporate working environment. We considered the concept of Role Based Access Control (RBAC) as a part of our access control mechanism. Sandhu introduced RBAC in [4] with early multiuser computer systems, where users’ access permissions are associated with roles, and users are made members of appropriate roles. A major advantage of RBAC is its ability to constraint access based on the concept of separation of duties which significantly simplifies the management of permissions. In the proposed solution, access control also depends on how the organizational structures. There are two types of RBAC constraints: dynamic and static. Authors in [3] described an approach of RBAC with dynamic constraints through an automated reasoning technique. Though we focused on static constraints on roles, rules were included within the ontology to infer new knowledge which can be passed back to the ontology. Through this verification of access control constraints defined

116


in the ontology are also achieved. The proposed solution can also adapt to ever changing company structure with less effort. Ubiquitous connectivity not only permits users to access resources anytime and anywhere but also complicating its control due to user/device mobility. To integrate this pervasive behavior, a context-aware access control framework has been developed for secure collaborations in pervasive computing environments [18]. It followed two main design principles: context-awareness to control resource access and to enable dynamic adaptation of policies based on context, and the use of OWL DL for context/policy specification. The authors considered location and time of the meeting as the context information collected dynamically. We used static contexts like, which departments/projects someone is involved though these were predefined. Among the access control technologies, ACL is widely used. The semantics of ACLs have been proved to be insecure in many situations. Instead of maintaining a centralized ACL, a trust based group has been proposed to delegate access rights in [7], [8] where FOAF (friend of a friend) [9] based social networks acted as a mean for the delegation. A private key based signature scheme was proposed to ensure the privacy of networks and users. But it requires secure distribution and maintenance of keys. A similar concept of trust or reputation has also been used by [10] to create and access communities. Distributed trust management approach is considered as one of the main components to secure the Semantic Web [11]. They intended to provide access to community resources and privacy solutions only by means of trust/reputation management. But access to sensitive business resources based on trust mechanism does not provide adequate security in business contexts. In addition, trust is affected by various factors and therefore difficult to quantify. In [12], authors presented an approach to reduce the inefficiencies of the management (coordination, verification and validation, and enforcement) of many role-based access control policies and mechanisms using OWL. They focused on the representation of XACML (eXtensible Access Control Markup Language) policies in DL. In [13], Kolovski introduces an algorithm for the translation of a subset of XACML into DL with the goal of offering relevant analysis services using an OWL DL reasoner, Pellet[14]. In this paper, we also formalize the organizational semantics, roles and access privileges using OWL DL. Finini in his paper [11] also proposed using OWL for constructing ontologies which define policies/privileges.

8

Conclusions

In this paper, we addressed the security and privacy challenges of project-based organizations through access control mechanism exploiting Semantic Web technologies. In this regard, we developed an ontology to represent the conceptual structure of an organization and the roles of individuals. In the ontology, all the relationships between the entities have not been defined explicitly. Semantic rules facilitated expressing these additional knowledge. Jess rule engine executed


117

these rules and new facts were transferred back to the ontology to enrich it and check its validity. Apart from these, we evaluated the inference capabilities of the proposed solution by restructuring the organization. As the proposed solution is based on centralized architecture, the future architecture should consider the scalability issues of it especially for a system of big enterprise. Our ultimate goal is to integrate this solution with a web application which controls access to resources.

References 1. Spafford, E.H.: director of the Purdue Center for Education and Research in Information Assurance and Security, Selected Quotes [accessed on January 4, 2007], http://homes.cerias.purdue.edu/∼ spaf/quotes.html 2. Chowdhury, M.M.R., Noll, J., Gomez, J.M.: Enabling Access Control and Privacy through Ontology. In: 4th International Conference on Innovations in Information Technology (Innovations 2007), Dubai, UAE (2007) 3. Dury, A., Boroday, S., Petrenko, A., Lotz, V.: Formal Verification of Business Workflows and Role Based Access Control Systems. In: International Conference on Emerging Security Information, Systems and Technologies (SECUREWARE 2007), Valencia, Spain (2007) 4. Sandhu, R.S., Coyne, E.J., Feinstein, H.L., Youman, C.E.: Role-Based Access Control Models. IEEE Computer 29(2), 38–47 (1996) 5. Chewar, C.M., McCrickard, D.S., Carroll, J.M.: Persistent virtual identity in community networks: Impact to social capital value chains. Technical Report TR-03-01, Computer Science, Virginia Tech (2003) 6. Walters, G.J.: Privacy and Security: An Ethical Analysis. Computers and Society, 8–23 (2001) 7. Kruk, S.R., Grzonkowski, S., Gzella, A., Woroniecki, T., Choi, H.-C.: D-FOAF: Distributed Identity Management with Access Roghts Delegation. In: 1st Asian Semantic Web Conference, Beijing, China (2006) 8. Kruk, S.R., Gzella, A., Grzonkowski, S.: D-FOAF Distributed Identity Management based on Social Networks. In: Demo session of ESWC 2006 (2006) 9. FOAFRealm project, http://www.foafrealm.org/ 10. Choi, H.-C., Kruk, S.R., Grzonkowski, S., Stankiewicz, K., Davis, B., Breslin, J.G.: Trust Models for Community-Aware Identity Management. Identity. In: Reference and the Web IRW 2006, WWW 2006 Workshop, Scotland, May 23 (2006) 11. Finin, T., Joshi, A.: Agents, Trust, and Information Access on the Semantic Web. ACM SIGMOD 31(4), 30–35 (2002), Special Issue: Special section on semantic web and data management 12. Smith, M.A., Schain, A.J., Clark, K.G., Griffey, A., Kolovski, V.: Mother, May I? In: OWL-based Policy Management at NASA European Semantic Web Conference 2007, ESWC 2007 (2007) 13. Kolovski, V., Hendler, J., Parsia, B.: Analyzing Web Access Control Policies. In: 16th International World Wide Web Conference, WWW 2007, Alberta, Canada, May 8-12 (2007) 14. Pellet, an OWL DL reasoner, http://pellet.owldl.com/

118


15. Fensel, D.: Ontologies: A Silver Bullet for Knowledge Management and Electronic Commerce. Springer, Heidelberg (2001) 16. Berners-Lee, T., Hendler, J., Lassila, O.: The Semantic Web. Scientific American (May 2001) 17. Motik, B., Sattler, U., Studer, R.: Query Answering for OWL-DL with Rules. In: International Semantic Web Conference 2004, pp. 549–563. Springer, Heidelberg (2004) 18. Toninelli, A., Montanari, R., Kagal, L., Lassila, O.: Semantic Context-Aware Access Control Framework for Secure Collaborations in Pervasive Computing Environments. In: International Semantic Web Conference 2006. LNCS, pp. 473–486. Springer, Heidelberg (2006)

A Framework for Context-Aware Home-Health Monitoring Alessandra Esposito, Luciano Tarricone, Marco Zappatore, Luca Catarinucci, Riccardo Colella, and Angelo DiBari University of Salento, Lecce, Italy {alessandra.esposito,luciano.tarricone, luca.catarinucci}@unile.it

Abstract. This paper presents a proposal for a context-aware framework. The framework is organized according to a general purpose architecture, centred around an ontological context representation. The ontology provides the vocabulary upon which software agents interoperate and perform rule-based reasoning, in order to determine the system response to context changes. The system components and their coordinated operations are described by providing a simple example of concrete application in a home-care scenario. Keywords: context-aware, ontology, rule, logic, agent, ubiquitous, health.

1 Introduction Progress in mobile devices, wireless networks and software technologies is bringing healthcare a new generation of systems, which are known under the name of pervasive and context-aware systems. The term ‘pervasive’ [1] refers to the seamless integration of devices into the user’s everyday life. Appliances vanish into the background to make the user and his tasks the central focus rather than computing devices and technical issues. Context-aware systems adapt their operations to the current context without explicit user intervention. These kinds of applications pose several technological challenges. First of all, a pervasive application is made up of heterogeneous and dispersed components which must be able to interoperate in a transparent manner. Moreover, in order to adapt to context variations, the system must elaborate raw data sensed by context sources, such as sensors, and extract high level context information from them. Both requirements lead to the identification of the core component of the system: a robust, reusable and sharable context representation. However, the definition of a shared context model provides the basis for 1) allowing the system components to cooperate and build together the system reaction to incoming events, 2) structuring sensed data in a meaningful and interpretable form. Currently, there is no standard modelling option for context [2]. Therefore, available frameworks adopt proprietary approaches. Among them, ontology-based ones [3,4,5] are more and more recognized as the most promising [6], as they provide a rich formalism for specifying contextual information and are reusable and sharable. The adoption of ontologies is encouraged by their capability of being embedded within multi-agent and rule-based systems. Indeed, designing the application in a multi-agent fashion allows one to organize it around the F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 119–130, 2008. © Springer-Verlag Berlin Heidelberg 2008

120

A. Esposito et al.

coordinated interaction of autonomous reasoning entities, which wrap the “natural” components of the pervasive system (such as sensing devices and consumer services). Rule-based logic supports agents in implementing advanced reasoning and in deriving high-level concepts from sensed information thus opening the application to sophisticated adaptation and pro-active behaviour. This work proposes a framework for context-aware pervasive systems built around the three above mentioned technologies: ontology representation, multi-agent paradigm and rule-based logic. The amenability of these technologies for contextaware pervasive environments has been demonstrated in a number of recent works. Among them, we recall CoBrA [5], an agent-based framework for intelligent meeting rooms centred around an OWL ontology. Gu et al. [6] propose SOCAM which is an ontology-based middleware for context-aware services in smart space domains. An ontology-based context model and context reasoning for ubiquitous health-care is presented in [4]. Health-care is the focus of Paganelli and Giuli [3] work as well, which describes a system built around OWL and rule-based inference. A further impulse to the integration of the three technologies is provided with our work, which is the result of an effort to enhance their interoperability. As a result, the ontology provides the knowledge codification needed to support both agent reasoning and communication. The effectiveness of the adopted model is shown by using a simple and concrete example in a home-care scenario. The paper is organized as follows. Section 2 introduces the general-purpose framework and its prototypal implementation. Section 3 is centered around context modeling. First the ontology representation, then the rule-based reasoning are illustrated. Section 4 focuses on how agent-based reasoning supports context elaboration. Section 5 focuses on data collection strategies. It describes “S-tag”, a novel low cost device enabling the integration of sensor networks with RFID systems. A home-health scenario is adopted throughout the entire paper as specific example of application and practical result.

2 System Architecture The architecture of our system (Fig. 1) follows a widely accepted abstraction [7], according to which context-aware systems are organized into three layers: context sources. Context management middleware and context consumer level Context sources include entities providing raw context data. They are conceptually partitioned into two groups: physical and virtual sources [8]. Physical sources include all hardware devices able to sense context data, such as RFID, sensors, positioning systems, etc. Virtual sources include software services able to gather context data, such as GUIs for user preferences input, databases, etc. Such data must be elaborated in an “intelligent” manner, so that the overall system reacts properly to context changes. This requires the availability of a machine-interpretable representation of context and of software components (agents) able to suitably process such knowledge. Both of them are conceptually situated at the intermediate level of the system architecture, the so-called middleware layer, as they provide the core building blocks of a context-aware application. Agents interoperate with one another thanks to the availability of a unified model of reality. Their behaviour is strongly influenced by

A Framework for Context-Aware Home-Health Monitoring

121

data provided by context sources and substantially determines the activities to be performed at the highest layer. Indeed, the context consumer layer includes all the entities, such as mobiles, Web interfaces, laptops, which interact with final users in response to meaningful context changes, thus determining the behaviour of the context-aware application as a whole.

Context Consumer GUIs

Applications

Mobiles

PDAs

PCs

…

Context Management Middleware Context Model

Inference Engine

JADE

APIs

…

Context Sources Physical Sensors

Virtual Sensors

Fig. 1. The three-layer system architecture. The lowest layer includes physical and virtual sensors, i.e. devices and software entities forwarding context data. The middle level hosts middleware components, such as software development kits and APIs. The context model is the core of the system and belongs to the middle level as well. End user devices and applications are positioned at the highest level.

The prototypal implementation of the framework is drawn in Fig.2. As shown in the figure, the distributed system is made up of a team of interoperating agents, which share a common knowledge representation and reason through an inference engine. The agents are dispersed over several nodes and play different roles. As explained in Section 4, Context Provider Agents (CPA) wrap context sources and are in charge of converting raw context data into high level context information. CPAs cooperate with Context Interpreter Agents (CIA) which are responsible for managing high level context information and of identifying the set of actions to be triggered as a consequence of an emergence. Finally, Context Consumer Agents (CCA) forward the message/signal describing the alarm situation to the most suited destination. Input data are provided by a physical context source obtained by integrating sensors with an RFID device (see Section 5) and by a virtual context source containing static information. In the following sections, the system components and their way of operating are described, with reference to a home-care scenario.

122

A. Esposito et al.

S-tag

CPA

CN3

CPA

CPA Agent Inference Engine

CCA

CN1

CIA CN2

Agent Behaviours Fig. 2. System Prototype. The system was implemented on three nodes (CN) connected by a local area network. It follows a multi-agent paradigm. Context Provider Agents (CPA) filter and integrate data coming from physical and virtual sensors, thus converting raw context data into high level context. Context Interpreter Agents (CIA) process high level context information to identify the actions and the actors best suited for managing an emergency. Context Consumer Agents (CCA) forward the alarm information to the final destination.

3 Context Representation An application is context-aware if it is able to adapt its behaviour to context by suitably reacting to context variations. In other terms, a context-aware application must exhibit a sort of “situation awareness”, i.e. it must manage and interpret context and its real-time evolution. Moreover, a context-aware application is generally made up of several components, possibly deployed on dispersed nodes, which need to interact with one another. Therefore it is fundamental for such components to share a common representation of context knowledge. These premises cast a twofold approach to context modelling. First of all, the fundamental entities, objects and relations of a situation must be represented. This is needed in order to ground system knowledge on a common basis and provide a unified concept taxonomy to the several components interacting in the application framework. Secondly, context changes originated by meaningful data variations must originate recognizable “high-level” events, so that situation switching can be interpreted and suitably managed by the system. In other terms, the system must be enabled to deduce high-level, implicit context from the low-level, explicit context directly acquired from sensors. This operation often requires reasoning over complex combinations of different data and context information. For example, an


123

increase of blood pressure (low level context) exceeding the patient threshold (low level context) may produce a switching to an “alarm” situation (high level context), which, on its turn may produce a series of operations finalized at identifying and invoking the most suited available and reachable doctor. Our system attacks the former need, i.e. machine representation of context entities, by the use of ontologies which enable to describe meaningful events, objects and relations of context and support several fundamental forms of reasoning, such as concept satisfiability, class subsumption, consistency and instance checking. The latter need, situation switching management, is approached by implementing logical rules and by embedding context events into facts. In this way, an iterative process is activated every time rules are matched. Indeed, fired rules infer new facts, i.e. new scenario switchings, which on their turn may fire other rules. This process allows the system to convert low level context into high level context, with the final result of producing the system reaction to the context switching which originated the process. For example the occurrence of low level events sensed by physical and virtual sensors, may originate the “diastolic blood pressure of patient X=100” and “threshold of patient X=90” facts. These facts may fire a rule inferring the high level fact “alarm for patient X=true”. The alarm situation, combined with facts concerning the availability of doctors, may produce the fact “doctor Y has to be called”, which generates the system response to the detected patient anomaly. As explained in the following subsections, the ontologies were implemented in the OWL language, whilst the rule-based domain knowledge was implemented with Jess on top of OWL ontologies. 3.1 Context Ontology Several context modeling techniques [9] exist such as key value, mark-up scheme, graphical, object-oriented, and ontology-based modeling. According to [9], the ontology-based approach fits well with the requirements of ubiquitous/context-aware computing. Indeed. ontologies facilitate knowledge sharing and reuse. Knowledge sharing is enabled by providing a common knowledge model to computational entities, such as agents and services, which need to interoperate with one another. Knowledge reuse is promoted by ontology amenability to be extended to different domains and to be integrated within wider ontology-based frameworks. Many ontology languages exist including Resource Description Framework Schema (RDFS) [10], DAML+OIL [11], and OWL [12]. OWL is a key to the Semantic Web and was proposed by the Web Ontology Working Group of W3C. Therefore, it was chosen for our prototype. A common practise, when developing ontologies, is to adopt a top level (upper) shared conceptualization [13] on top of which domain ontologies are built. Top level ontologies codify general terms which are independent of a particular problem or domain. Once the top level ontology is available, several lower level ontologies can be introduced, with the scope of incrementally specializing concepts from the high level generic point of view of the upper ontology to the low level practical point of view of the application. This way of structuring knowledge promotes sharing and reuse of ontologies in different application domains.

124

A. Esposito et al.

ContextualEntity Person

Device hasDevice MeasuringDevice

Patient hasPhysiological Parameter PhysiologicalParameter BloodPressure hasCurrentSBPValue hasNormalMaxSBPValue Upper Context Ontology Class Context Ontology Class

Sensor WearableSensor Sphygmomanometer measuresBloodPressure

Is-A Relationship Object Property Datatype Property

Fig. 3. Some classes from the context domain ontology referring to the “patient environment”

Therefore, we divided context ontology into two levels: the top-level context and the domain context ontology. The top-level context ontology includes concepts which refer to context-aware computing, independently from the application domain. The domain context ontology refers explicitly to the health-care domain. The vocabulary related to context entities was defined starting from the widely accepted definition of context, provided in [14]: “Context is any information that can be used to characterize the situation of an entity.” Therefore, an entity is a person, place, computational entity, or object which is considered relevant for determining the behavior of an application. Therefore, as shown in Fig.3 and Fig.4, “person”, “device” and “TriggeredAction” are contextual entities which specialize into different con?epts depending on the context subdomain they belong to. For instance, a person can be a patient, a relative of the patient or a health operator. Devices can be of different types as well. For the sake of brevity, the figures show just small ontology fragments referring to the example used throughout the remaining part of the paper. 3.2 Context Rules The proposed framework utilizes Jess [15] to structure knowledge in the form of declarative rules. Jess is a widely adopted rule engine and scripting environment written in Java. It adopts the Rete algorithm [16] to implement efficiently the rule matching. Jess rules are used to convert low-level information, given in a raw form by sensors, into high-level context. This is conceptually performed in an incremental


125

ContextualEntity Person

TriggeredAction

Patient

Alarm hasRelative

Relative

isInChargeOf

HealthcareOperator Nurse

hasAlert Level AlertLevel

Physician hasAvailability Availability

triggersProcedure ContactProcedure

Upper Context Ontology Class

Is-A Relationship

Context Ontology Class

Object Property

Fig. 4. Some classes from the context domain ontology referring to the “application consumer environment”

fashion.When the system starts to work, the sensor network or other devices get data from physical world. Depending on the incoming events captured by sensors and context, the facts in the Jess working memory are updated. A set of first-order rules determines if an alarm has to be triggered and which alarm level should be activated, according to measurement values and corresponding thresholds. Jess pattern matcher then searches automatically through the available combinations of facts to figure out which rules should be fired. Such rules, when matched, infer new facts which express the context switching to “situation of alarm”. In other terms, the system acquires a sort of anomaly awareness, i.e. the raw data interpretation infers facts which express the occurrence of an anomaly. For instance, the following example shows a rule activating an alarm when the systolic blood pressure (“sbp-c” variable) exceeds the patient threshold (“spb-max” variable). When the rule is fired, the fact “status abnormal is true” (“sbp-s” variable) is inferred and the action “notify the abnormal event” is activated (“sbp-anomalynotification” action): (defrule verify-SystolicBloodPressure (measurement (hasPID ?m-pid) (hasMeasurementDate ?d) (hasMeasurementTime ?t)) (patient (hasPatientID ?pid) (hasCurrentSBPValue ?sbp-c) (hasNormalMaxSBPValue ?sbp-nmax)

126

A. Esposito et al.

(SBPStatusAbnormal ?sbp-s)) (test (> ?sbp-c ?sbp-max)) => (bind ?sbp-s true) (sbp-anomaly-notification) ) The “anomaly awareness” facts may fire other rules, which may on their turn infer other facts. This determines the switching to the higher level context. In other terms, the context switches to a new situation, which we may call “procedure awareness”, in which the activities to be performed in order to manage the alarm situation are known. The following example shows a rule fired as a consequence of an abnormal status due to both systolic and diastolic blood pressure. The rule infers the fact “alarm is true” and the action “find-available-physician” (defrule set-alert-level (patient (hasPatientID ?pid) (SBPStatusAbnormal ?sbp-s) (DBPStatusAbnormal ?dbp-s) (HighAlertLevel ?hal)) (test (eq ?sbp-s ?dbp-s true)) => (bind ?hal true) (find-available-physician) ) Once that the procedures needed to manage the anomaly have been identified, the context consumers come into action by performing suited “anomaly management” actions. As detailed in the following section, the kind of reasoning above described is carried out with the support of suited agents.

4 Agent-Based Reasoning All the agents are implemented by using the Java Agent Development Environment (JADE). JADE [17] is a software framework to develop and run agent applications in compliance with the FIPA specifications [18] for interoperable intelligent multi-agent systems. Inter-Agent communication is based on the FIPA ACL which specifies a standard message language by setting out the encoding, semantics and pragmatics of the messages. As shown in Fig.5, the semantics of agent messages and reasoning is built over OWL concepts and predicates, which are matched with Jess and JADE vocabulary. Figure 6 shows the proposed multi-agent framework, which assigns three fundamental roles to agents: Context provider agents (CPA). These agents wrap context sources to capture raw context data and instantiate the ontology representation. CPAs may encapsulate single


127

OWL public class SystolicBloodPressure extends BloodPressure{ {private int hasNormalMaxSBPValue; public void setHasNormalMaxSBPValue(int value) {this.hasNormalMaxSBPValue=value;} }} JADE (deftemplate MAIN::SystolicBloodPressure MAIN::SystolicBloodPressure "$JAVA-OBJECT$ .SystolicBloodPressure" (declare (from-class .SystolicBloodPressure))) JESS Fig. 5. The system implementation is based on the matching between OWL vocabulary with agent inner context representation (in the form of Java classes) and Jess facts codification

sensors or multiple sources. In the former case (“single domain CPAs”) they are mainly responsible for gathering and filtering data and info from sensor devices. In the latter case, [19] they interact with single domain CPAs, in order to aggregate context information from various context sources (for instance sensed data must be aggregated with patient thresholds). Both kinds of CPAs are responsible also of making low level context inference and putting relevant context information into the rule engine as facts. Context interpreter agent (CIA). Context Interpreter Agents are responsible for observing context changes sensed by CPAs, and, as consequence of these changes, to identify the set of actions that should be performed by context consumer agents. Context consumer agent (CCA). Context consumer agents are responsible for performing the actions triggered by CIAs. Actions provide the application reaction to context information changes, which may assume diverse forms, such as the generation of a signal, the delivery of a notification or a web services request.

5 Context Sources As previously stated, the input raw data of the proposed architecture is represented by the set of values, usually physical parameters, collected by the so-called physical sources. Nevertheless, in order to be effectively and conveniently integrated in the scheme proposed in Fig.2, the capability to measure a physical value (such as temperature, blood pressure or oxygen saturation), is only one of the several

128

A. Esposito et al.

Data CPA

Data Retrieval Working Memory Manipulation Anomaly Awareness

CIA

Working Memory Manipulation Procedure Awareness

CCA

Agent Behaviours Working Memory

Anomaly Management

Intra-Agent Communications Inter-Agent Communications

Fig. 6. The multi-agent framework. Context Provider Agents (CPA) are responsible for inferring a potential “anomaly awareness” situation from data provided by context sensors. Context Interpreter Agents (CIA) process high level knowledge provided by CPAs to acquire a sort of “procedure awareness”. Context Consumer Agents (CCA) forward signals and messages to the suited destination as requested by CIAs (anomaly management). The three categories of agents embed a Jess rule engine, update the working memory of Jess and interoperate by using ACL messages.

requirements a physical source should satisfy. The measured data, for instance, should be sent to a data collector by using a wireless connection, and the choice of the most adequate one is not univocal: wi-fi, Bluetooth, GPRS, UMTS, GSM are only a few of the many possible candidates. In order to allow the indispensable capillary diffusion of physical sources, though, the ratio benefit/cost cannot be left out of consideration, thus imposing the choice of a cost-saving wireless technology. Moreover, the capability to be easily interfaced with Internet could be another added value. On such basis, the integration of sensor-networks with RFID systems appears to be the most practicable way. RFID technology, in fact, is quite inexpensive (passive RFID tags are as cheap as few euro-cents) and naturally compatible with Internet [20]. Moreover, as demonstrated in the following, it could be slightly modified in order to transmit sensor-like data. Indeed, the scheme proposed in Fig.7 represents the actually designed and realized (patent pending) general purpose Sensor-Tag (S-Tag) connected to a generic sensor. The architecture of the S-Tag does not substantially differ from standard RFID systems, thus allowing us to maintain the compatibility between this system and devices already available and internationally standardized. The working principle is as


129

S-tag DIGITAL IN/OUT SWITCH

SENSOR

CHIP (ID1)

CHIP (ID2)

CHIP (ID3)

CHIP (ID4)

MULTI-ID CHIP

SENSOR-TAG ANTENNA

Fig. 7. A simplified scheme of the RFID sensor tag (S-tag)

easy as effective: data measured from a generic sensor are used as input to the S-Tag; when the Tag is in the region covered by the RFID reader, it sends back a signal containing a different combination of several identity codes (IDs) depending on the value of the input itself, thus facilitating the transmission of sensor data. More specifically, the internal microwave circuit of the S-Tag samples the value at its input (which has been measured by the sensor) and quantizes it by using a number of bits equal to the number of available different IDs. For each bit with value equal to 1, the integrated micro-switch selects the corresponding ID to be transmitted; the combination of bits can be hence received by a standard RFID reader and easily decoded in order to rebuild the sensor-measured waveform. This is not the most adequate context for a more exhaustive explanation of the implementation issues of the S-Tag; we only would like to observe that, as apparent from the picture, the sensor is an external unit. In such a way, generic sensors, with the only requirement of a digital output, can be used. Such sensors are not integrated into the S-Tag, so that they do not influence the tag-cost. Moreover, thanks to an accurate electromagnetic design of the tag antenna and of the microwave circuit (microcontroller, RF-switch and so on), also the implemented technological innovation is reasonably inexpensive.

6 Conclusions In this paper we presented a framework for context-aware computing and its prototypal implementation to a home-care scenario. The framework is based on a context codification obtained by integrating an ontology model and a rule-based representation. The main components of the proposed system are its multi-agent

130

A. Esposito et al.

behaviour, as well as the harmonization of heterogeneous technologies, such as agents, ontologies and rule-based inference engines, combined with the low-cost and flexible properties of RFID systems. The overall system is now available and tested in real-life situations in homehealth applications.

References 1. Weiser, M.: The computer for the 21st century. Scientific American, 94–104 (1991) 2. Baldauf, M., Dustdar, S., Rosenberg, F.: A survey on context-aware systems. Int. J. Ad Hoc and Ubiquitous Computing 2(4), 263–277 (2007) 3. Paganelli, F., Giuli, D.: An Ontology-based Context Model for Home Health Monitoring and Alerting in Chronic Patient Care Networks. In: 21st International Conference on Advanced Information Networking and Applications Workshops (AINAW 2007) (2007) 0-7695-2847-3/07 4. Ko, E.J., Lee, H.J., Lee, J.W.: Ontology-Based Context Modeling and Reasoning for UHealthCare. IEICE Trans. Inf. & Syst. E90–D(8), 1262–1270 (2007) 5. Chen, H., Finin, T., Joshi, A.: An ontology for context-aware pervasive computing environments. Special Issue on Ontologies for Distributed Systems, Knowledge Engineering Review (2003) 6. Gu, T., Pung, H.K., Zhang, D.Q.: A service oriented middleware for building contextaware services. Journal of Network and Computer Applications 28, 1–18 (2005) 7. Indulska, J., Sutton, P.: Location management in pervasive systems. In: CRPITS 2003 Proceedings of the Australasian Information Security Workshop, pp. 143–151 (2003) 8. Strang, T., Popien, C.L.: A context modeling survey. In: Workshop on Advanced Context Modeling, Reasoning and Management as Part of UbiComp 2004, The 6th International Conference on Ubiquitous Computing, pp. 33–40 (2004) 9. W3C, RDFS (RDF Vocabulary Description Language 1.0: RDF Schema), Recommendation (February 10, 2004), http://www.w3.org/TR/rdf-schema/ 10. DAML site, http://www.daml.org 11. W3C, OWL Web Ontology Language Overview, Recommendation, (February 10, 2005), http://w3.org/TR/2004/RDC-owl-features-20040210/ 12. Guarino, N.: Formal Ontology and Information Systems. In: Guarino, N. (ed.) Proceedings of the 1st International Conference on Formal Ontologies in Information Systems, FOIS 1998, Trento, Italy, pp. 3–15. IOS Press, Amsterdam (1998) 13. Chen, H., Finin, T., Joshi, A.: An ontology for context-aware pervasive computing environments. In: The Knowledge Engineering Review, vol. 18, pp. 197–207. Cambridge University Press, Cambridge (2003) 14. Dey, A.K., Abowd, G.D.: Toward a better understanding of context and contextawareness, GVU Technical Report, GIT-GUV-99-22 (1999) 15. Ernest Friedman-Hill “Jess In Action”, Edited by Manning 16. http://herzberg.ca.sandia.gov/jess/docs/52/rete.html 17. Jade, http://jade.cselt.it 18. Fipa, http://fipa.org/repository/index.html 19. Dockhorn Costa, P., Ferreira Pires, L., van Sinderen, M.: Architectural patterns for context-aware services platforms. In: 2nd International Workshop on Ubiquitous Computing (IWUC 2005), in conjunction with ICEIS 2005, Miami, USA (2005) 20. Finkenzeller, K.: RFID Handbook: Fundamentals and Applications in Contactless Smart Cards and Identification. John Wiley and Sons Ltd, Chichester

Semantic Learning Space: An Infrastructure for Context-Aware Ubiquitous Learning Zhiwen Yu1, Xingshe Zhou2, and Yuichi Nakamura1 1 Academic Center for Computing and Media Studies, Kyoto University, Japan [email protected], [email protected] 2 School of Computer Science, Northwestern Polytechnical University, P.R. China [email protected]

Abstract. In order to facilitate the development and proliferation of contextaware ubiquitous learning services, there is a need for architectural support in the user context processing and learning content management. In this paper, we propose a context-aware ubiquitous learning infrastructure called Semantic Learning Space. It leverages the Semantic Web technologies to support explicit knowledge representation, flexible context reasoning, interoperable content integration, expressive knowledge query, and adaptive content recommendation. The architectural design and enabling technologies are described in detail. Several applications and experiments are presented to illustrate and evaluate the key features of the infrastructure.

1 Introduction The emergence of e-learning allows the learners to access electronic course contents easily and conveniently through the Web. With the vision of ubiquitous computing coming to reality, people will be living in an environment surrounded ubiquitously by a lot of networked computers (e.g. PCs, TVs) and mobile devices such as PDAs, cellular phones, etc.. Learners can therefore access their desired learning content anytime anywhere with the accessible devices. These two trends have precipitated the advent of ubiquitous learning. One of the most important features of ubiquitous learning is adaptability – learners can get the right information at the right place in the right way [1]. To achieve learning adaptability, the provisioning of content needs to take into account the learner’s changing context (e.g., prior knowledge, learning style, current activities and goals), which we call context-aware learning. In a smart learning environment, a contextaware learning service can, for example, analyze the user’s knowledge gap by comparing his current competencies and tasks, and then suggest suitable learning content for him [2]. While many e-learning systems have been developed and used in recent years, building context-aware learning systems in ubiquitous environment is still complex and time-consuming due to inadequate infrastructure support, e.g., context processing, content provisioning, and content selecting. We propose a context-aware ubiquitous learning infrastructure called Semantic Learning Space that leverages the Semantic Web technologies to support explicit F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 131–142, 2008. © Springer-Verlag Berlin Heidelberg 2008

132

Z. Yu, X. Zhou, and Y. Nakamura

knowledge representation, flexible context reasoning, interoperable content integration, expressive knowledge query, and adaptive content recommendation. The Semantic Web enables computers and people to work in cooperation by representing machine-interpretable information and automating some processes. It satisfies the user-centric, personalized and active learning requirements of context-aware learning. The Semantic Learning Space exploits the Semantic Web standards and tools to provide architectural support for facilitating the development and proliferation of context-aware ubiquitous learning services. The main characteristics of the infrastructure include: (1) systematically handling context management, i.e., aggregation, inference, and query of context; (2) interoperably integrating content from diverse sources; (3) adaptively recommending learning content according to various kinds of context. The rest of this paper is organized as follows. Section 2 discusses previous work relevant to this paper. Section 3 presents the ontology model to express knowledge about the learner, content, and the domain being learned. Section 4 describes the Semantic Learning Space infrastructure and its components in detail. The prototype implementation and evaluation are presented in Section 5. Finally, Section 6 concludes the paper.

2 Related Work There has been much work done in the area of context-based learning in the past few years. LIP project [2] aims to provide immediate learning on demand for knowledge intensive organizations through incorporating context into the design of e-learning systems. iWeaver [3] is an e-learning system which offers learners with different media experiences based on their different learning styles. Sakamura and Koshizuka [4] propose ubiquitous learning that enables people to learn about anything at anytime and anywhere by deploying RFIDs onto a variety of objects, such as foods, medicines, vegetables, and resorts. JAPELAS [5] is a context-aware language-learning support system for Japanese polite expressions learning. It provides learner with appropriate polite expressions deriving the learner’s situation and personal information. Paraskakis [6] proposes a paradigm of ambient learning aiming at providing access to high quality e-learning material at a time, place, pace and context that best suits the individual learner. To achieve this aim, it includes three main functions: multimodal broadband access, content management and integration, and context management. KInCA [7] is an agent-based system supporting personalized, active and socially aware e-learning. The personal agent is aware of the user’s characteristics and cooperates with a set of expert cognitive agents. Recently, some efforts have been put into applying the Semantic Web technologies for learning. Stojanovic et al [8] present an e-learning architecture with ontologybased description of the learning materials. Tane et al [9] propose an ontology tool (displaying, interaction, and evolution of ontology) for semantic resource management in e-learning. Elena project [10] works towards a smart space for learning, which is based on peer-to-peer and Semantic Web technologies. It adopts semantic web ontology languages to describe educational services. Although much work has been done to provide efficient and smart learning services by taking into account user’s context as well as utilizing the Semantic Web

Semantic Learning Space

133

technologies, our work differs from and perhaps outperforms previous work in several aspects. While existing work either focuses on context-based learning or Semantic Web-based learning, we integrate them together, i.e., applying the Semantic Web technologies to accomplish context-aware learning. Although Huang et al [11] propose to exploit Semantic Web technologies to represent context in e-learning, which is similar to us, they do not describe how to use context to accomplish content adaptation. Second, our work enhances the development of context-aware learning services through systematically handling context management (e.g., context aggregation, inference, storage and query) as opposed to ad hoc manner in existing systems. Third, we propose an adaptive recommendation approach to realize context-awareness in learning content provisioning while existing systems simply use match or rule-based selection.

3 Ontology Model We use ontologies to model context about the learner, knowledge about the content, and the domain knowledge (the taxonomy of the domain being learned). Within the domain of knowledge representation, the term ontology refers to the formal and explicit description of domain concepts, which are often conceived as a set of entities, relations, instances, functions, and axioms [12]. By allowing learners or contents to share a common understanding of knowledge structure, the ontologies enable applications to interpret learner context and content features based on their semantics. Furthermore, ontologies’ hierarchical structure lets developers reuse domain ontologies (e.g., of computer science, mathematics, etc.) in describing learning fields and build a practical model without starting from scratch. Upper-level ontologies are designed to provide a set of basic concepts that allows the definition of new concepts in terms of subclasses to complement the upper-level classes. In our system, we have designed three ontologies: Context Ontology, Learning Content Ontology, and Domain Ontology. The Context Ontology shown in Fig. 1a depicts contexts about a learner, e.g., content already mastered, learning goal, available learning time, location, learning style, and learning interests. It also describes the characteristics of the terminal that the learner uses (e.g., displaying feature and network connection). The learning goal may be an abstract subject or a particular content. lco and do stand for Learning Content Ontology and Domain Ontology, respectively. Properties of contents as well as relationships between them are defined by the Learning Content Ontology (see Fig. 1b). The relation hasPrerequisite describes content dependency information, i.e., a content necessary to be taken before the target content. Actually, nowadays most of the departments in universities provide a course dependency chart when issuing their courses. The Domain Ontology is proposed to integrate existing consensus domain ontologies such as computer science, mathematics, chemistry, etc. The domain ontologies are organized as hierarchy to demonstrate topic classification. For instance, the hierarchical ontology of computer science domain is presented in Fig. 1c. It derives from the well-known ACM taxonomy (http://www.acm.org/class/1998/).

134


We adopt OWL (Web Ontology Language) [13] to express ontology enabling expressive knowledge description and data interoperability of knowledge. It basically includes ontology class definition and ontology instance markups.

(a)

(b)

(c) Fig. 1. Ontology design, (a) context ontology, (b) learning content ontology, (c) computer science domain ontology

4 Semantic Learning Space Infrastructure The Semantic Learning Space infrastructure consists of seven collaborating components (see Fig. 2): context aggregator, context reasoner, learning content integration, ontology mapping, knowledge query engine, learning content recommender, and content deliver. The Context aggregator is responsible to aggregate context data from physical sensors as well as virtual sensors, transform the raw context data into context mark-ups, and assert them into the context knowledge base. The Context reasoner infers high-level context information, e.g., user behaviour, from basic sensed contexts, and check for knowledge consistency in the context KB. Learning content integration is responsible for integrating multiple heterogeneous content repositories. Ontology mapping transforms different kinds of content description metadata into the generic ontology mark-ups. The Knowledge query engine handles persistent queries and allows learning content recommender to extract desired content description, context information, and domain knowledge from the knowledge bases. The Learning


135

content recommender selects learning content and determines its presentation form according to learner’s context. Finally the Content deliverer retrieves and delivers the selected learning content to users.

Content deliverer

Learning content recommender

Knowledge query engine Context reasoner

Content knowledge base Domain knowledge base

Context knowledge base

Context aggregator

Ontology

mapping

Learning content integration

Outlook calendar Workflow Human resource Semantic annotation Virtual sensors

Camera Microphone RFID GPS Content repositories

Physical sensors

Fig. 2. Semantic Learning Space architecture

4.1 Context Aggregator Nowadays there are many kinds of sensors deployed in smart home and smart classroom. Such sensors can be utilized as sources of context about learning. It is necessary to combine different information from video sensors, RFID sensors, and other types of sensors for analyzing complicated context, e.g., learning behavior. The context aggregator therefore aggregates diversity of context information from an array of diverse information sources, i.e., physical sensors and virtual sensors. Physical sensors like camera, microphone, RFID sensors and pressure sensors can detect user’s basic context. Virtual sensors, like outlook calendar service and organization workflow, on the other hand, can extract user’s semantic context, such as learning schedule, task, etc. Context aggregation helps to merge the required information which is related to a particular learner. It then asserts them into the context knowledge base for further reasoning.

136


4.2 Context Reasoner Higher-level contexts (What is the user doing? What is the activity in the room?) augment context-aware learning by providing summary descriptions about a user’s state and surroundings. Context reasoner infers abstract, high-level contexts from basic sensed contexts, resolves context conflicts, and maintains knowledge base consistency. It includes both certain and uncertain reasoning. For certain reasoning, we can apply OWL ontology reasoning using DL (Description Logic), and user-defined reasoning based on first-order logic. To perform user-defined context inference, an application developer needs to provide horn-logic rules for a particular application based on its needs. To enable context reasoning to handle uncertainty, we can use mechanisms such as Bayesian Networks, fuzzy logic, and probabilistic logic. Our current system applies Jena2 generic rule engine [14] to support user-defined reasoning over the OWL represented context. The context reasoner is responsible for interpreting rules, connecting to context KB, and evaluating rules against stored context. We have specified a rule set based on the forward-chaining rule engine to infer high-level learning contexts, e.g., a user’s learning behavior, based on basic sensed context. For instance, the following rule examines whether a given person is currently engaged in English studying on the basis of his location, action, book on the desk — if he is in the study room, he is sitting, and the book on the desk is English book, he is likely to be studying English. The user’s location is tracked by RFID. The user’s action, namely standing, sitting, and walking are recognized by video. The English book is detected by RFID sensor to be on the desk. type(?user, Person), locatedIn(?user, StudyRoom), action(?user, Sitting), on(EnglishBook, Desk) =>involvedBehavior(?user, StudyingEnglish) 4.3 Learning Content Integration Learning content integration is responsible for integrating multiple heterogeneous content repositories. Our current system applies Sakai 2.1 tools [15] to integrate varied learning content repositories from a wide range of content providers. It allows to manage integration without including the complexity inherent in supporting heterogeneous means of communication and data exchange, and therefore gains access to content in a manner that hides the technical detail by which that content is provided. To facilitate flexible integration, each repository needs to be wrapped with a plugin developed with O.K.I (Open Knowledge Initiative) Repository OSID (Open Service Interface Definition) [16]. It contracts between service consumers and providers. The OSIDs for eLearning services include repository, assessment, grading, and course management. The Repository OSID also defines methods for managing object lifecycle, data maintenance, and searching. Sakai 2.1 offers the Repository OSID as a repository service. The repository service works with many plug-ins. It finds out what plug-ins are available, loads the plug-ins, and federates requests and responses among them.


137

4.4 Ontology Mapping Different organizations have developed description languages for learning objects which combine terms from multiple vocabularies (e.g., LOM, Dublin Core, and SCORM). To facilitate the semantic interoperability of learning content, we should provide a universal query interface. The ontology mapping mechanism transforms different kinds of content description metadata into the generic ontology mark-ups. It enables universal query across heterogeneous content repositories that are described using different metadata vocabularies. The ontology mapping involves two ontologies: core ontology and mapping ontology. The core ontology of learning content is defined in Section 3. It provides a common understanding of the basic entities and relationships of learning content. We adopt W3C SKOS Mapping [17] to describe the mapping ontology. It contains a set of properties for specifying mapping relations between concepts from different organizations’ ontologies (e.g., exactMatch, broadMatch, narrowMatch, majorMatch, minorMatch). Our current system maps LOM, SCORM, ARIADNE, IMS, OKI, and Dublin Core metadata. The mapping is done semi-automatically. We first manually specify the mapping relations used for mapping concepts in a particular learning content ontology to the core ontology. After this the term mapping in content metadata querying is automatically done according to the manual setting. 4.5 Knowledge Query Engine There are mainly three knowledge bases in our system: Context Knowledge Base (Context KB), Content Knowledge Base (Content KB) and Domain Knowledge Base (Domain KB). The knowledge bases provide persistent knowledge storage and eventbased retrieval by the Knowledge Query via proper interfaces. The Context KB also allows some manipulations, e.g., context reasoning. The Domain KB integrates existing consensus domain ontologies such as ontology of computer science, mathematics, chemistry, etc. The domain ontologies are organized as hierarchy to demonstrate topic classification. Since the three kinds of knowledge are represented with ontology and described with OWL, we can inherently provide a universal query interface. The knowledge query engine handles expressive query triggered by the learning content recommender. It provides an abstract interface for applications to extract desired knowledge from the Context KB, Content KB and Domain KB. In our current system, we adopt RDQL (RDF Data Query Language) [18] to query knowledge about content, context, and domain. It is widely used for querying RDF or OWL represented ontology-based information. 4.6 Learning Content Recommender The content recommender provides the right content, in the right form, to the right person based on learner context. It addresses two problems about providing learning content: (a) which content should be provided, and (b) in which form should the selected content be presented.

138


For the purpose of efficient context processing in content recommendation, we classify context into two categories: personal context (e.g., prior knowledge and goals) and infrastructure context (physical running infrastructure, e.g., terminal capability). The right content is determined by personal context, while the right presentation form depends on infrastructure context and user’s QoS requirements. The content recommender first utilizes knowledge-based semantic recommendation [19] to determine the right content which the user really wants and needs to learn. It computes the semantic similarity between the learner and the learning contents and generates a learning path for a particular target course. The content recommender also suggests content according to a user’s situation context. For example, knowing the user has studied a subject (e.g., mathematics) for a long time and is somewhat tired for it, it will recommend another subject, e.g., English. Then, the content recommender determines appropriate presentation form of the learning content according to user’s QoS requirements and device/network capability [20]. The recommendation process is conducted based on fuzzy theory. An adaptive QoS mapping strategy is employed to dynamically set quality parameters of learning content at running time according to the capabilities of client devices. 4.7 Content Deliverer The content deliverer is responsible for streaming or downloading learning content to various devices through different networks. If the modality of recommended content is video, audio or flash, i.e., continuous ones, the content deliverer streams the content to terminals. On the other hand, if the modality is image or text, i.e., discrete ones, the content deliverer merely downloads the content. The content deliverer should support a wide variety of video, audio, image formats adapting to different terminal and network conditions. In our system, the media delivery supports transferring content over such networks like wired Internet and wireless IEEE802. MPEG-4 is employed as video codec for media streaming.

5 Implementation and Evaluation 5.1 Implementation We have developed a prototype of the Semantic Learning Space infrastructure. It consists of an embodiment of the above defined architecture and a set of APIs for supporting the integration of sensors, content repositories, and the development of context-aware applications. Several context-aware learning applications supported by the proposed infrastructure have been implemented to illustrate the key features of it. Content adaptation is conducted within the applications based on different kinds of contexts. One application is to enable recommending learning content according to user’s learning goal and prior knowledge. The aim of developing this application is to test the infrastructure support for semantic recommendation of learning content. Fig. 3


139

Fig. 3. Main interface for semantic recommendation

shows the main interface for the semantic content recommendation. It mainly consists of four parts. The top part provides GUI for context capturing, i.e., learning goal and the courses already learned. When the learner wants to get content based on these context, the system queries context and makes recommendation. Then, the recommendation list is presented below. The learning path is generated and shown in the left bottom column, while the recommendation package for the content selected from the path is presented in the right bottom column. For learners, how to learn the materials in a more efficient way is very important. The second application developed aims to improve learning efficiency through giving learning advices in arranging learning subjects based on user’s behavior and task. For example, through inference on basic sensed information the context reasoner gets the high-level behavior context that the user has studied a subject (e.g., mathematics) for a long time and is somewhat tired for it. The system then suggests the learner to study another subject (e.g., English), that belongs to today’s learning task and has yet to be finished. The everyday learning task can be made up by the teacher or the parents. The advice message can be given in different ways, e.g., displaying a message on the computer screen or sending an email to parents’ mobile phone. Fig. 4 shows an advice message displayed on the computer. The learner’s current learning behavior, today’s learning task, and learning advice are presented.

Fig. 4. Learning advice

140


The third application is named ELAA (English Learning Anytime Anywhere) with the intention of illustrating the infrastructure support for content adaptation according to the infrastructure context like device capability and network condition. The user can on-line access English learning material through different networks (wired and wireless) and display it on different devices (PC, laptop, PDA, and mobile phone). Suppose that a learner is currently using a Sony VAIO handheld PC that uses either wired Ethernet or wireless network to access the popular English learning material, Family Album USA. As network condition changed, the content presentation form varied accordingly. For example, with the low-bandwidth wireless network, the video playing features 240*176 frame size and 5fps frame rate (see Fig. 5a), while a highquality video with larger size (480*360) and high frame rate (30fps) displaying when the user uses the wired network (see Fig. 5b).

(a)

(b)

Fig. 5. System screenshots: (a) playing low quality video; (b) playing high quality video

5.2 Evaluation We evaluated the Semantic Learning Space from system perspective. Among the infrastructure functions, content recommendation and context reasoning are relatively time-consuming. We therefore mainly tested the overheads of these two components in terms of response time. The experiments were conducted on a Linux workstation with 2.4 GHz Pentium 4 CPU and 512 MB RAM. We measured the content recommending time by varying the number of learning content metadata ranging from 50 to 250. The result is shown in Fig. 6a. We observed that the recommendation process was quite computationally intensive. In particular, as the amount of metadata (or contents) grows, the time spent increases proportionally to the size of the learning content database. For context reasoning, we used five context data sets: 1000, 2000, 3000, 4000, and 5000 RDF triples to test the inference engine’s scalability. The rule set contains 20 user-defined rules. The results (Fig. 6b) show that the response time largely depends on the size of context knowledge base. As the content database and context knowledge become larger, the response time will lead human perceivable delay. However, our user test shows that most of the invited learners found the response time acceptable. We believe the infrastructure is generally feasible for non-time-critical learning applications.


141

Recommending time (ms)_

2500

2000

1500

1000

500

0 50

100

150

200

250

4000

5000

Number of content metadata

(a) 1600

Reasoning time (ms)

1400 1200 1000 800 600 400 200 0 1000

2000

3000

Number of RDF triples

(b) Fig. 6. System performance: (a) content recommending; (b) context reasoning

6 Conclusion and Future Work In this paper, we propose an infrastructure for context-aware learning in ubiquitous computing environment. It supports explicit knowledge representation, flexible context reasoning, interoperable content integration, expressive knowledge query, and adaptive content recommendation. Three applications are given to illustrate the key features of the infrastructure. In the future work, we will continue to develop more context-aware ubiquitous learning applications using the infrastructure and improve the design. Also we are trying to measure the user’s satisfaction with the system by conducting user study.

Acknowledgement This work was partially supported by the Ministry of Education, Culture, Sports, Science and Technology, Japan under the project of “Cyber Infrastructure for the Information-explosion Era”, the High-Tech Program of China (863) (No. 2006AA01Z198), and the Innovation Fund of Northwestern Polytechnical University of China (No. 2006CR13).

142


References 1. Chen, Y., et al.: A Mobile Scaffolding-Aid-Based Bird-Watching Learning System. In: The 1st IEEE International Workshop on Wireless and Mobile Technologies in Education (WMTE 2002), pp. 15–22 (2002) 2. Schmidt, A., Winterhalter, C.: User Context Aware Delivery of E-Learning Material: Approach and Architecture. Journal of Universal Computer Science 10(1), 28–36 (2004) 3. Wolf, C.: iWeaver: Towards ‘Learning Style’-based e-Learning in Computer Science Education. In: The 5th Australian Computing Education Conference (ACE 2003), Adelaide, Australia, pp. 273–279 (2003) 4. Sakamura, K., Koshizuka, N.: Ubiquitous Computing Technologies for Ubiquitous Learning. In: WMTE 2005, Tokushima, Japan, pp. 11–20 (2005) 5. Ogata, H., Yano, Y.: Context-Aware Support for Computer-Supported Ubiquitous Learning. In: The 2nd IEEE International Workshop on Wireless and Mobile Technologies in Education (WMTE 2004), Taoyuan, Taiwan, pp. 27–34 (2004) 6. Paraskakis, I.: Ambient Learning: a new paradigm for e-learning. In: The 3rd International Conference on Multimedia and Information & Communication Technologies in Education (m-ICTE2005), Spain, pp. 26–30 (2005) 7. Nabeth, T., Razmerita, L., Angehrn, A., Roda, C.: InCA: a Cognitive Multi-Agents Architecture for Designing Intelligent & Adaptive Learning Systems. Computer Science and Information Systems 2(2), 99–114 (2005) 8. Stojanovic, L., Staab, S., Studer, R.: eLearning based on the Semantic Web. In: Proceedings of the World Conference on the WWW and Internet (WebNet 2001), Orlando, Florida, USA (2001) 9. Tane, J., Schmitz, C., Stumme, G.: Semantic Resource Management for the Web: An ELearning Application. In: The 13th International World Wide Web Conference on Alternate Track Papers and Posters, New York, USA, pp. 1–10 (2004) 10. Simon, B., Mikls, Z., Nejdl, W., Sintek, M., Salvachua, J.: Smart Space for Learning: A Mediation Infrastructure for Learning Services. In: The 12th International Conference on World Wide Web, Budapest, Hungary (2003) 11. Huang, W., Webster, D., Wood, D., Ishaya, T.: An intelligent semantic e-learning framework using context-aware Semantic Web technologies. British Journal of Educational Technology 37(3), 351–373 (2006) 12. Gruber, T.: A Translation Approach to Portable Ontology Specification. Knowledge Acquisition 5(2), 199–220 (1993) 13. McGuinness, D.L., Harmelen, F.: OWL Web Ontology Language Overview. W3C Recommendation (2004) 14. Carroll, J., et al.: Jena: Implementing the Semantic Web Recommendations. In: WWW 2004, New York, pp. 74–83 (2004) 15. Sakai, http://sakaiproject.org/ 16. Open Service Interface Definition, http://www.okiproject.org/ 17. W3C SKOS Mapping, http://www.w3.org/TR/swbp-skos-core-guide/ 18. Miller, L., Seaborne, A., Reggiori, A.: Three Implementations of SquishQL, a SimpleRDF Query Language. In: Horrocks, I., Hendler, J. (eds.) ISWC 2002. LNCS, vol. 2342, pp. 423–435. Springer, Heidelberg (2002) 19. Yu, Z., et al.: Ontology-Based Semantic Recommendation for Context-Aware E-Learning. In: UIC 2007, Hong Kong, China, pp. 898–907 (2007) 20. Yu, Z., et al.: Fuzzy Recommendation towards QoS-Aware Pervasive Learning. In: AINA 2007, Niagara Falls, Canada, pp. 846–851 (2007)

A Comprehensive Approach for Situation-Awareness Based on Sensing and Reasoning about Context Thomas Springer1 , Patrick Wustmann1 , Iris Braun1 , Waltenegus Dargie1 , and Michael Berger2 1 2

TU Dresden, Institute for Systems Architecture, Computer Networks Group Siemens AG, Corporate Technology, Intelligent Autonomous Systems CT IC 6 {Thomas.Springer, Patrick.Wustmann, Iris.Braun, Waltenegus.Dargie}@tu-dresden.de, [email protected]

Abstract. Research in Ubiquitous Computing and Ambience Intelligence aims at creating systems able to interact in an intelligent way with the environment, especially the user. To be aware of and to react on the current situation usually a complex set of features describing particular aspects of the environmental state has to be captured and processed. Currently, no standard mechanisms are available to model and reason about complex situations. In this paper, we describe a comprehensive approach for situation-awareness which covers the whole process of context capturing, context abstraction and decision making. Our solution comprises an ontology-based description of the sensing and reasoning environment, the management of sensing devices and reasoning components and the integration of these components into applications for decision making. These technological components are embedded into an conceptual architecture and generic framework which enable an easy and flexible development of situation-aware systems. We illustrate the use of our approach based on a meeting room scenario.

1

Introduction

Research in Ubiquitous Computing and especially Ambience Intelligence aims at creating systems able to interact in an intelligent way with the environment, especially the user. ”Machines that fit the human environment instead of forcing humans to enter theirs will make using a computer as refreshing as a walk in the woods” [1]. Thus, a system able to recognise the current situation can adapt its behaviour accordingly. For instance, an application for supporting mobile workers during their tasks in the field could adapt the interaction modalities to improve the interaction with the user (e.g. speech input and output could be used if the workers hands are not free or gesture input could be used if the surrounding noise level is very high). In a similar way, an assistance application for elderly people could intelligently support planning of daily activities, e.g. selecting convenient connections of public transportation systems for carrying out shopping activities, visiting the doctor or meeting relatives or friends. F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 143–157, 2008. c Springer-Verlag Berlin Heidelberg 2008

144

T. Springer et al.

To create such systems, sensing the environment and adapting the behaviour according to its current state are major prerequisites. A system has to be able to capture information about the environment and the involved users based on heterogeneous and distributed information sources, e.g. sensor networks, extracted application data, user monitoring or other methods for gathering context information. The information captured in this way is usually lower-level and has to be abstracted and fused to create an understanding of the overall situation a system is currently within. A large set of schemes for reasoning about the current situation exist which include a wide range of logic and probabilistic based estimation, reasoning and recognition schemes most of which have been employed in artificial intelligence, image and signal processing, control systems, decision theory, and stochastic processes. All these schemes have their advantages and disadvantages and can be applied for different types of sensed data and application areas. Currently, no standard mechanisms are available to model and reason about complex situations. There is no common understanding about what features are relevant for a certain situations and how such features and their interrelations can be identified and modelled. Moreover, the adoption of different reasoning schemes is in an early state, especially with respect to their combination and the overall performance. In this paper, we describe a comprehensive approach for situation-awareness. In particular, the capturing of low-level context information based on sensor networks and further sensing devices, the abstraction of higher-level context using heterogeneous reasoning schemes and the derivation of system decisions are covered. Our solution comprises an ontology-based description of the sensing and reasoning environment, the management of sensing devices and reasoning components and the integration of these components into applications for decision making. These technological components are embedded into a generic framework and a design methodology which enable an easy and flexible development of situation-aware systems.We illustrate the use of our approach based on a meeting room scenario. Our paper is organised as follows: Related work in the areas of context sensing and reasoning as well as for architectures and frameworks for context-awareness is discussed in chapter 2. We introduce our concepts, giving a detailed description of the requirements, the major concepts, the proposed architecture together with its components and a development methodology in chapter 3. In chapter 4 we describe the implementation of the generic framework and our feasibility study based on one example scenario is presented in chapter 5. We conclude the paper with summarising the lessons learned and giving an outlook to future work.

2

Related Work

Several context reasoning architectures and prototypes have been proposed in the recent past. The Sylph [2] architecture consists of sensor modules, a proxy core, and a service discovery module. A sensor module provides a standard means for initialising and accessing sensor devices. A service discovery module advertises

A Comprehensive Approach for Situation-Awareness

145

sensors through a lookup service, and a proxy core manages application queries and serves as a semantic translator between applications and sensor modules. The iBadge prototype, developed with the Sylph architecture, tracks and monitors the individual and social activities of children in kindergartens (loneliness, aggression behaviour, etc.) It incorporates orientation and tilt sensing, environmental sensing, and a localisation unit. The Mediacup [3] is an ordinary coffee mug to which a programmable hardware for sensing, processing, and communicating context is embedded. The hardware is a circular board designed to fit into the base of the cup. It incorporates a micro controller, a memory for code processing and data storage, an infrared transceiver for communication, and an accelerometers and a temperature sensor for sensing different contexts. The same system architecture was used to embed a sensing system into a mobile phone. A three-layered context recognition framework is used to reason about the status of the mug and the mobile phone. It consists of a sensor layer, a cue layer, and a context layer. The sensor layer is defined by an open-ended collection of sensors which capture some aspects of the real world. The cue layer introduces cues as abstractions of raw sensory data. This layer is responsible for extracting generic features from sensed data, hiding the sensor interfaces from the context layer. The context layer manipulates the cues obtained from the cue layer, and computes context as an abstraction of a real world situation. SenSay [4] takes the context of a user into account to manage a mobile phone. This includes adjusting the functional units of the mobile device (e.g. setting the ringer style to a vibration mode) or it can be call related. For the latter case, SenSay prompts remote callers to label the degree of urgency of their calls. Korpip¨ aa¨ et al. [5] exploit several sensors to recognise various everyday human situations. The authors propose a multi-layered context-processing framework to carry out a recognition task. The bottom layer is occupied by an array of sensors enclosed in a small sensor box and carried by the user. The upper layers include a feature extraction layer incorporating a variety of audio signal processing algorithms from the MPEG-7 standard; a quantisation layer based on fuzzy sets and crisp limits; a classification layer employing a na¨ıve Bayesian classifier which reasons about a complex context. Their implementation involves a three-axis accelerometer, two light sensors, a temperature sensor, a humidity sensor, a skin conductance sensor and a microphone. The approaches above support the dynamic binding to context sources. On the other hand, their deployment setting as well as the context reasoning task is predetermined at the time the systems are developed. This limit the usefulness of the systems as users’ and applications’ requirements evolve over time. Subsequently, there is a need for dynamic integration of sensing, modelling and reasoning. We build upon the experiences of previous work to support flexible context reasoning and usage. In parallel, efforts were started to create more general models and services to model and reason about context based on ontologies. Recent projects (e.g., [6]) covered the creation of comprehensive and generic context models with the

146

T. Springer et al.

goal of identifying and integrating characteristics of contextual information. Especially, ontology-based modeling and reasoning is addressed ([7,8,9]) with focus on knowledge sharing and reasoning. In [8,7], approaches for defining a common context vocabulary based on a hierarchy of ontologies are described. An upper ontology defines general terms while domain-specific ontologies define the details for certain application domains. Both approaches use a centralized architecture and work on local scenarios from the smart home or intelligent spaces domain. These solutions focus on the modelling of context and apply particular reasoning schemes. Thus, a comprehensive approach starting with sensor integration and classification is not considered. Moreover, the systematic placement of sensors and a decomposition of situations are not addressed.

3

Conceptual Architecture

Real world situations usually have to be derived from a complex set of features. Thus, a situation-aware system has to capture a set of features from heterogeneous and distributed sources and to process these features to derive the overall situation. Thus, major challenges for the creation of situation-aware systems are to handle the complexity of the situation and the related features, to manage the sensing infrastructure and to find appropriate reasoning schemes that efficiently derive the overall situation from low-level context features. In the following we present a conceptual architecture for the creation of situation-aware systems. The approach is intended to be comprehensive, i.e. it comprises all components and processing steps necessary to capture a complex situation, starting with the access and management of sensing devices up to the recognition of a complex situation based on multiple reasoning steps and schemes. To handle complex situations the concept of decomposition is applied to the situation into a hierarchy of sub-situations. These sub-situations can be handled autonomously with respect to sensing and reasoning. In this way, the handling of complex situations can be simplified by decomposition. We focus on sensors installed in the environment, i.e. they are immobile and usually not part of a mobile device carried by the user. The main idea is to exploit cheap and already available sensors to create new or extended applications. For example, buildings are more and more equipped with sensors, e.g. for measuring temperature or light intensity to enable building automation. Such installations can be extended and then exploited to create ”more intelligent” situation-aware applications. We start with the discussion of the requirements for our solution and discuss our conceptual architecture in detail. 3.1

Requirements

The goal of our research work was to develop a comprehensive approach for situation awareness which covers the whole process of context capturing, situation recognition based on context abstraction and decision making. Because of the dynamic properties in the field of situation-awareness, systems have to be flexible and extensible. Therefore, the approach should be independent


147

of the type of information sources involved, i.e. the types of sensors, the structure of particular sensor networks and the different real world situations which could occur. Furthermore, the approach should not depend on the application scenario, the used reasoning schemes and the type of the derived higher-level context. Based on the situation and application-specific use of the sensing infrastructure, the capturing and abstraction should be (re-)configurable, e.g. regarding the frequency of data collection, the used classification mechanisms and the aggregation of sensor values. Moreover, scalability and modularity are important. That means, the approach should not restrict the amount of deployed sensors and their distribution. It should also allow a flexible combination of different schemes for reasoning of higher-level context information. For a better reuse, the resulting system should be designed in building blocks, so it is easy to replace a single building block by another one (e.g. a particular reasoning scheme with a different one). Especially, a situation should not be modelled in a monolithic way and varying availability of sensors and resulting incompleteness of information should be manageable. Last but not least, the system should be independent from a certain application scenario. Rather, it should support the set-up of different scenarios regarding the involved sensors and their configuration, the used reasoning schemes and the overall situation captured. Thus, a formalised scenario description which can be interpreted, instantiated and exchanged should be supported by the system. 3.2

Conceptual Architecture Based on Situation Decomposition

Our conceptual framework is based on the decomposition of complex situations. From previous experiments we have observed, that complex situations can be decomposed into sub-situations which can be handled independently with respect to sensing and reasoning. Each sub-situation represents a certain aspect of the overall situation and has to be fused with other sub-situations at a certain level of a hierarchical reasoning process. Below that point, a sub-situation can be handled separately. This enables especially a parallel and distributed sensing and reasoning process. Moreover, for each sub-situation the appropriate reasoning scheme can be applied. At the same time, the decomposition enables the modularization of the system, because sub-situation can be assigned to separate processing modules. Extensibility and flexibility can be supported, because during the decomposition process, alternative sub-situations can be identified and new aspects can be easily added to the hierarchical situation description. According to our validation example, the overall situation could be the current use of a meeting room. Among others, the meeting room could be empty, used for a discussion between two people, a discussion in a larger group, a presentation, or a party. Each of these instances of the overall situation can be captured based on the aggregation and processing of different aspects of that complex situation. For instance, the number of persons in the room, their activity, the lighting, the noise level and the state of the beamer could be used to detect the overall situation. Thus, the initial step for creating a situation-aware system according to our conceptual framework is the decomposition of the complex situation the

148

T. Springer et al.

Fig. 1. Abstraction process for situation detection based on sensor data

system should be aware about. The resulting hierarchy of sub-situations can be refined or extended during the development process as well as during the whole lifetime of the system. The leaves of the situation hierarchy represent atomic sub-situation which can not or should not be decomposed further. These atomic sub-situations are the starting points for the creation of the situation-aware system. To reflect all necessary steps to derive the overall situation from sensed information, our conceptual architecture consists of three layers: a sensing layer, a feature extraction layer and a reasoning layer. These layers are depicted in figure 1. Each of these layers will be described in more detail below. Sensing Layer. The sensing layer comprises solutions for two major issues, namely the integration of heterogeneous sensing devices and the placement and organization of sensing devices according to their semantic relation to subsituations. In fact, there is a broad range of different sensors which can be considered for gathering context information like audio, video, a whole wireless-sensor network, etc. These sensors have to be accessed with the help of a specific programming interface. So the sensors deliver different types of (raw) values. Usually the interfaces are provided by the manufacturer of the sensors. Therefore mainly the sensors can be used directly with only little implementation effort, except the special implementation of a wireless-sensor network, for implementing special sensor configurations. Usually, sensors relevant for a certain sub-situation belong to a certain location. Thus, in contrast to other approaches, which often assume a uniform or random distribution of sensing devices and calculate average values out of several values of sensors of the same type in a certain area, we


149

identify distinct ”‘areas of interest”’ which are relevant to capture sensor data relevant for a certain sub-situation. For instance, the presentation area or a table could represent such an area of interest (see figure 3). At these areas of interest different types of sensor devices should be placed and logically grouped. Thus, in our concept each sensor is dedicated to a certain area of interest. Especially, sensors of the same type which belong to different areas of interest are handled separately in the classification and lower-level reasoning steps. The idea is, that an average value of all sensors for the light level at a certain location could be useless for capturing a certain situation, while the values of the light level at certain areas of that location could have a high relevance (e.g. at the projection area of a beamer to decide if the beamer is on or off, or at the surface of a seat to decide if the seat is taken or free). The information of the organisation of a set of sensors into an area of interest is exploited in the classification and reasoning steps described next. Feature Extraction Layer. Because of the heterogeneous sensing devices and the resulting different sensor values, each value has to be classified by an appropriate classifier. Classifiers are dividing the sensor data into individual, application depended classes. These classed are labelled with a symbolic name. The used classifiers can be Neural Networks, Hidden-Markov-Models, Decision Trees, Rule-Sets, Bayesian-Nets, Clustering with a matching heuristic or simply a quantisation over the data. Because of the possible use of different classifiers for the different sensors it is difficult respectively almost not possible in addition to the classified sensor data to have a common quality statement out of the different classifiers, which can be used as quality statements, which influence the reasoning steps. Therefore, the results of the classifiers are logic facts of the particular sensors on their areas of interest. These facts are forwarded then to the reasoning steps. Reasoning Layer. Reasoning is in our concept done in multiple hierarchical steps. Based on the facts resulting from classification new facts, i.e. new highlevel context attributes, are inferred. This is done with the help of different reasoning schemes, which can be deployed separately or work in parallel. Example reasoning schemes are ontology reasoning applying description logics reasoners, rule-based reasoning, case-based reasoning, neuronal networks or bayesian nets. Schemes like the last two can be used, but they have to be trained with a sufficient large training set. The advantages of these methods are, that contradictory facts, which are resulting from measurement failures of the sensors can be handled better. In fact, no wrong high-level context facts are reasoned out of these wrong sensor facts, because the result of these reasoning schemes are statements of the quality or the probability of the inferred context attributes. For example, a trained Bayesian Network for a context attribute calculates the probability to that attribute. On that probability it can be decided if that attribute should be further considered in the following reasoning steps or not. The disadvantage is the application depended and high training effort in advance. The resulting new facts can be further forwarded to reasoners of next superior areas, which combine these facts to another, more abstract fact. After a certain

150

T. Springer et al.

level of abstraction is reached the context attributes of all levels can be delivered to a context-aware application, which can than involve this information for internal decision making, i.e. triggering actions, performing adaptations, etc. To create situation-aware systems according to our conceptual architecture, the developer can intuitively decompose the relevant situation. Based on the experiences and knowledge about classification and reasoning schemes, the different layers of the conceptual architecture can be defined. Usually, the whole process of situation decomposition and the identification of the components of the different layers of the conceptual architecture are determined based on an iterative process. Starting with a small set of sub-situations, the developer can test the system and extend it stepwise to create a more-and-more complex system. Especially, our experiences show, the understanding of a complex situation grows during testing and practical trials.

4

Generic Framework for Situation-Awareness

For implementing situation-aware systems according to our conceptual model we propose a generic framework. The framework integrates several sensor devices and provides a set of classifiers and reasoners which can be adopted in different scenarios. Moreover, it supports the specification of application scenarios based on an ontology which describes the sensor infrastructure, relevant physical values, applied classifiers and reasoners together with their particular configurations and data base settings for storing the captured raw data. The architecture of our framework is depicted in figure 2. In the following sections all components of the framework are described in detail. 4.1

Scenario Manager

The Scenario Manager is the control component of the framework. It is configured by an ontology modelling the settings, sensors, locations and physical values relevant for the current scenario. It evaluates this information, instantiates and configures the required components and invokes the components at runtime. Thus, by providing a new scenario ontology to the Scenario Manager, the framework can be completely reconfigured for another scenario. 4.2

Sensor Integration

The concept of the framework supports the integration of arbitrary sensing devices. We have currently integrated two types of sensors into our framework. Firstly, we implemented a wireless sensor network (WSN) with a flexible amount of sensor nodes at different locations. These sensor nodes are able to obtain different types of sensed data, e.g. light level and temperature, at once. We have used MicaZ Motes from Crossbow in our current implementation. Crossbow offers a wide range of sensor boards which can be connected to the MicaZ and which include many kinds of sensors. For example, the basic sensor board (MTS310CA) provides the following sensors: Photo (Clairex CL94L), Temperature (Panasonic


151

Fig. 2. Architecture of the proposed generic framework for situation-awareness

ERT-J1VR103J), Acceleration (ADI ADXL202-2), Magnetometer (Honeywell HMC1002), Microphone (max. 4KHz), Tone Detector and Sounder (4.5kHz). Based on TinyOS we have implemented basic senor access based on the programming language NesC. We have implemented several commands for initialization, read access, reset, setting of sample rate and network statistics. Secondly, a microphone was utilized for getting audio context information. It is cheap and easy to integrate into a sensor environment. The aim is to extract audio information from the environment, to classify these data according experience, and to identify the most likely context. The integration of the microphone in the prototype was simply done with the javax.sound package. 4.3

Classifiers

For performing the step of classification of sensed data the framework comprises a mechanism for the integration of classifiers and a repository containing a set of classifier implementations. Currently, we have implemented several classifiers for the different sensor types available with the WSN. The result of the classification operations are facts in form of qualitative values (e.g. dark, bright and medium for temperature). For the microphone the underlying model of the classification we used is the Hidden Marokov Model (HMM). Before the HMM can be applied to audio data two the extraction of audio features followed by a quantization of these features has to be performed. Based on training data provided as wave files for a specific scenario, the classifiers can recognise the specified situations (e.g. discussion, presentation or panic in a meeting room). 4.4

Reasoners

Because we assume that all classifiers produce facts we focused on deterministic reasoning schemes. In the current implementation we support rule-based and

152

T. Springer et al.

ontology-based reasoning. Especially, for the reasoning about the overall situation we adopt an ontology-based approach. Therefore, a situation is modelled as a set of concepts and roles in the TBox of the ontology. The current values of concepts and roles related to sensor data or lower-level reasoning steps are included as individuals into the ABox by the scenario manager. To reason about ontologies, a description logic reasoner, namely Pellet [10] is applied. Especially, we use the DL reasoning service realization which works on the Abox. Realization. Given an individual I, Abox A and Tbox T, return all concepts C from T s.t. I is an instance of C w.r.t. A and T, i.e., I lies in the interpretation of concept C. If all Realization is performed for all individuals in the Abox, we speak about the Realization of the Abox. The Abox updates are implemented based on the Semantic Web Framework Jena. Based on realization, concepts can be identified which represent the situation according to the facts in the ABox. For instance, the concept BeamerOn can be defined for the meeting room example. 4.5

Scenario Settings

To ensure the configurability of the framework we introduce an ontology-based scenario description. The description consists of two parts: an upper ontology defining general concepts and roles for the scenario description and a scenariospecific ontology, containing concepts, roles and individuals for the definition of a concrete scenario. The upper ontology defines basic concepts for modelling application scenarios, i.e. available sensors, sensor locations, physical values measured by the sensors, and the assignment of sensors to sensor motes (if existing). The concept Sensor represents the sensors available in the environment. Each sensor is described by a certain location which is assigned to the sensor with the role property locatedAt. The concept Location allows the modelling of semantic locations relevant in the scenario. The sensors available at a certain location are modelled by the property role hasSensor, which is the inverse role of locatedAt. Furthermore, each sensor is described by the physical value, which can be measured by the particular sensor type. The concept PhysicalValue represents a value captured from the environment, e.g. temperature or light intensity. The relation between a sensor and its physical value is modelled by the property measures. Further scenario settings are related to available reasoners, a microphone, the database for storing sensed values and general settings for the scenario. The values are defined as datatype properties for the concepts. In the lower ontology, which is specific to a certain scenario, the general concepts of the upper ontology can be refined by deriving scenario specific concepts. To define concrete instances of reasoners or classifiers, individuals have to be defined in the lower ontology. To add new sensors, classifiers or reasoners, appropriate components have to be implemented and registered with the Scenario Manager. If these components require additional settings, the upper ontology and the controller have to be extended as well. All changes can be done easily based on the current framework implementation. Especially, as long as the


153

upper ontology is extended without changing existing concepts and properties, all existing scenario settings can remain usable without any changes. 4.6

User Interface

For the visualization of the functionality of the framework and the easy setup of demonstrations, a graphical user interface was created. The user interface consists of thee parts enabling the configuration of the WSN, the monitoring of sensor data and captured context and to reason about the current situation.

5

Validation Example

In the following the usage of the proposed conceptual architecture and the generic framework is illustrated by an application scenario, namely the capturing of the current situation in a meeting room. In most companies and universities, conference and lecture rooms, special places in cafeterias and lounges, and other public places should be reserved in advance to conduct public meetings, presentations, parties, etc. This ensures that business is conducted as planned and no inconvenience or conflict of interests occurs which inhibits the overall goal of the company or university. On the other hand, this well planned and well organised execution of business should accommodate rooms for impromptu or short term decisions to organise get together, staff meetings and bilateral discussions. For the meeting room scenario the goal is to determine the current activity in the room based on the state of devices (e.g. the beamer), the noise level and the occupation of seats (number of persons) in the room. The activity inside a room can be a meeting, a casual discussion of two or three people, a presentation involving many people, a party, a person who is reading or idle. Based on this knowledge, the usage of that meeting room could be optimized. For instance, an empty meeting room could be detected for a spontaneous meeting or the location of a certain meeting could be detected. 5.1

Areas of Interest

The identified areas of interest are shown in figure 3. To distinguish between several activities in the room (e.g. one person reading, two persons discussing, a working meeting, a presentation or a party) and an empty room the individual chairs were identified as areas of interest similarly to the train compartment application scenario. For each seat area we installed a temperature and a light sensor to measure the light level and the temperature directly at the surface of the seat. Based on the measured values we can distinguish between the occupation by a person or by an object and an empty seat. Moreover, the table represents an area of interest. In that area we installed a microphone to capture the audio data from the room which helps us to distinguish between a discussion between two people, a larger discussion, a party and a presentation (i.e. one person is speaking from a certain distance to the table). A light sensor installed in that area captures the ambient light in the room.

154

T. Springer et al.

Fig. 3. Areas of interest in the meeting room scenario

Additionally, we identified a so called presentation area, the area where the image of the beamer is projected on and where the person is standing while giving a talk. In that area we installed a light sensor to capture the beamer state (i.e. on or off). In combination with the light level in the room we can identify the situation that the beamer is on and the ambient light in the room is dimmed as it would be the case during a presentation. 5.2

Lower Ontology

For the scenario the sensors TemperatureSensor, LightSensor and AudioSensor are defined. These sensors measure the physical values Temperature, Light and Audio respectively. For Temperature the three qualitative values HighTemperature, MediumTemperature and LowTemperature are defined. For Light four qualitative values are defined, namely Dark, Medium, Bright and VeryBright. The values for Audio represent an activity captured by microphone measurements in the room. Currently we distinguish between Discussion and Presentation. By adding more training examples and extending the ontology more values can be added. To reason about the situation in a meeting room we modelled the chairs around a table in the meeting room. Table and Seat as well as PresentationArea are modelled as sub concepts of MeetingroomLocation which itself is a sub concept of Location. Free seats are modelled by the following concept: F reeSeat = ∃hasLightSensor(∃uso : hasV alue(Bright ∨ M edium))∧ ∃hasT emperatureSensor( ∃uso : hasV alue(LowT emperature ∨ M ediumT emperature))

For occupied seats we distinguish between seat occupied by persons and by objects. A seat occupied by a person is described as follows: P ersonOccupiedSeat = ∃hasLightSensor(∃uso : hasV alue(Dark))∧ ∃hasT emperatureSensor(uso : hasV alue(HighT emperature))


5.3

155

Reasoning

The reasoning about the situation in the meeting room is based on the individual MyMeetingroom. Again, the type of situation is determined by computing the types of this individual based on the reasoning service realisation. MyMeetingRoom belongs to the concept MeetingRoom. The concept represents the overall situation. All partial situations, namely BeamerOff, BeamerOn and PresentationMeetingRoom are modelled as sub concepts of MeetingRoom. Based on the light sensor installed at the presentation area, the state of the beamer can be determined: BeamerOf f = ∃hasP resentationArea(∃hasLightSensor( ∃uso : hasV alue(Bright ∨ M edium ∨ Dark))) BeamerOn = ∃hasP resentationArea(∃hasLightSensor( ∃uso : hasV alue(V eryBright))) ∧ ∃hasRoom( ∃hasLightSensor(∃uso : hasV alue(M edium ∨ Dark)))

In combination with the audio signal the overall situation is determined: P resentationM eetingRoom = BeamerOn∧ ∃hasRoom(∃hasAudioSensor(∃uso : hasV alue(P resentation)))

New scenarios can be implemented by creating a new scenario ontology based on the upper ontology for configuring the generic framework.

6

Summary and Outlook

The major result of our work is a comprehensive solution for modelling and reasoning about complex situations. The solution is comprehensive in the sense that is starts at the sensor layer and comprises all steps necessary to abstract the captured low level sensor values to an overall notion of a complex situation. In our solution we considered the installation and access of sensors, the classification of captured sensor data, several steps of abstracting sensor data and reasoning about sub-situations and the overall situation. In particular, we developed a systematic approach for the decomposition of complex situations. Based on the identification of sub-situations we have shown that it is feasible to place sensor devices in so called areas of interest. Each area of interest can than process the captured sensor data separately (at least the aggregation and classification of sensed values). In higher level reasoning steps the sub-situations are than integrated into the overall situation. To demonstrate the feasibility of our approach we designed and implemented a generic framework for situation awareness. The framework comprises implemented components of every level of situation awareness as described in the third chapter of this document (see figure 1). Each layer is extensible regarding to new components and technologies (i.e. sensors, classifiers and reasoners).

156

T. Springer et al.

Our solution enables an easy, fast and flexible development of situation-aware applications. New scenarios can be implemented by creating a new scenario ontology based on the upper ontology for configuring the generic framework. By using an OWL ontology, scenarios and sensor configurations are clearly defined, easy readable and easy to understand. 6.1

Lessons Learned

From our approach we learned the following lessons: 1. It is possible and reasonable to decompose complex situation into partial situations. At least in all scenarios considered in our project it was possible to decompose complex situations in a way that each sub-situation characterises a certain aspect of the complex situation and is by itself meaningful. Moreover, the partial situations can than be composed to infer the overall situation even if only a subset of all modelled partial situations is considered, i.e. information about environment is incomplete. 2. The identification of areas of interest and the well defined placement and combination of sensors in that area seams to be reasonable and efficient. Compared to an equal distribution of sensors in the environment, the placement of sensors in areas of interest is more focused and environmental state can be captured systematically. 3. The combination of several classifiers and reasoners seams to be possible. In our generic framework a set of classifiers and reasoners is available and we combined them in different ways in the two scenarios. Based on the ontologybased scenario definition, the classification and reasoning algorithms can be configured and thus reused for different scenarios to some degree. 4. We found out that the programming of WSNs and the placement/organisation of sensors is highly related to the scenario. Usually WSNs have to reprogrammed and sensor placement has to be adopted for different scenarios. Thus, the reusability of particular sensor networks and sensor settings is limited. Nevertheless, the generic framework itself is configurable and thus can be reused in different scenarios. 6.2

Future Work

The results demonstrate that our approach is reasonable and advantageous compared to scenario specific and monolithic approaches. While our approach is more flexible, it assumes a closed sensing infrastructure. Especially, the sensors should be under control of the system developer. To exploit existing sensor infrastructures, our approach could be extended to support sensor discovery and dynamic integration of sensing devices. Especially, a location-based discovery of sensors should be considered. Another important point is a further decoupling of all components and layers of our framework. Classifiers and reasoners are available in a local repository and can’t be distributed in the current implementation. Thus, a distribution of these components as well as a discovery and integration mechanism similar to the sensor integration will be considered in the future. Moreover, we will adopt our approach to further application scenarios.


157

References 1. Weiser, M.: The Computer for the 21st Century. Scientific American, 66–75 (September 1991) 2. Srivastava, M., Muntz, R., Potkonjak, M.: Smart kindergarten: sensor-based wireless networks for smart developmental problem-solving environments. In: The 7th Annual international Conference on Mobile Computing and Networking, pp. 132– 138 (2001) 3. Peltonen, V., Tuomi, J., Klapuri, A., Huopaniemi, J., Sorsa, T.: Computational auditory scene recognition. In: International conference on acoustic speech and signal processing (2002) 4. Siewiorek, D., Smailagic, A., Furukawa, J., Krause, A., Moraveji, N., Reiger, K., Shaffer, J., Wong, F.: Sensay: A context-aware mobile phone. In: The 7th IEEE international Symposium on Wearable Computers (2004) 5. Korpip¨ aa ¨, P., M¨ antyj¨ arvi, J., Kela, J., Kernen, H., Malm, E.-J.: Managing context information in mobile devices. IEEE Pervasive Computing 2(3), 42–51 (2003) 6. Henricksen, K., Livingstone, S., Indulska, J.: Towards a hybrid approach to context modelling, reasoning and interoperation. In: Ubi-Comp 1st International Workshop on Advanced Context Modelling, Reasoning and Management, pp. 54–61 (2004) 7. Chen, H., Finin, T., Joshi, A.: An ontology for context-aware pervasive computing environments (2003) 8. Gu, T., Pung, H.K., Zhang, D.Q.: Toward an osgi-based infrastructure for contextaware applications. IEEE Pervasive Computing 3(4), 66–74 (2004) 9. Christopoulou, E., Goumopoulos, C., Kameas, A.: An ontology-based context management and reasoning process for ubicomp applications. In: sOc-EUSAI 2005: Proceedings of the 2005 joint conference on Smart objects and ambient intelligence, pp. 265–270. ACM Press, New York (2005) 10. Sirin, E., Parsia, B.: Pellet: An OWL DL reasoner. In: Haarslev, V., M¨ oller, R. (eds.) Proc. of the 2004 Description Logic Workshop (DL 2004), vol. (104) (2004), http://www.mindswap.org/2003/pellet/index.shtml

Context-Adaptive User Interface in Ubiquitous Home Generated by Bayesian and Action Selection Networks Han-Saem Park, In-Jee Song, and Sung-Bae Cho Department of Computer Science, Yonsei University 134 Shinchon-dong, Seudaemun-gu, Seoul 120-749, Korea {schunya,sammy}@sclab.yonsei.ac.kr, [email protected]

Abstract. Recent home theater system requires for users to control various devices such as TV, audio equipment, DVD and video players, and set-top box simultaneously. To obtain the services that a user wants in this situation, user should know the functions and positions of the buttons in several remote controllers. Since it is usually difficult to manipulate them and the number of the devices that we can control increases, we get to confuse more as the ubiquitous home environment is realized. Moreover, there are a lot of mobile and stationary controller devices in ubiquitous computing environment, so that user interface should be adaptive in selecting the functions that user wants and in adjusting the features of UI to fit in a specific controller. To implement the user and controller adaptive interface, we model the ubiquitous home environment and use the modeled context and device information. We have used Bayesian network to get the degree of necessity in each situation. Action selection network uses predicted user situation and necessary devices, and it selects necessary functions in current situation. Selected functions are used to construct adaptive interface for each controller using presentation template. For experiments, we have implemented ubiquitous home environment and generated controller usage log in this environment. We have confirmed the BN predicted user requirements effectively as evaluating the inferred results of controller necessity based on generated scenario. Finally, comparing the adaptive home UI with the fixed one by 14 subjects, we confirm that the generated adaptive UI is more useful for general tasks than the fixed UI. Keywords: Adaptive user interface, ubiquitous home, Bayesian network, action selection network.

1 Introduction People use remote controllers belonging to the appliances when they want to control them. If they purchase more appliances, the number of controllers is getting bigger according to one of appliances. For example, users have to control several devices such as cable TV set-top-box, DVD player, television, audio equipment, video, and DivX players to use a recent home theater system. Controllers for those devices have different interfaces by their companies though they looked similar. Thus, it is not easy to get accustomed to all controller interfaces [1]. Though the controllers have various F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 158–168, 2008. © Springer-Verlag Berlin Heidelberg 2008

Context-Adaptive User Interface

159

functions, only about a third is used practically. If ubiquitous home is generalized, most home devices like lights and boilers as well as home appliances would be controlled using remote controllers. Therefore, it is required to study a context adaptive user interfaces that provide users necessary functions among many of them. In this situation, PUC (Personal Universal Controller), automatically generating user interfaces in PDA or smart phones, has been investigated recently. J. Nichols and B.A. Myers in CMU developed the system, which can generate an optimized controller interface for smart phone using hierarchical menu navigation system [2] and presented HUDDLE, which generates automatically task-based user interface [1]. Besides, they verified that automatically generated interface has better usability in terms of the time efficiency and consistency than general interfaces for device control using user study [3]. These studies generated and provided the useful PUC, but they cannot consider user’s context. We implemented the ubiquitous home environment and used the space and device information. In addition to the user and sensor input, system have used Bayesian network to infer the necessity of devices given context. Also, action selection network has used the necessity of devices as input and selected the function which is necessary in current context. Adaptive user interface consists of these selected function using presentation template. Modeling using Bayesian networks provides good performance as effectively incorporating the domain knowledge [4]. We also modeled the user interface with action selection network, so it can adapt to the user input constantly. For experiment, we have made the log of the use of devices based on scenario and infer the necessity of devices with this log. After that, we evaluated the proposed method using GOMS model. 14 subjects were asked to perform 10 tasks with both conventional fixed home UI and proposed adaptive home UI. The results showed that subjects dealt with the general tasks more effectively using the proposed UI than using conventional one.

2 Related Works 2.1 Bayesian Networks for Context-Awareness Context can have several meanings. Dey defined context as any information that can be used to characterize the situation of an entity such as a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and the application themselves [4]. Generally, context influences the user's preference to a service, and it does to controlling devices in home. It is because the devices user want to control can change according to user's context. Bayesian network, one of the models that are used for context-awareness, is a model to infer the context and provide reliable performance with uncertain and incomplete information [5]. It can be modeled using the data and can be designed using expert knowledge, and has been used to classify and infer the mobile context based on these strengths. Korpipaa et al. in VTT utilized naive Bayes classifier to learn and classify the user's context [6]. E. Horvitz et al. in MS Research proposed the system that infers what the user concentrated in a certain time point in an uncertain environment [7].

160

H.-S. Park, I.-J. Song, and S.-B. Cho

2.2 Action Selection Network Action selection network was presented by P. Maes for robot agent control [8]. Figure 1 illustrates an example of representative action selection network. In this network, competition of actions is the basic characteristic, and it is shown as a change

Fig. 1. An example of action selection network: Solid line represents predecessor link and dashed line represents successor link

of activation level in each action. An action with the highest activation level is selected after the activation spreading, it is calculated as follows. Nodes can have different types of links that encode various relationships and stimulate one another, and each node has five components of preconditions, add list, delete list, activation level, and the executable code. Links divides into internal and external, and internal link subdivides into predecessor link, successor link and conflict link. External link is connected to sensor and goals. If the value of a certain sensor S1 is true and S1 is in the precondition list of an action node A, a link from S1 to A is activated. If goal G1 has an activation larger than zero and G1 is in the add list of A, a link from G1 to A is activated. The procedure to select an action node executed at each step is as follows: 1. Calculate the excitation from sensors and goals. 2. Spread excitation along the predecessor, successor and conflictor links, and normalize the activation level, so the average becomes equal to a constant π . 3. Check any executable nodes, select the one with the highest activation level, execute it, and finish. A node is executable if all preconditions are satisfied and its activation level is greater than threshold. If there is no executable node, reduce the threshold and repeat the process. Links are set by a designer according to the given task. Using this, the system can select the proper action for its goal given a set of sensor states.


161

3 Context Adaptive User Interface for Ubiquitous Home Figure 2 summarizes the process of proposed adaptive UI generation for ubiquitous home devices. To begin with, Bayesian network infers the necessary devices in current context. With this result and the description of devices and controllers, action selection network is constructed in order to select the necessary functions for each device. Finally, user interface for a given controller is generated using UI template.

Fig. 2. Adaptive UI generation process

3.1 Modeling Ubiquitous Home Ubiquitous home environment for adaptive user interface generation is modeled as follows. - Ε = {Location_List, Device_List, Sensor_List} - Location_List = {Location1, Location2, ..., LocationN} - Device_List = {Device1, Device2, ..., DeviceN} - Sensor_List = {Sensor1, Sensor2, ..., SensorN} - Location = {Location_Name, Location_ID, Neighbors} - Device = {Device_Name, Device_ID, Location_ID, Function_List} - Function_List = {Function1, Function2, ..., FunctionN} - Function = {Function_Name, Function_ID, Function_Type, Function_Value, Add_List, Delete_List} - Sensor = {Sensor_Name, Sensor_ID, Sensor_Type, Sensor_Value}

162


Ubiquitous home environment (E) includes location list, which tells locations of users, rooms and devices, device list including appliances like TV and video and equipments like boiler and lights, and sensor list, which observes the states of users and environments. Device information includes the name, location, and its functions, and function information includes functions of each device and the constraints. Function also includes Add_List and Delete_List for action selection network. Add_List has functions, which are desirable to be displayed with that function. Delete_List, on the other hand, has functions, which are not desirable to be displayed with. Location information is represented as a room name. 3.2 Predicting Necessary Devices Using Bayesian Network We have used the Bayesian network to infer the devices that seem to be necessary for a given context in ubiquitous home. Since Bayesian network can be made by expert knowledge even though there are no or little data for learning [9], the system provides reliable performance in an uncertain home environment where it just set. After enough amount of log is obtained, it is possible to learn the Bayesian network. Personalized model can be learned using the log of individual users. We have used the K2 algorithm proposed by Cooper and Herskovits [10]. To calculate the necessity of each device, basic context such as user location, current time, and the date (whether the day is holiday or not) should be used as evidences. The logs of devices related to each device also have been used as evidences. Related devices mean the devices with similar function or in the same room. After all evidences are set, BN inference is conducted to predict the necessary devices. 3.3 Selecting Necessary Functions Using Action Selection Network Once we get the necessities of each device, an action selection network is designed using Device_List (D) and Function_List (F) explained in section 3.1. Assuming these necessities of devices as a set of environmental nodes E and the states of devices’ functions as a set of functional nodes B, E and B are defined as Equation (1) and (2) using F and D. (1) E = {ei | ei ∈ D ∧ executable(ei )}

B = {bi | bi ∈ F ∧ Executable(bi )}

(2)

After making the nodes, predecessor links, successor links, and conflictor links between these nodes are generated. Predecessor link is generated when Equation (3) is satisfied. Predecessor link, which connects two function nodes or a function node and an environment node, is generated when both nodes belong to the same device and they are related positively. Here, a positive relation occurs among functions that are likely to be executed after a certain function is executed. This link is used to make a hierarchical structure of functions in one device. Successor and conflictor links are used to represent the relations of functions in the different devices. Successor link is decided as Equation (4). It is similar to the predecessor link, but they are different in that successor link connects the functions of different devices and connects functional


163

nodes only. Conflictor link is generated if Equation (5) is satisfied. Differing from the successor link, it is generated when two functions have negative relation (confliction). ⎧ ⎪ ⎪1 precondition(n p , ns ) = ⎨ ⎪ ⎪0 ⎩

⎧ ((n p ⊂ B) ∨ (n p ⊂ E )) ⎪ if ⎨ Device(n p ) = Device(ns ) ⎪relation(excute(n ), excute(n )) p s ⎩ otherwise

⎧ ⎧ ((n p ⊂ B) ∨ (n p ⊂ E )) ⎪ ⎪ ⎪1 if ⎨ Device(n p ) ≠ Device(ns ) succesor (n p , ns ) = ⎨ ⎪relation(excute(n ), excute(n )) ⎪ p s ⎩ ⎪0 otherwise ⎩ ⎧ ⎪ ⎪1 conflictor (n p , ns ) = ⎨ ⎪ ⎪0 ⎩

⎧ ((n p ⊂ B ) ∨ (n p ⊂ E )) ⎪ if ⎨ Device(n p ) = Device(ns ) ⎪confliction(excute(n ), excute(n )) p s ⎩ otherwise

(3)

(4)

(5)

Fig. 3. Change of activation level in an action selection network and constructed user interface after the user controlled TV

164


Constructed action selection network is basically represented as a tree, which has device nodes as parents and their functions as children, and each link is added to that tree based on the functional relations. In this network, activation functions are evaluated as F : E × B → [0...1] . Using the inferred necessities of devices, a set of environment node E is made. After that, the procedure to select a necessary function in action selection network follows as explained in section 2.4. After the procedure, we can get an activation level of each function node. Active function node bi(t) is selected as Equation (6). We let the several functions be selected at once while the conventional action selection network does not, so user interface can use these selected functions to display at next step. ⎧ α i (t ) ≥ θ ⎧ ⎪1 if ⎨ bi (t ) = ⎨ (6) executable (bi , t ) = 1 ⎩ ⎪0 otherwise ⎩ Constructed network is evaluated again when the device list user needs changes a lot or when user requests it. As explained in Figure 3, the necessity of TV gets larger if a user watches it, so network evaluation is updated. After evaluation, user interface includes more functions to control TV.

4 Experiments For evaluation, we have conducted experiments in a simulated ubiquitous home environment. Comparing the proposed adaptive UI with the conventional fixed UI, we have confirmed that the adaptive UI provided better performance.

Fig. 4. A plane figure of ubiquitous home environment


165

4.1 Experimental Environment

Simulated home environment is illustrated in Figure 4. It has 5 rooms of a living room, a dining room, a bedroom, a study, and a bathroom, and each room has devices summarized in Table 1. Lists for devices and functions are represented as XML. Table 1. Devices and functions in each room

Place

Living room

Dining room

Bedroom

Study

Restroom

Device

Name Power Channel TV Volume Brightness Light intensity Power Audio equipment Mode Play Volume Power Ceil light Light intensity Power Wall light Light intensity Window Open/Close Curtain Open/Close Power Coffee maker Status Power Ceil light Light intensity Power Ceil light Light intensity Window Open/Close Curtain Open/Close Bed Fold/Unfold Status Set (am/pm) Alarm Set (hour) Set (minute) Power Computer Status Phone Status Power Ceil light Light intensity Power Ceil light Light intensity Instantaneous Power water heater Temperature Power Fan Mode

Function Type Enum Enum Range Range Range Enum Enum Enum Range Enum Range Enum Range Enum Enum Enum Enum Enum Range Enum Range Enum Enum Enum Enum Enum Range Range Enum Enum Enum Enum Range Enum Range Enum Range Enum Enum

166


4.2 Usability Test for Adaptive User Interface

To evaluate the usability of the proposed method, we assumed 10 situations that could happen in home environment and evaluated adaptive interface generated with GOMS model [11]. Situations are provided in Table 2. Each situation has detailed tasks. For example, situation #1 is “Having a breakfast while listening to music with audio” and its detailed tasks are as follows. “Turn on the ceil light in a dining room Æ Turn on the coffee maker Æ Turn on the audio equipment in a living room Æ Set the frequency Æ Set the volume Æ Have a breakfast Æ Turn off the ceil light in a dining room” Table 2. 10 situations for usability test Number 1 2 3 4 5 6 7 8 9 10

Situation

Having a breakfast while listening to music with audio Watching TV in the evening Playing computer games Getting up from the bed in the morning Taking a shower Closing all windows and curtains Turning of all lights Watching TV while having dinner Having a phone conversation in a bed at night Going to sleep at night

Fig. 5. A user interface used in current home network environment (A fixed UI only)

Using these situations, we have evaluated the user interfaces of Figure 5 and Figure 6 using GOMS. Figure 5 used tab-type interface for selecting place and device. To control a certain device, 2 steps of tab selection are required. This is an interface that


167

Fig. 6. A user interface including context-adaptive interface generated by proposed method (A fixed UI with an adaptive UI)

is based on a controller design of home network widely used. An interface in Figure 6 adds a context-adaptive interface at the right side. It changes according as the user controls the devices or the context changes. Using these two interfaces, we have compared the time that consumed for 14 users to perform the tasks in 10 situations. Figure 7 summarizes the result. When using an adaptive interface together, the reduction rate of time was 38.63% comparing to one only using a fixed user interface.

Fig. 7. Time consumed to perform tasks in 10 situations

168


5 Conclusion This paper proposed the method for context-adaptive user interface generation. There can be several devices and controllers in a ubiquitous home environment, proposed method generated the user interface considering both controller and devices. Bayesian network was used to infer the necessary devices, and an action selection network was used to select the necessary functions according as the context and user selection changes effectively. Finally, we have conducted the usability test of the user interface generated by the proposed method. For future work, we are planning to compare the proposed model with others.

Acknowledgements This research was supported by the MKE (Ministry of Knowledge Economy), Korea, under the ITRC (Information Technology Research Center) support program supervised by the IITA (Institute of Information Technology Advancement) (IITA-2008(C1090-0801-0046))

References 1. Nichols, J., Rothrock, B., Chau, D.H., Myers, B.A.: Huddle: Automatically generating interfaces for systems of multiple connected appliances. In: UIST 2006, pp. 279–288 (2006) 2. Nichols, J., Myers, B.A.: Controlling home and office appliances with smart phones. IEEE Pervasive Computing 5(3), 60–70 (2006) 3. Nichols, J., Chau, D.H., Myers, B.A.: Demonstrating the viability of automatically generated user interfaces. In: CHI 2007, pp. 1283–1292 (2007) 4. Kleiter, G.D.: Propagating imprecise probabilities in Bayesian networks. Artificial Intelligence 88(1-2), 143–161 (1996) 5. Dey, A.K.: Understanding and using context. Personal and Ubiquitous Computing 5, 20– 24 (2001) 6. Korpipaa, P., Koskinen, M., Peltola, J., Makela, S.-M., Seppanen, T.: Bayesian approach to sensor-based context awareness. Personal and Ubiquitous Computing 7(2), 113–124 (2003) 7. Horvitz, E., Kadie, C.M., Paek, T., Hovel, D.: Models of attention in computing and communications: From principles to applications. Communications of the ACM 46(3), 52–59 (2003) 8. Maes, P.: How to do the right thing. Connection Science Journal 1(3), 291–323 (1989) 9. Cooper, G., Herskovits, E.A.: A Bayesian method for the induction of probabilistic networks from data. Machine Learning 9(4), 109–347 (1992) 10. Card, S., Moran, T.P., Newell, A.: The Psychology of Human Computer Interaction (1983)

Use Semantic Decision Tables to Improve Meaning Evolution Support Systems Yan Tang and Robert Meersman Semantic Technology and Application Research Laboratory (STARLab), Department of Computer Science, Vrije Universiteit Brussel, Pleinlaan 2 B-1050 Brussels, Belgium {yan.tang, robert.meersman}@vub.ac.be

Abstract. Meaning Evolution Support Systems have been recently introduced as a real-time, scalable, community-based cooperative systems to support the ontology evolution. In this paper, we intend to address the problems of accuracy and effectiveness in Meaning Evolution Support Systems in general. We use Semantic Decision Tables to tackle these problems. A Semantic Decision Table separates general decision rules from the processes, bootstraps policies and template dependencies in the whole system. Recently, DOGMA-MESS (“Developing Ontology Grounded Methodology and Applications” framework based “Meaning Evolution Support Systems”) is developed at VUB STARLab as a collection of meaning evolution support systems. We embed Semantic Decision Tables in DOGMA-MESS to illustrate our approach. Semantic Decision Tables play the roles in both top-down and bottom-up processes of the meaning evolution cycle. The decision rules that consist of templates dependency rules are mainly responsible for the top-down process execution. The bottom-up process execution relies on the ones that contain the concept lifting algorithms. Keywords: ontology, Meaning Evolution Support System, Semantic Decision Table.

1 Introduction Nowadays, a vast amount of ontology capturing methodologies and tools are available. In [9] and on the OnToWorld wiki website1, a general survey on the ontology capturing methodologies is explored, such as the Grüninger and Fox method [11], the Uschold and King method [27], Methontology [8], CommonKADS [18], the linguistic based methodologies (such as in [2, 15]), Opjk methodology [4], and DOGMA methodology [22]. DOGMA (Developing Ontology-Grounded Methods and Applications) is in general a framework for ontology capturing methodologies and applications. Amongst these mentioned methodologies, DOGMA methodology is designed partly based on the best practice of the other methodologies: 1) the Grüninger and Fox method for TOVE project uses the competency questions as a way to scope the domain of 1

http://ontoworld.org/wiki/Ontology_Engineering#Ontology_Building_Methodologies

F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 169–186, 2008. © Springer-Verlag Berlin Heidelberg 2008

170

Y. Tang and R. Meersman

interests and conceptualization evaluation; 2) the Uschold and King method for Enterprise Ontology emphasizes the importance of incorporating brainstorming and defining/grouping terms in a natural language; 3) Methontology and CommonKADS focus on the structural knowledge management activities. Each activity produces a deliverable as the output; 4) the linguistic based methodologies ground the concepts on the basis of natural languages. Hence, it is necessary to use the natural language processing technologies while building multilingual ontologies from scratch; 5) the Opjk methodology adapts the argumentation method (so called “Diligent”). It underlines the socio aspect. Seeing the importance of community aspect in the notions of ontology [10, 12], Semantic Web2, Web 2.0 [3], and some socio aspect focused methodologies (e.g. [4]), the trends towards community impacts on ontology engineering result in a growing interest in community-grounded, knowledge-intensive methodology. DOGMA Meaning Evolution Support System (DOGA-MESS) is thus developed at the VUB STARLab [6]. As an extension to the DOGMA, DOGMA-MESS is a machine-guided ontology versioning, merging and alignment system to support scalable ontology engineering. In practice, we observe that it is hard to do in an interorganizational setting, where there are many pre-existing organizational ontologies and rapidly evolving collaborative requirements. Current ontology merging systems 3 mainly focus on how to integrate different ontologies, such as in [14]. Researches concentrate on combining several (sub-) ontologies into one ontology by removing inconsistency and reducing conflicts among them. DOGM-MESS does not focus on how to solve these problems, but to gradually build interoperable ontologies amongst a large, knowledge-intensive community. One communicates with others’ needs, trying to find the overlapping interests, with which we make interorganizational ontologies. The core activity in meaning evolution support systems is to reach the final consensus on the conceptual definitions. How to manage the negotiation among the members (also called “domain experts”) of the community is crucial. In DOGMAMESS, the technology of meaning negotiation [5] is thus integrated, which constructs the kernel of community management. However, the community behaviors are not thoroughly studied, which leads to the fact that the outcomes of the DOGMA-MESS processes often become “messy”. Therefore, a novel design by embedding Semantic Decision Tables (SDTs) in DOGMA-MESS micro process was proposed in our early paper [26]. Semantic Decision Tables are used to capture the community’s behaviors at the macro level and guide the community at the micro level. Recently, we get increasing requirements of managing ontological structure at a high level, automatically checking the dependencies of different knowledge blocks, and the ability of the quick and accurate adaptation of the knowledge elicitation processes in DOGMA-MESS. These requirements become the challenges of this paper. Based on our early work [26], we enrich the model of DOGMA-MESS by integrating Semantic Decision Tables in DOGMA-MESS in this paper. We focus on how Semantic Decision Tables are used in both bottom-up and top-down processes of the 2 3

http://www.sciam.com/article.cfm?articleID=00048144-10D2-1C70-84A9809EC588EF21 We consider the ontology merging systems as a kind of systems in scalable ontology engineering.

Use Semantic Decision Tables to Improve Meaning Evolution Support Systems

171

meaning evolution cycle in DOGMA-MESS. The accuracy and effectiveness that Semantic Decision Tables can bring for the meaning evolution support systems in general are stressed in this paper. The remainder of this paper is structured as follows. In section 2, we present the background of the paper. We compare our work with the existing technologies, and discuss both the advantages and disadvantages of our work in section 3. We design the model of DOGMA-MESS embedded with Semantic Decision Tables in section 4. Different Semantic Decision Tables hold different semantically rich decision rules. The decision rules that consist of templates dependency rules are mainly responsible for the top-down process execution (section 4.1). The bottomup process execution relies on the ones that contain the selection algorithms, which can be evaluated (section 4.2). Section 5 contains the paper conclusion and the future work.

2 Background This section is organized as follows. First, we explain the background of DOGMA (Developing Ontology-Grounded Methods and Applications, [21]) and its philosophical foundation of double articulation (section 2.1). Second, we recall the model of DOGMA-MESS (Meaning Evolution Support System) methodology in section 2.2. Third, we explain the notion of Semantic Decision Tables in section 2.3. 2.1 DOGMA The research efforts on DOGMA (Developing Ontology-Grounded Methods and Applications, [21]) approach to ontology engineering have been performed at the VUB STARLab over ten years. It was designed as a methodological framework inspired by the tried-and-tested principles of modeling conceptual databases. In the DOGMA framework one constructs (or converts) ontologies by the double articulation principle into two layers: 1) the lexon base layer that contains a vocabulary of simple facts called lexons, and 2) the commitment layer that formally defines rules and constraints by which an application (or “agent”) may make use of these lexons. A lexon is a quintuple < γ , t1, r1, r2, t2>, where γ is a context identifier. γ is assumed to point to a resource, and serves to disambiguate the terms t1, t2 into the intended concepts. r1, r2, which are “meaningful” in this specific context γ , are the roles referring to the relationships that the concepts share with respect to one another. For example, a lexon < γ , Driver’s license, is issued to, has, Driver>4 explains a fact that “a driver’s license is issued to a driver”, and “a driver has a driver’s license”. The linguistic nature of a lexon represents that a fundamental DOGMA characteristic is its grounding in the linguistic representation of knowledge. The community of domain experts chooses (or has to agree on) a given (natural) language, e.g. English, to store and present lexon terms and roles. 4

In this paper, we do not focus on the discussion of the context identifier γ , which is omitted in other lexons. E.g. < γ , Driver’s license, is issued to, has, Driver> is thus written as .

172


A commitment corresponds to an explicit instance of an intentional logical theory interpretation of applications. It contains a set of rules in a given syntax, and describes a particular application view of reality, such as the use by the application of the (meta-) lexons in the lexon base. This describing process is also called ‘to commit ontologically’. The commitments need to be expressed in a commitment language that can be easily interpreted. Suppose we have a lexon , which has the constraint as “one driver’s license is issued to at most one driver”. We apply the uniqueness constraints UNIQ on the lexon written as below: p1 = [Driver’s license, is issued to, has, Driver]: UNIQ (p1).5 Just like the same database can be viewed and used by different database applications, the same lexon base can be queried, constrained and used by different ontology based application. The commitments interface the lexon base and different applications. The commitments can be further modeled graphically with many popular modeling tools, such as ORM (Object Role Modeling, [13]), CG (Conceptual Graph, [19]) and UML (Unified Modeling Language6). Ontologies modeled in DOGMA can be further implemented in an ontology language, such as OWL7 and RDF(S)8. 2.2 Meaning Evolution Support Systems DOGMA-MESS (Meaning Evolution Support System, [6]) is to organize the process of interorganizational 9 ontology engineering. Its basic characteristic is to find the relevance of concepts defined by different domain experts in order to capture the overlapping interests in the community. The goal is not to create a “complete” domain ontology that covers all the concept definitions in the domain, but a dynamically evolving “interorganizational” domain ontology. A generic model of interorganizational ontology engineering is presented in Fig. 1. An interorganizational ontology cannot be produced once, but needs to evolve over time. The process results in different versions – e.g. v_1 and v_m in Fig. 1. Each version of the interorganizational ontology contains three different layers: OO (Organizational Ontology), LCO (Lower Common Ontology) and UCO (Upper Common Ontology). •

5

Each domain has its own UCO, which contains the common concept type hierarchy and the domain canonical relations. For example, “employee”, “employer” and “personnel” are the common concept types in the domain of human resource management. The concept types “employee” and “employer” are the subtypes of “personnel” in the type hierarchy. The domain

The syntax of the formalized commitment and the examples can be found at: http://www.starlab.vub.ac.be/website/SDT.commitment.example 6 UML is specified by OMG (Object Management Group), http://www.uml.org/ 7 OWL Web Ontology Language: http://www.w3.org/TR/owl-features/ 8 RDF(S) Resource Description Framework (Schema): http://www.w3.org/TR/rdf-schema/ 9 The name of “Interorganizational ontology” was coined by de Moor in [6]. The name “Interorganizational” indicates that the ontology only contains the overlapping interests and is used between the organizations.


•

• •

173

canonical relations are the relations that are common to the domain. For example, the ones between “employee” and “employer” can be “hire” or “fire”. OO is created by individual domain experts based on the ontological structure at UCO level. Each OO within one ontology version is represented by one domain expert that may have different insights from the others’. At LCO level, the concept definitions from different OO are aligned and merged. The overlapped domain interests are elicited. This process happens within a version. When the interorganizational ontology evolves, all the definitions of the concepts at LCO level of version N are standardized and lifted to UCO level of version N+1.

Fig. 1. A Model of Interorganizational Ontology Engineering, [6]

DOGMA-MESS supports the users playing three main roles: the knowledge engineer, the domain experts and the core domain experts. The knowledge engineer has thorough knowledge of analyzing formal semantics in general. He helps the core domain experts to standardize the concept definitions at LCO level. The core domain experts are recognized domain experts who have excellent domain expertise and a clear overview of the domain. The domain experts constitute the majority of the system users. Every enterprise has at least one domain expert who has his own insights of his own domain. They are responsible to create their own organizational ontologies. 2.3 Semantic Decision Table A Semantic Decision Table (SDT, [24]) uses the tabular presentation as a decision table does. There are three basic elements in a decision table: the conditions, the actions (or decisions), and the rules that describe which actions might be taken based on the combination of the conditions. A condition is described by a condition stub and a condition entry. A condition stub contains a statement of a condition. Each condition entry indicates the relationship between the various conditions in the condition stub.

174


An action (or decision) contains an action stub and an action entry. Each action stub has a statement of what action to be taken. The action entries specify whether (or in what order) the action is to be performed for the combination of the conditions that are actually met in the rule column. Table 1. A Simple Example of a traditional decision table 1

2

3

Yes

Yes

No

Condition Bad weather It’s far from home

Yes

No

Yes

Money in pocket

Yes

Yes

Yes

Action Take a taxi back home Walk back home

*

* *

Table 1 is a simple decision table with three conditions: “Bad weather”, “It’s far from home” and “money in pocket”; and two actions: “Take a taxi back home” and “Walk back home”. The condition “Bad weather” has two condition entries - “Yes (The weather is bad now)” and “No (The weather is not bad now)”. The rule column with ID ‘1’ expresses a decision rule as “If the weather is bad, it’s far from home, and, there is money in pocket, then the decision is to take a taxi back home”. In the collaborative settings, one decision maker might misunderstand (or have his own comprehension of) the meaning of a decision item designated by others. For example, the condition “It’s far from home - Yes” in Table 1 can have different measures of distance. Does it mean that the distance is more than 1 km, or more than 3 km? A traditional decision table itself doesn’t support the collaborative setting. In [24], we have listed the following problems that occur when using traditional decision tables: 1) ambiguity in the information representation of the condition stubs or action stubs, 2) conceptual duplication amongst the conditions, 3) uncertainty in the condition entries, and 4) difficulties in managing large tables. The notion of Semantic Decision Table (SDT, [24]) was introduced to tackle the mentioned problems. What makes an SDT different from a traditional decision table is its semantics. Unlike traditional decision tables, the concepts, variables and decision rules are explicitly defined. A decision group shares the most important decision knowledge within a group decision making session. SDTs are modeled in DOGMA (section 2.1). Accordingly, an SDT contains a set of SDT lexons, SDT commitments and a specific decision task. The question on how to construct an SDT within a decision group is answered in our recent publications [23, 24, 25]. Although an SDT contains SDT lexons and SDT commitments, SDT itself is not an ontology. It is because each SDT is used for a specific decision task. The SDT commitments can contain both static, ontological axioms and temporal, changeable rules. Also note that the usage of SDTs is not restricted to a specific system, such as DOGMA-MESS in this paper. Instead, we use DOGMA-MESS as an example to


175

demonstrate how SDTs can improve community-grounded systems. SDTs can be applied in many group decision making domains, such as collaborative human resource management.

3 Related Work and Discussion Consensual knowledge base introduced in [7], meeting techniques applied in [4], cooperative domain ontology studied in [1], “Diligent” in Opjk methodology [4] are promising related work in building consensus on ontology level. However, authors in [6] discuss that those methodologies are lack of community consideration although those methodologies work out some basic principles for building ontological consensus. DOGMA-MESS focuses on the community aspects of scalable ontology engineering, and provides a “fat” community grounded model. In our current projects (e.g. the EC Prolix project15), we observe that there are several advantages and disadvantages while applying Semantic Decision Tables to DOGMA-MESS. We stress the advantages as follows: • • • •

The tabular reports generated based on SDT commitments, in general, are extremely convenient and user-friendly for non-technical domain experts. Semantic Decision Tables are used to constrain the dependencies between the templates. The accuracy of the system is thus improved. Semantic Decision Tables are used to capture the behaviors of the community, manage and guide the community systematically and automatically. Therefore, the effectiveness of the system increases. The flexibility at the system management level increases because the knowledge engineers can create different algorithms and decision rules based on their needs.

A big disadvantage is the complexity. The knowledge engineers need to know how to construct Semantic Decision Tables. However, we observe that, in the projects, it is rather easy for the experts to write Semantic Decision Tables if they already know the DOGMA. It is because the notion of Semantic Decision Table is modeled in the DOGMA framework, and the ontology developed at STAR lab also takes the DOGMA format.

4 Embed Semantic Decision Tables in a Meaning Evolution Support System Recently, modeling layered ontologies has been studied so far. The scalable ontology model we describe focuses on neither the typology of ontology nor the construction of layered ontologies. Instead, we focus on the idea of how to gradually build ontologies within layered ontologies. Based on the work in [28, 6], we model a scalable ontology into four layers: Top Level Ontology (TLO), Upper Common Ontology (UCO), Lower Common Ontology (LCO) and Organizational/Topical Ontology (OTO). In [28], topical ontology structure for scalable ontology engineering is introduced to represent the knowledge

176


Fig. 2. Interorganizational ontology engineering model with the integration of Semantic Decision Tables

structure of the domain experts (the stakeholders of a community) by involving different view points of analysis and modeling. Later on, the interorganizational ontology model is designed to match the requirements for meaning evolution [6]. We integrate SDT into the topical ontology model and interorganizational ontology model illustrated in Fig. 1. Fig. 2 shows our design. The dotted lines with arrows in Fig. 2 indicate the specialization dependencies between the ontologies of different levels. Comparing to Fig. 1, Fig. 2 contains an extra ontological level – the level of Top Level Ontology (TLO)10. It defines the abstract concept types, such as ‘Actor’, ‘Object’, ‘Process’ and ‘Quality’. Conceptualization at this level is not allowed to be changed. The relations between these concept types fall into two categories: i) the hierarchical relations reflected by the type hierarchical construct. This kind of relations is also called subsumption ontological roles (e.g. “subtype of” relationship in [20]). ii) Other Core Canonical Relations, such as “part-of” merelogical relation in [12], “property-of” relation and “equivalent” relation. Another difference is the OTO Level. In Fig. 1, the lowest level is OO (organizational ontology) level. In Fig. 2, the lowest level is OTO (organizational and topical ontology) level, which includes OO level. Organizational Ontology and Topical Ontology (OTO) seek to represent systematically the knowledge structure the domain experts has on the given themes (or tasks) individually. A Topical Ontology “lays 10

It was called “MO (Meta Ontology)” level in the old papers [6, 26]. However, we have debated whether to name it as MO level or not in the OTM’07 conferences (http://www.cs.rmit.edu.au/fedconf/). As the structures at this level don’t necessarily model the ontology itself, we conclude that it is better to call it TLO.


177

foundation for application (or task) specific ontologies and conceptual models… its semantic space covers multiple subjects and dynamic evolution of the core concepts within a topic” [28]. The concepts within a topic represent the terminology of the application structure, assumption and framework. Within a version, every domain expert (or every enterprise-wise stakeholder group) is responsible to build his own OTO based on the ontology models in UCO. How does DOGMA-MESS execute the model in Fig. 2? Suppose we have some top level models at TLO level designed by the knowledge engineer. The whole process in Fig. 2 starts with creating the templates at UCO level based on the models at LCO level. These templates at UCO level form the first version of the interorganizational ontology (ontology version V1). Then, these templates are delivered to the domain experts from different enterprises. When they receive the templates, they start to create the ontologies at OTO level. After a time period, the system collects the newly introduced concepts, selects a few and lifts them from OTO level to LCO level. When the lifting process is finished, the core domain experts empty the concept set at LCO level by standardizing and formalizing them. Then, the core domain experts merge them at UCO level. A new ontology version V2 is created. As the starting point of creating the ontology V3, the domain experts (again) introduce new concepts based on the updated templates at UCO level, and so forth. By executing the model visualized in Fig. 2 recursively, the ontology evolves. 4.1 Top Down: Use Semantic Decision Tables to Mange Templates Dependencies We use Semantic Decision Tables (SDTs, section 2.3) to manage the dependencies between different templates at UCO level (Fig. 2). In DB theory, several classic templates dependencies are defined in [17]. In the most general sense, dependencies are possible constrained relations. Among all kinds of templates dependencies, multivalued dependencies, subset dependencies, join dependencies, mutual dependencies, and generalized mutual dependencies are the mostly used. Except the multivalued dependencies, the others are also called “functional dependency”. Based on their work, we carefully reuse the notions for ontology engineering. The dependencies in [17] are the constraints at the instance level. Note that the definitions of “instance” in DB theory and in ontology engineering are disparate. In DB theory, a record in a table is called an instance, e.g. in [17]. Suppose that we have a table column called “vehicle” with two record “bus” and “TOYOTA VIP-234”. These two records are both called the instances of “bus”. In ontology engineering, “bus” is a type at the conceptual level. “TOYOTA VIP-234”, which is a car with a plate license, points to a unique object in the UoD. Hence, “TOYOTA VIP-234”, in ontology engineering, is an instance at the instance/application level. In this paper, we follow the definition of “instance” in ontology engineering. We use SDT to manage the dependencies at both the instance level and the conceptual level. In the following subsections, different kinds of dependencies are explained. 4.1.1 Multivalued Dependencies The multivalued dependencies are used to constrain the generating process of the templates. For example, we have two templates that are relevant to the concept “course” (Fig. 3).

178


Fig. 3. Two templates in Conceptual Graph [20] at UCO level

If we apply the multivalued dependency of “course” on these two templates, and if a course type (e.g. “Java course”) is introduced by using one template in Fig. 3, “Java course” will be automatically added to another template as a subtype of “Course”. The constraint is stored as the following SDT commitment: (P1 = [Teacher, teach, , Course], P2 = [Teaching, teach, , Course]): MULTIVAL_DEP(P1(Course), p2(Course)). Seeing that the same templates can sometime be applied to different contexts11 in an ontology in DOGMA-MESS, we stress that to use the multivalued dependencies is necessary. 4.1.2 Subset Dependencies The subset dependencies in [17] is similar to the is-a subsumption relationship in ontology engineering. We say that concept A is the subset of concept B when the subtypes and instances of A belong to the set of concept B. For example, we have two templates: one is related to the concept “Teacher” and the other is related to the concept “Lecturer” (Fig. 4). In the context of “university”, “Lecturer” is a subtype of “Teacher”.

Fig. 4. One template relevant to “Teacher” and the other to “Lecturer”

We store this subset dependency in SDT commitments in two ways: 1) to create a new lexon constrained with a subset constraint; 2) to write a subset constraint directly between these two templates. The SDT commitments are shown as follows: P3 = [Lecturer, is a, , Teacher]: SUBSET(P3(Lecturer), P3(Teacher)). //method 1 (P1 = [Teacher, teach, , Course], P4 = [Lecturer, has ID, , Personnel ID]) : SUBSET(P4(Lecturer), P1(Teacher)). //method 2

11

For example, the members of “course” in the context of “university” are different from the ones of “middle school”.


179

SUBSET(P4(Lecturer), P1(Teacher)) (in method 2) records the subsumption relationship between “Lecturer” and “Teacher” defined in lexon P4 and P1. A “Teacher” is a super type of “Lecturer”. SUBSET() constraint can also be applied in the same lexon, e.g. in method 1. Both methods are equivalently used. 4.1.3 Join Dependencies The join dependencies are the constraints while joining two templates. We often use them together with other constraints, such as subset and equal. Take the same example as in section 4.1.2 (Fig. 4), we write the following SDT commitment to indicate that “Teacher” is joined with “Lecturer”. (P1 = [Teacher, teach, , Course], P4 = [Lecturer, has ID, , Personnel ID]): JOIN_DEP(P1(Teacher), P4(Lecturer)), SUBSET(P1(Lecturer), P3(Teacher)). JOIN_DEP(P1(Teacher), P4(Lecturer)) means that the concepts “Teacher” and “Lecturer” defined in lexons P1 and P4 have a join dependency. In this example, the join dependency is used together with a subset constraint – “Lecturer” is a subset of “Teacher”. Thus, when we join “Lecturer” into “Teacher”, the structure of “Lecturer” is not changed. Fig. 5 shows the result of applying this SDT commitment in the system. If we apply a join dependency combined with an equal constraint, then the templates of both concepts should be updated.

Fig. 5. the result of applying join dependency to the two templates in Fig. 4

4.1.4 Other Dependencies Other dependencies have been studied so far, e.g. the constraints of information modeling and relational database in [13]. We mostly use the following constraints • •

Uniqueness. The uniqueness constraint and how to write its commitment are discussed early in section 2.1. Mandatory. A role is mandatory “if and only if, for all states of the database, the role must be played by every member of the population of its object type…” [13, pp. 166]. For example, we want to have a fact that “a lecturer must have a Personnel ID”. We then apply a mandatory constraint on the lexon written as:

P4 = [Lecturer, has ID, , Personnel ID]: MAND(P4(has ID)). •

Equality. The equality constraint discussed in [13, pp. 231] is very similar to the multivalued dependency we deal with in section 4.1.1. A big difference between them is that an equality constraint is applied to two concepts with different names, while a multivalued dependency deals with two concepts

180


with the same name. For example, the following SDT commitment means that “Course” and “Lecture” are equivalent. (P1 = [Teacher, teach, , Course], P5 = [Teacher, teach, Lecture]): EQUAL(P1(Course), P5(Lecture)). •

Exclusion. Type A and type B are mutually exclusive if and only if they don’t share any members. This dependency constraint is checked when a new concept type is introduced as a member of Type A or B. If a domain expert tries to add the same concept type to another type (B or A), he would violate this constraint. Suppose we don’t allow a teacher to be a student at the same time. We shall apply the dependency of exclusion to “Teacher” and “Student” written in the following SDT commitment:

(P1 = [Teacher, teach, , Course]), P6 = [Student, 12 learn, , Course]): OR (P1(Teacher), P6(Student)). 4.1.5 Tabular Reports Generated by SDT in the DOGMA-MESS Top Down Process Early in this section, we have discussed different kinds of template dependencies and how to write the SDT commitments respectively. In each DOGMA-MESS top down process iteration, SDT is used not only to constrain the templates dependencies, but also to generate tabular reports. Once a domain expert introduces a new concept, the system checks the dependencies stored in the SDT commitments. A tabular report, which is generated when a domain expert tries to add new concepts, stores the dependencies information of every new concept. Table 2 is a simple example generated when a domain expert add “Trainer” as a subtype of “Lecturer”, and “Tutor” as a subtype of “Teacher”. As there is a subset dependency between “Lecturer” and “Teacher” (“Lecturer” is a subset of “Teacher”, section 4.1.2), the subtypes of “Lecturer”, e.g. “Trainer”, are automatically added as a subtype of “Teacher” (the column named “Trainer”, Table 2). The reason is explained in the column named “1Lecturer” in Table 2: “Lecturer” is the subset of “Teacher”. Table 2. A table that shows validity of introduced concepts Candidate

Trainer

Tutor

Apprentice

1

Lecturer

Teacher

Teacher, Student

Teacher

Lecturer

2

Teacher

Dependency SUBSET EXCLUSION

Student

Action/Decision Add

*

Add to others

*1 {Teacher}

*

Add conflicted Add Action Denied 12

*2

OR is derived from logical disjunction operator – or. For the exclusive-or in logical theory can be considered equivalently to the mutual exclusion in set theory.


181

If the domain expert tries to add “Apprentice” as a subtype of both “Teacher” and “Student”, it cannot be added because “Teacher” and “Student” have exclusion dependency (see the SDT commitment of exclusion in section 4.1.4). Thus, the add action of “Apprentice” is denied (the column named “Apprentice”, Table 2), the reason of which is explained in the columns named “2Teacher” in Table 2. Table 2 shows a very simple example that the DOGMA-MESS top down processes can benefit from SDT. In the next subsection, how to use SDT to guide the concepts elicitation in a bottom up process is discussed. 4.2 Bottom Up: Use Semantic Decision Tables to Guide the Concepts Elicitation Within an ontology version, the system needs to select a few concepts amongst the ones at OTO level. We use Semantic Decision Tables (SDT) to store the lifting algorithms and manage the process.

Fig. 6. The concept “Teacher” is designed at UCO level

Fig. 7. A new relevant concept “Patience” is introduced by a domain expert at OTO level

Let’s first look at a simple lifting algorithm. When we lift a concept from OTO level to LCO level, we need to choose some concepts at OTO level. Let Sc be the concept set at OTO level, and let Sl be the resulting lifted concept set at LCO level. In order to compute this process automatically, we hereby introduce two important condition stubs used to form SDT condition lexons – the relevance score Rc and the interest point Ip. A concept Ci at OTO level is considered as a relevant candidate concept when it gets certain amount of relevance score Rc. Rc is set zero when a new concept is defined at the first time. It increases when the concept is defined in other organizational ontologies designed by different domain experts. For example, if we get the concept “skill of safety” defined within the same context from two different organizational ontologies, the Rc of this concept is increased by one. Interest point Ip starts from zero. Ip is assigned to an existing concept at UCO level. It increases by one when a new concept, which is connected to this existing concept, is introduced. For example, we have the concept definition of “Teacher” at UCO level (Fig. 6). When a domain expert adds a new relevant concept “Patience” to “Teacher” at OTO level (Fig. 7), Ip of “Teacher” is increased by one. Ip reflects the focusing interests of the stakeholder community. We consider the concept “Patience” as a candidate concept that will be lifted to UCO level only when Ip of its connected concept “Teacher” meets a certain threshold value. When a new concept at OTO level is connected to more than one concept at UCO level, we choose the biggest number of Ip. Accordingly, we formalize a lift rule into the SDT commitments as illustrated in Table 3.

182


Table 3. An example of SDT commitments and their explanations in the natural language ID 1

Commitment (P1 = [concept, has, is of, Ip], P2 = [Ip, is of, has, concept])

Verbalization Each concept has at most one interest point. And each interest point is of at most one concept.

: UNIQ (P1, P2). 2

(P3 = [concept , has, is of, Rc], P4 = [Rc, is of, has, concept])

Each concept has at most one relevance score. And each relevance score is of at most one concept.

: UNIQ (P3, P4). 3

(P5 = [concept, has, is of, Rc],

The fact that a concept is lifted to UCO level depends on two conditions: 1. whether its relevance score is more than T1 or not; And 2. Whether its interest point P7 = [concept, is lifted to, contain, UCO level]) is more than T2 or not. : IMP(AND(P5(Rc) >= T1, P6(Ip)>=T2 ), P7). P6 = [concept, has, is of, Ip],

Commitment 1 in Table 3 uses the uniqueness constraint (‘UNIQ’) to express the one-to-one relationship between ‘concept’ and ‘interest point’. So does commitment 2. Commitment 3 uses the propositional connectives ‘IMP’ (the implication connective) and ‘AND’ (the conjunction connective) to express that: a candidate concept can be lifted to UCO level if and only if its relevance point and interest point meet their threshold (‘T1’ and ‘T2’). Based on Table 3, two concepts at the OTO level – ‘Patience’ and ‘Oral comprehension’, which are considered as two candidate concepts, are analyzed in Table 4. Table 4 contains the decision whether the concepts ‘Patience’13 and ‘Oral comprehension’ at OTO level can be lifted to LCO level or not. The tabular report is automatically generated by the SDT plug-in in the DOGMA-MESS tool14. As the relevance score (20) of the concept ‘Oral comprehension’ doesn’t reach the threshold (25), it is kept at OTO level for next MESS iterations. Table 4. A tabular report generated based on the SDT, which contains the commitments in Table 3 (T1=25, T2=25) Candidate concept

Patience

Oral comprehension

…

Condition Super Type

N/A

Competence

…

Relevance Score

30

20

…

Relevant concept at UCO

Teacher

Teacher

…

Interest Point

30

30

…

...

…

…

Action/Decision Keep for next MESS iteration Lift to LCO

13 14

… …

* *

Its concept is given by Fig. 7. The DOGMA-MESS tool currently developed in STARLab is a web portal to assist domain experts to design ontologies: http://www.dogma-mess.org/


183

The resulting concept set is then provided to the core domain experts, who are responsible for standardizing the concepts and merging them at UCO level. As the concepts are defined and visualized in the Conceptual Graph models (e.g.), the core domain experts can use many available conceptual graph matching manners, such as in [16], to merge the new concepts automatically into the existing concepts. During this merging phase, some extra links between new concepts and existing concepts need to be constructed. For example, conceptually equivalent concepts need to be linked with the “equivalent” canonical relation. The merging process results in several reorganized conceptual graphs at UCO level. 4.2.1 Process Evaluation Using SDT Table 3 is as an example of a lifting rule. In practice, users are free to choose their preferred lifting rules. Furthermore, the algorithm in SDT can be evaluated. Let Ncan be the number of the candidate concepts, which resides at OTO level. And let Nsel be the number of selected concepts that are lifted. The concept selection rate O is defined as: O = Nsel/Ncan. The selection rate O can be studied by setting up different values of T1 and T2 (Table 5). Table 5. A result of the selection rate O in tabular report15 taking from the PROLIX project16

T1

Exp1

Exp2

Exp3

Exp4

Exp5

8

20

37

44

56

T2

15

15

15

15

15

O

25%

12%

12%

10%

0%

In practice, core domain experts need to adjust the parameters T1 and T2 based on real situation. For example, they may set T1 and T2 higher when there are a lot of concept candidates and they don’t have a lot of time to standardize them. An advantage of using the generated tabular reports (e.g. Table 5) is to help the core domain experts to determine which values that they should assign to T1 and T2. In this section, we focus on how Semantic Decision Tables can be used in both bottom-up and top-down processes of the meaning evolution cycle in DOGMA-MESS. In the top-down processes, Semantic Decision Tables are used to constrain the dependencies between the templates at the Upper Common Ontology level. The accuracy of the concepts provided by the domain experts gets improved. In the bottom-up elicitation processes, Semantic Decision Tables bring the effectiveness to the system. For the behaviors of the community are captured, managed and guided by Semantic Decision Tables. The concepts created by the domain experts are no longer selected manually. In addition, the lifting algorithms are no longer hard coded in the system. Thus, the flexibility increases. 15 16

This report is also automatically generated in the tool. The objective of PROLIX is to align learning with business processes in order to enable organizations to faster improve the competencies of their employees according to continuous changes of business requirements. In this project, DOGMA-MESS is used to create the competency ontology for every domain of the test beds. URL: http://www.prolixproject.org/

184


5 Conclusion and Future Work In this paper, we focus on the discussion of using Semantic Decision Tables in community grounded, knowledge intensive Meaning Evolution Support Systems for ontology evolution. As DOGMA-MESS is recently developed at our STARLab as a collection of meaning evolution support systems, we use it as a sample17 system to illustrate the strength of Semantic Decision Tables (SDTs). We improve the accuracy and the effectiveness of DOGMA-MESS by embedding SDT in it. More explicitly, SDTs are utilized when designing the templates dependencies in the top-down process, and lifting new relevant concepts to the domain level in the bottom-up process. In the top-down process, the decision rules of SDTs are used to check the validity of the concepts. By doing that, the accuracy is improved. We specified seven categories of templates dependencies that are mostly used. They are multivalued dependencies, subset dependencies, join dependencies, uniqueness dependencies, mandatory dependencies, equality dependencies and exclusion dependencies. Except the multivalued dependencies, others are also called “functional dependencies”. In the bottom-up process, the decision rules of SDTs are utilized to draw the decisions whether the concepts are lifted or not. The decisions are executed automatically in the processes. Thus, the system is enhanced with effectiveness. In addition, we emphasize the importance of capturing the community’s behavior and guide it respectively. The community’s behaviors are coded as the parameters in the lifting algorithm stored in SDTs. After the execution of each process (no matter whether it is the top-down process or the bottom-up process), a tabular report is generated based on SDTs. We consider such a tabular report as a complementary mechanism for non-technical users. We have developed a tool to support constructing and visualizing SDTs. The current version supports modeling some specific commitment types. A web portal to support DOGMA-MESS methodology has been developed [6]. In this paper, we focus on the system management while introducing new concepts that matter the overlapped interests of the community. In practice, we observe that the concepts in an ontology can be obsolete after a long time period. Therefore, we need to update the ontology by modifying (e.g. deleting, replacing with others, and redefining) the concepts. A future work is to use SDTs in this kind of modifying processes. Acknowledgments. The research is partly supported by the EC Prolix project.

References [1] Aschoff, F.R., Schmalhofer, F., van Elst, L.: Knowledge mediation: a procedure for the cooperative construction of domain ontologies. In: Proc. of the ECAI 2004 workshop on Agent-Mediated Knowledge Management, pp. 29–38 (2004) [2] Benjamin, P., Menzel, C., Mayer, R., Fillion, F., Futrell, M., de Witte, P., Lingineni, M.: Idef5 method report. Technical Report F33615-C-90-0012, Knowledge Based Systems Inc., Texas (1994) 17

We argue that the usage of SDT is not limited to DOGMA-MESS system.


185

[3] Braun, S., Schmidt, A., Walter, A.: Ontology maturing: a collaborative web 2.0 approach to ontology engineering. In: Proceedings of WWW 2007 (2007) [4] Casanovas, P., Casellas, N., Tempich, C., Vrandecic, D., Benjamins, R.: Opjk modeling methodology. In: Lehmann, J., Biasiotti, M.A., Francesconi, E., Sagri, M.T. (eds.) Legal Ontologies and Artificial Intelligence Techniques, International Conference on Artificail Intelligence and Law (ICAIL) WorkshopSeries, vol. 4, pp. 121–134. Wolf Legal Publishers (2007) [5] de Moor, A.: Ontology-Guided Meaning Negotiation in Communities of Practice. In: Mambrey, P., Gräther, W. (eds.) Proc. of the Workshop on the Design for Large-Scale Digital Communities at the 2nd International Conference on Communities and Technologies (C&T 2005), Milan, Italy (2005) [6] de Moor, A., De Leenheer, P., Meersman, R.: DOGMA-MESS: A Meaning Evolution Support System for Interorganizational Ontology Engineering. In: Schärfe, H., Hitzler, P., Øhrstrøm, P. (eds.) ICCS 2006. LNCS (LNAI), vol. 4068, pp. 189–203. Springer, Heidelberg (2006) [7] Euzenat, J.: Building consensual knowledge bases: context and architecture. In: Mars, N.J.I. (ed.) Towards Very Large Knowledge Bases – Proc. of the KB&KS 1995 conference, pp. 143–155. IOS press, Amsterdam (1995) [8] Fernández, M., Gómez-Pérez, A., Juristo, N.: Methontology: from ontological art towards ontological engineering. In: Proceedings of the AAAI 1997 Spring Symposium Series on Ontological Engineering, Stanford, USA, pp. 33–40 (1997) [9] Gómez-Pérez, A., Fernández-López, M., Corcho, O.: Ontological Engineering. Advanced Information and Knowledge Processing Series. Springer, Heidelberg (2004) [10] Gruber, T.R.: Toward Principles for the Design of Ontologies Used for Knowledge Sharing. In: Workshop on Formal Ontology, Padva, Italy. In book, Guarino, N., Poli, R. (eds.): Formal Ontology in Conceptual Analysis and Knowledge Representation. Kluwer Academic Publishers, Dordrecht (1993) [11] Grüninger, M., Fox, M.: Methodology for the design and evaluation of ontologies. In: Skuce, D. (ed.) IJCAI 1995 Workshop on Basic Ontological Issues in Knowledge Sharing (1995) [12] Guarino, N., Poli, R. (eds.): Formal Ontology in Conceptual Analysis and Knowledge Representation. The International Journal of Human and Computer Studies 43(5/6) (1995) (special issue) [13] Halpin, T.: Information Modeling and Relational Database: from Conceptual Analysis to Logical Design. Morgan-Kaufmann, San Francisco (2001) [14] Madhavan, J., Bernstein, P., Domingos, P., Halevy, A.: Representing and reasoning about mappings between domain models. In: Eighteenth National Conference on Artificial Intelligence (AAAI 2002), Edmonton, Canada, pp. 80–86. American Association for Artificial Intelligence (2002) ISBN:0-262-51129-0 [15] Mizoguchi, R.: Tutorial on ontological engineering. New Generation Computing 21(2) (2003) [16] Myaeng, S.H., Lopez-Lopez, A.: Conceptual graph matching: a flexible algorithm and experiments. International Journal of Pattern Recognition and Artificial Intelligence 4(2), Special issue Conceptual graphs workshop, pp. 107–126. World Scientific Publishing Co., Inc, Singapore (1992) ISSN:0218-0014 [17] Sadri, F., Ullman, J.D.: Template dependencies: a large class of dependencies in Relational Databases and its complete axiomatization. Journal of the ACM (JACM) 29(2), 363–372 (1982)

186


[18] Schreiber, G., Akkermans, H., Anjewierden, A., de Hoog, R., Shadbolt, N., Van de Velde, W., Wielinga, B.: Knowledge Engineering and Management — The CommonKADS Methodology. The MIT Press, Cambridge (1999) [19] Sowa, J.F.: Conceptual Structures: Information Processing in Mind and Machine. Addison-Wesley, Reading, Massachusetts (1984) [20] Sowa, J.: Knowledge Representation: Logical, Philosophical, and Computational Foundations. Brooks Cole Publishing Co., Pacific Grove (2000) [21] Spyns, P., Meersman, R., Jarrar, M.: Data Modeling versus Ontology Engineering. SIGMOD Record: Special Issue on Semantic Web and Data Management 31(4), 12–17 (2002) [22] Spyns, P., Tang, Y., Meersman, R.: A model theory inspired collaborative ontology engineering methodology. Journal of Applied Ontology, Special issue on Guizzardi, G., Halpin, T.: Ontological Foundations for Conceptual Modeling, vol. 4(1), pp. 1–23. IOS Press, Amsterdam (2007) [23] Tang, Y.: On Conducting a Decision Group to Construct Semantic Decision Tables. In: Meersman, R., Tari, Z., Herrero, P. (eds.) OTM-WS 2007, Part I. LNCS, vol. 4805, pp. 534–543. Springer, Heidelberg (2007) [24] Tang, Y., Meersman, R.: Towards building semantic decision table with domain ontologies. In: Man-chung, C., Liu, J.N.K., Cheung, R., Zhou, J. (eds.) Proceedings of International Conference of information Technology and Management (ICITM 2007), pp. 14–21. ISM Press (2007) ISBN 988-97311-5-0 [25] Tang, Y., Meersman, R.: On constructing semantic decision tables. In: Wagner, R., Revell, N., Pernul, G. (eds.) DEXA 2007. LNCS, vol. 4653, pp. 34–44. Springer, Heidelberg (2007) [26] Tang, Y., Meersman, R.: Organizing Meaning Evolution Supporting Systems Using Semantic Decision Tables. In: Meersman, R., Tari, Z. (eds.) OTM 2007, Part I. LNCS, vol. 4803, pp. 272–284. Springer, Heidelberg (2007) [27] Uschold, M., King, M.: Towards a methodology for building ontologies. In: Skuce, D. (ed.) IJCAI 1995 Workshop on Basic Ontological Issues in Knowledge Sharing (1995) [28] Zhao, G., Meersman, R.: Architecting ontology for Scalability and Versatility. In: Meersman, R., Tari, Z. (eds.) OTM 2005. LNCS, vol. 3761, pp. 1605–1614. Springer, Heidelberg (2005)

Combining User Profiles and Situation Contexts for Spontaneous Service Provision in Smart Assistive Environments Weijun Qin1,2 , Daqing Zhang2 , Yuanchun Shi1 , and Kejun Du2,3 1

3

Key Laboratory of Pervasive Computing, Ministry of Education, Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China [email protected], [email protected] 2 TELECOM & Management, SudParis, Evry 91000, France [email protected], [email protected], [email protected] School of Computer Science, Northwestern Polytechnical University Xi’an 710072, P.R. China [email protected]

Abstract. Wireless hotspots are permeating our workplace, home and public places bringing interesting services and spontaneous connectivity to mobile users. Since mobile users will be immersed in thousands of different kinds of pervasive services, it’s of paramount importance to provide appropriate services to the right person in the right form. Existing approaches offer a rather simplistic solution to the problem and seldom consider the multi-dimensional contextual information. In this paper we present a service provision mechanism which can enable effective service provision based on semantic similarity measure with the combination of user profiles and situation context in WLAN enabled environment. We illustrate our ideas through a scenario of service discovery and provision in public places (e.g. shopping mall) where a disabled person can select appropriate services with the constraints of his mobile device.

1

Introduction

Recent research advances in the areas of pervasive and ubiquitous computing focus on novel application scenarios that are related to various living spaces such as home, office, hospital, shopping mall, museum, etc.. Mobile and pervasive technologies are opening new windows not only for ordinary people, but also for elderly and dependent people due to physical/cognitive restriction. As wireless hotspots are permeating the globe bringing interesting services and spontaneous connectivity to mobile users, one trend is to gradually transform various physical environments into ”assistive smart spaces”, where dependent people can be supported across the heterogeneous environments and be fully integrated into the society with better quality of life and independence. F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 187–200, 2008. c Springer-Verlag Berlin Heidelberg 2008

188

W. Qin et al.

In order to provide services in various assistive environments, impromptu service discovery and provision with heterogeneous mobile devices become critical. There are two possible ways to make the services accessible to mobile users. One way is to get connected to a global service provider and access the location-based services through the service provider. In this case, the service provider needs to aggregate all the services in the smart spaces and have the indoor/outdoor location of the user. Due to the evolutionary nature of the autonomous smart spaces, it is unlikely to have such a powerful service provider in the near future. The other way, which we are in favor of, is to enable the mobile user interact with individual smart space directly when the mobile user physically enters the space, that is, the mobile user can automatically discover and provision the services provided by the smart spaces. In the latter case, the mobile user needs to get connected directly to the individual smart space using short range wireless connectivity (such as WLAN or Bluetooth). There are two key challenges to achieve impromptu service discovery and provision. The first challenge is how to automatically discover the relevant services upon entering the smart space, making reasonable assumptions on mobile devices and service providers who offer services in the specific smart space; The second challenge is how to automatically provide appropriate services to the right person with the right form with the consideration of contextual information in smart environment. To address the first challenge, various software [1][2] and hardware [3] ”plugin” solutions have been proposed, in addition to the service discovery standards such as UPnP [4], Jini [5], etc. The key idea was to embed specialized software or hardware components in both the mobile device and the environment which follow the same service discovery protocol, however, there is still no dominant solution that was accepted by the whole community. Compared to the first challenge, the second challenge received much less attention. Existing approaches eithor offer a rather simplistic solution to the problem and seldom consider the multi-dimensional contextual information, or are still limited to key-value based semantic matching to select appropriate service to the user [6]. In this paper, we aim to propose service provision framework combining the user profiles and situation contexts to select appropriate services to the end-users from thousands of desultory services with the constraints of mobile device. The rest of the paper is organized as follows: Section 2 starts with a use scenario for assisting a person in a shopping mall, from the use case six system requirements are identified for supporting impromptu service discovery and access in unfamiliar assistive environments. After presenting the overall system for impromptu service discovery in Section 3, a context-aware service provision process based semantic similarity measure are described in Section 4. Then Section 5 gives the design issues. Finally, some concluding remarks are made.

2

Use Scenario

In this section we introduce an assistive service scenario in a shopping mall that needs secure impromptu service discovery and access in smart environments. In

Combining User Profiles and Situation Contexts

189

this paper, we focus on the issue of context-aware service provision with the consideration of user profiles and situation contexts. 2.1

Service Provision in Shopping Mall Scenario

Bob, a disabled person with wheelchair, would like to buy a dust coat in a shopping mall. Upon entering the shopping mall, he is authenticated implicitly and presented 3 general services on his wireless PDA including store direction service, goods directory service, and discount service. He uses goods directory service and discount service to search several dust coat brands with discount according to his preferences such as Burberry, Boss, etc. He decides to look around Boss speciality store first, and uses store direction service to find the location. When Bob decides to go to the store, he is presented with 2 in-store services: one is coat finder which helps him locate the coat in a specific shelf, the other is coat recommendation service that recommends coats based on his preference. Besides, Bob could uses general goods directory service to compare different brands to help him make decision. 2.2

Requirements of the Scenario

From the scenarios described above, we can elicit the following requirements that support impromptu, secure and context-aware service discovery and provision with mobile devices: - R1: Minimum assumption on mobile devices: One of the first ideal features for impromptu service access is that the mobile device doesnt need to install any specialized software or hardware in order to interact with the services in a certain environment. - R2: Automatic discovery of services in unfamiliar environments: Another ideal feature for impromptu service access with mobile devices is the automatic discovery of the relevant services in the new environments, when needed. - R3: Heterogeneous service aggregation: As the devices and services are heterogeneous, they also vary greatly in different environments. There should be an effective mechanism and platform that facilitates the management of both static and dynamic services associated with each smart space. - R4: User and environment context consideration: In order to present the right services to the right person in the right form, user and environment context should be taken into account for service discovery and access in each smart space. - R5: User interface generation according to user context and device capability: As the users and access devices vary, impromptu and personalized user interface generation based on user preference and device capability should be supported. - R6: Secure service access with ensured privacy: When the mobile users access certain services, user inputs and the output of the services need to be exchanged in a secure manner. User privacy should be guaranteed so that he might choose not to reveal his identity during service discovery and access.

190

3

W. Qin et al.

Impromptu Service Discovery and Access

To address the above-mentioned requirements, we propose and implement a impromptu service discovery and access framework which enables the services in unfamiliar environments accessible to any WLAN enabled mobile device, according to individual user’s preference, situation and device capability. While an open standard-based service platform OSGi [7] has been adopted for service aggregation (R3), the Semantic Space [8] is deployed for user and environment context management (R4). The spontaneous interaction framework(R3), security and privacy issue (R6) and user interface generation (R5) have been studied and discussed in separate papers. In this section, we mainly focus on what the minimum requirements are for mobile devices in our design (R1) and how we achieve impromptu service discovery under the minimum assumptions (R2). In the next section we will present the service provision process combining with user and situation contexts (R4). 3.1

Minimum Assumptions on Mobile Device

In order to communicate locally with the smart space, we assume that the mobile devices should have at least the built in WiFi chipset to allow wireless connectivity and a web browser to access the web server hosting the services of the smart spaces. The requirement on mobile devices is minimum as it doesn’t rely on any specialized software or hardware. 3.2

Impromptu Service Discovery Mechanism

Following the minimum assumption on mobile devices, it requests that the services are associated with a specific ”smart space” like a shopping mall, a lift, or a coat shop. When the mobile devices are detected within a certain smart space, the services can be automatically discovered and presented in the mobile devices. To enable the service aggregation and automatic discovery, a captive portal and service portal mechanism is proposed. We adopted the open standard-based service platform such as OSGi [7] to aggregate various services and provide the service portal in a specific smart space, as such various devices and communication protocols are already supported. The main function of captive portal is to connect the mobile device to a wireless network automatically and direct the web browser of the mobile device to the service portal of a specific smart space, it enables the automatic discovery of the services which are provided by each service provider available in the smart space. In our approach, we suppose each service provider records the user’s behavior and infers the semantic dependency between the service and the user using reasoning model (e.g. Dynamic Bayesian Network) [9]. Different from the normal captive portal mechanism, our captive portal is able to detect what the preferred networks list the client device has, it can then configure the WLAN access point in such a way the mobile device with wireless auto configuration capability can automatically connect to the network. Thus when the mobile device switches on, the device is automatically connected to the network without


191

Fig. 1. When the user enters a smart space, his mobile device is detected by the Captive Portal and connected to the wireless access point(1), and after launching the web browser, his web browser is directed to the Service Portal(2). He can then browse the available services(3), and invoke a service through his mobile device(4).

configuration. When the web browser is launched by the user, the mobile device will be directed to the service portal of the smart space so that the user can choose to invoke the service needed. The whole process is illustrated in Fig. 1. 3.3

Overall System Architecture

The overall system architecture is shown in Fig. 2 where each smart space has an impromptu service discovery and provision framework embedded as a server. When the dependent people move from one space to another with a WLAN enabled mobile device, her mobile device will be detected and connected to a wireless network in a smart space. And the web browser will be directed by the captive portal in the framework to the service portal of the new smart space which aggregates and hosts the services offered. The middleware core refers to the OSGi service platform. The context manager is responsible for context representation, aggregation, reasoning, storage and query processing, which facilitates service discovery and access with mobile devices [8]. Combining user profiles and situation context, The SemanticMatcher delivers the appropriate associated services to the users automatically.

4 4.1

Context-Aware Service Provision Process Context Modeling

Context is a description of semantic entities such as the user, a device, the situation and the environment [10]. It’s obvious that there are many different types of contextual information that can be used by applications in a smart assistive

192

W. Qin et al. Application Layer

Situation Profile Repository

Semantic Matcher

Reasoning Engine

UI Generation

Service Description Repository

Service Invocation

Service Discovery

. . .

Context Manager

Security

Administration

HTTP Request & Response

Doman‐Specific Services (e.g. Home, Office, Public Places)

Service Layer

OSGi Service Framework Java Runtime Environment System Layer

Operating System Hardware (e.g. IEEE 802.11, Ethernet, Bluetooth)

Physical Interface Layer

Fig. 2. Impromptu service discovery and provision framework for multiple spaces

environment, including physical contexts (e.g. time, location), personal contexts (e.g. identity, preference, mood), device contexts(e.g. display size, power), activity contexts(e.g. shopping list, meeting schedule). Besides those, other types of information are still considered as crucial context which may be invisible to the participants e.g. systematic contexts (e.g. CPU power, network bandwidth), and application contexts (e.g. agents, services), environmental contexts (e.g. light, temperature), etc. In order to distinguish the specific properties or attributes of different kinds of contexts, Context Modeling plays an important role in designing and developing context-aware applications for smart assistive environment. Based on the previous work on context modeling in [11], we build an ontology-based context model with the powerful capabilities of RDF (Resource Description Framework) and OWL (Web Ontology Language). In our model, we divide context ontology into two sets: core context ontology for general conceptual entities in smart assistive environment and extended context ontology for domain-specific environment, e.g. shopping mall domain. The core context ontology attempts to define the general concepts for context in smart environment that are universal and sharable for building context-aware applications. The extended context ontology attempts to define additional concepts and vocabularies for supporting various types of domain-specific applications. The core context ontology investigate seven basic concepts of User, Location, Time, Activity, Service, Environment and Platform, which are considered as the basic and general entities existed in smart assistive environment. Part of the core context ontology is adopted from several different widely-accepted consensus ontologies, e.g. DAML-Time [12], OWL-S [13], etc. Since user profiles, environmental contexts and device capability have deeply influenced service provision process in smart assistive environment for mobile users, the model considers context information ranging from user profiles, location, time and activity to environment capability as user’s situation context. Fig. 3. shows a partial context ontology used in the shopping mall scenario


193

Fig. 3. A partial context ontology for service provision process in smart assistive environment, which delineates the concepts and relationships between the user’s situation context and service context

above which delineates the concepts and relationships among the user’s situation context and service context. The User ontology in our sense will consist in a classification of users type and of features of users, and each categorized class will be linked with associated information, such as preferences, knowledge, capabilities and so on. The Service ontology defines the multi-level service specifications in order to support service discovery and composition. Each instance of Service will present a ServiceProfile description, be describedBy a ServiceModel description, and support a ServiceGrouding description in OWL-S specification. The Condition property and the additional hasPrecondition parameter in OWLS specify one of the preconditions of the service which can be considered as the instance defined according to the restriction of user’s situation contexts. User Profiles. The User profiles are of major importance to provide intelligent and personalized service in smart environments [6]. Based on User Ontology, we divide the user profile into two parts: static part which is relevant to user’s type and characteristics, such as Knowledge, PersonalData(e.g. demographic features), Characterics(e.g. profession, language, etc.), Capabilities(e.g. competencies), PhysicalState(e.g. blood pressure, etc.), CognitiveState(e.g. mood, etc.) and so on, and dynamic part which is relevant to user’s behavior, such as Interest, Preference, InvolvedActivity, InvolvedLocation, and their subclasses [14]. The profiles can be stored in the PDA (client) and updated according to specific contexts. The same user can have multiple profiles. Each profile could be the one corresponding to the current user’s location or activity. Table 1 shows a possible profile for the same person related to his role as customer in the scenario. Defining user ontologies facilitates to provide general and shared common concepts and properties relevant to user’s personalized information, preferences and activities.

194

W. Qin et al. Table 1. A partial example for user profile used in shopping mall

ID: QIN001 ... InvolvedActivity: Shopping InvolvedLocation: Shopping Mall Gender: Male Age: 40 Language: French Capabilities: Disabled Interest: Coat Preference: Shape, Color, Material, Size, Brand ...

Situation Context Description. The Situation Context captures the current user’s situation including Location, Activity, Time and PhysicalEnvironment information which is obtained from embedded sensors in smart environment. After the representation, aggregation and calculation processes of context-aware system, situation context can be represented as RDF description form. Table 2 shows an partial example for situation context description in the system. From the basis of context model, the hierarchical structures of Location, Time, Activity and PhysicalEnvironment are well organized with the capability of ontologies. For example, The Store is the subclass of the Shopping Mall with the different granularity of Location. In sensor-based context-aware systems, each data result from sensors may have imprecise factors such as sensor disconnection, signal distortion, etc. With the consideration of sensor data imperfectness, probabilistic model is adopted to represent imperfect data with the form of P r(A) = a, where A represents the situation context instance, and a represents probability value which is calculated from context reasoner. Table 2. An partial example for situation context description used in a store

... Location: Boss speciality store Device: PDA Time: 1:00pm EST, Jan. 1st, 2008 ...

Service Description. The Service Description represents the capability and properties of services and the degree of required information. Current initiatives such as UDDI (Universal Description Discovery and Integration), WSDL (Web Service Definition Language) and WSFL (Web Services Flow Language) are considered complementary to the expression of service definition and properties


195

description, however these initiatives lack in considering of combing with the situation contexts in real smart environment. From the view of end-user interaction, a taxonomy of service provide a schema to describe the features of service instance including UUID, ServiceName, ServiceProvider, Category, Precondition, InputParameter, OutputParameter, Dependency etc. The UUID provides the unique identity of services in global domain. The ServiceName presents the service instance’s name. The Category identifies the type of services for end users, such as Navigation Service, Recommendation Service. The InputParameter and OutputParameter aim to delivery the interface parameters from/to the system. The Dependency is used to describe the semantic dependency between the service and the user which is captured, inferred and measured by the service provider. The Dependency has two properties Feature and Value with the form of KeyValue pair: the former is to describe the context objects related to the service interaction, and the latter one is to measure the semantics relevance degree between the two objects that ranges from -1 to 1. For example, the Coat Recommendation Service enumeratess the semantic dependency between the user’s feature, such as Gender, Age, etc., and evaluate the semantics relevance degree between each feature and the service. Table 3 shows an partial example for the Coat Recommendation Service provided by the store. Table 3. An partial example for service description provided by a store

... UUID: S001 ServiceName: Coat Recommendation Service ServiceProvider: Boss speciality store Category: Recommendation Service Feature: Gender DependencyDegree: 0.95 Feature: Age DependencyDegree: 0.8 ...

4.2

Semantic Matching Based on Similarity Measure

The goal of semantic matching between context objects is to determine which user or service profile item with ratings is more relevant for the current context. For example in the scenario above, when Bob wants to enter the shopping store, the Recommendation Service provided by the store may be more appropriate for him based on his location and involved activity. Therefore for each context type, the quantifiable measure of the similarity should be needed to evaluate the semantics relevance between two context objects. Formally we define context (including user profile, situation context, and service) here as a n-dimension vector based on Vector Space Model [15] to represent context object: C = (c1 , c2 , ..., cn ) (1)

196

W. Qin et al.

where ci , (i ∈ 1..n) is quantified as a context type (e.g., Activity, Location) from the feature extraction ranging from -1 to 1. We use the Pearson’s correlation coefficient to calculate the similarity weight between two different contexts with their rating values. The similarity of context C1 and C2 is formally defined as, n αi βi C1 · C2 Similarity(C1 , C2 ) = = n i=12 n (2) 2 ||C1 || × ||C2 || i=1 αi i=1 βi where C1 = (αi ), C2 = (βi ), (i ∈ 1..n) and Similarity(C1 , C2 ) returns the relevance of two context over all the item ratings from the system. The advantage of semantic similarity measure is that similarity-based approaches combines the semantics of both the object’s hierarchical (subclass relationship) and relevant non-hierarchical (cross links) components, while the retrieved results from keyword-based approaches have low recall (some relevant items are missing) and low precision (some retrieved item are irrelevant)[6]. The disadvantage of Pearson correlation coefficient is that the correctness of the vector item’s assumption has influenced the similarity between two context objects, while Pearson correlation indicates the strength of a linear relationship between two n-dimensional vectors. Given the user profiles set and situation context captured from the system, the service provision process first select appropriate user profile based on current user’s location or involved activity information. And then, combine with the selected user profile U ∗ ={u∗1 , u∗2 , ..., u∗m } and situation context C ={w1 , w2 , ..., wn } as composite context CC ={v1 , v2 , ..., vp }={u∗1 , u∗2 , ..., u∗m , w1 , w2 , ..., wn }, where p = m + n and a service description S ={s1 , s2 , ..., sp }, the cosine similarity is calculated as p vi si S · CC Similarity(S, CC) = = p i=12 p (3) 2 ||S|| × ||CC|| i=1 vi i=1 si Based on the similarity result, the M services with high ratings are selected as the nominated presented to the end user. The number M depends on the UI design of user’s client device. {S ∗ } = {Uj |maxm {Similarity(S, CC)}}

5

(4)

Design and Implementation Issues

In this section, we discuss design issues for implementing spontaneous service provision combining with user profiles and situation context. 5.1

Captive Portal to Enable Spontaneous Access without Zero Installation and User Profile Automatic Infusion

By means of the captive portal technique, clients can spontaneously discover the services available in the smart space. When the client’s mobile device connects


197

to the wireless hotspot, it issues a DHCP request, and the router would assign with an IP for the device, but the device would initially be firewalled. When the client browser is opened, and requests a website, the listener on the router would return the IP address of the captive portal, instead of the requested site. In our solution, clients are seamlessly authenticated at the captive portal and redirected to the Service Portal, where they can discover available services. Since the captive portal mechanism transparently redirects the client to the Service Portal, through DNS manipulation, the client only needs wireless connectivity and a web browser. No additional software or hardware beacons are needed. The process of capturing browser requests and redirecting to the Service Portal is illustrated in Fig. 1. After authentication, the system create a user profile infusion service to transport multiple user profiles from mobile device to the center server automatically, implicitly and securely. 5.2

Thin Client

Aggregation of services and content is done on the server side at a Spontaneous Interaction Server at each location. The information is presented in webpage format to the client browser through HTTP. Existing Wifi-enabled client devices can easily function in such smart environments, since they do not need special hardware or software to detect other devices or existing services.

Service 1

...

Aggregator 1

...

Service 2

Aggregator n

Core

Service n

Aggregator Service 1

Aggregator n

Service 1.1 Service 1.2

Service n.n

Fig. 4. The core is a thin middleware to provide a means to add services as modules. Services installed can be added to the core directly through built-in aggregators, or through aggregator services as sub-services. In the latter case, the sub-services need not be visible to the core layer.

5.3

Aggregation of Services and Contents

The system contains a core with the functionality to install services as modules (see Fig. 4). Whole ranges of services can be added when aggregators based on existing protocols (such as UPnP, Jini, and X10) are added to the core. Some

198

W. Qin et al.

services can also be aggregator services which collate sub-services. However, these subservices may not be visible directly to the core. In the case of the Media Service, it is an aggregator service that also aggregates media content from various media sources, while being a service itself in the home smart space. 5.4

Semantic Matching Combining with User Profiles and Situation Context

Using the aggregation of situation contexts and user profiles, rating or scoring process can be done to generate the Vector description set. The system can calculate the semantic similarities of all kinds of services existed in the smart environment and provide appropriate numbers of personalized service to end users according to the capabilities of mobile device, such as screen display size, media support type, and so on. 5.5

Middleware Support

Building over the Semantic Space [8] and CAMPS [11], the middleware consist of several individual, collaborating module such as ContextWrapper, ContextAggregator, InferenceEngine, KnowledgeBase, ServiceProvision and QueryFilter as shown in Fig. 5.

Fig. 5. Middleware support for service provision

5.6

Initial Prototype Implementation

We have implemented a prototype as Fig. 6(a) shown for the impromptu service discovery and access framework using the ProSyst OSGi service platform as the middleware core. All the assistive services are wrapped as OSGi bundles in the framework. In each logical smart space, we install the captive portal on a Linksys WRT54GL WLAN router. On entering the smart space, a mobile device (Samsung Q1 UMPC)


199

is detected with a WLAN card and connected automatically to the router. After launching the web browser in the UMPC, it is automatically directed to the service portal showing the offered services. For aggregation of devices, we implemented a UPnP wrapper service built over Kono’s CyberLink for Java UPnP library, providing web access capabilities. Links generated in the service portal point to the presentation URLs of the UPnP services. For the shopping scenario in a Coat shop, a snapshot of assistive services including a Coat finder and a Coat recommender is shown in Fig. 6(b). By choosing the Coat finder service, the user can locate the coat collections in a certain shelf (Fig. 6(c)).

Fig. 6. Initial prototype implementation: (a) Impromptu service discovery and provision platform; (b) User interface for Coat recommender; (c) A in-store service: Coat finder

6

Conclusion and Future Work

The paper suggests the combination of user profiles and situation contexts to provide a more pervasive service experience in smart assistive environment with mobile device. Leveraging a captive and service portal mechanism, we have allowed mobile clients with a web browser and wireless connectivity to spontaneously discover and access services. With the semantics similarity combination of user profiles and situation context, the scope of assistive services has been reduced efficiently, personalized to user’s interaction intention and adapted to the capabilities of user’s mobile client. In the future work, we will evaluate service provision based semantic similarity measure.

References 1. Lee, C., Helal, S., Lee, W.: Universal interactions with smart spaces. IEEE Pervasive Computing Magazine 5(1), 16–21 (2006) 2. Nakajima, T., Satoh, I.: A software infrastructure for supporting spontaneous and personalized integration in home computing environments. Personal and Ubiquitous Computing, 379–391 (2006) 3. Rukzio, E., Leichtenstern, K., Callaghan, V., Holleis, P., Schmidt, A., Chin, J.: An experimental comparison of physical mobile interaction techniques: Touching, pointing and scanning. In: Proc. UbiComp, pp. 87–104 (2006)

200

W. Qin et al.

4. Universal Plug and Play (UPnP) Forum, UPnP Device Architecture, Version 1.0 (December 2003), http://www.upnp.org 5. Jini.org, http://www.jini.org 6. Yu, S., Al-Jadir, L., Spaccapietra, S.: Matching user’s semantics with data semantics in location-based services. In: Workshop on Semantics in Mobile Environments (SME 2005) (2005) 7. Open Service Gateway Initiative (OSGi), http://www.osgi.org 8. Gu, T., Pung, H.K., Zhang, D.Q.: A service-oriented middleware for building context-aware services. Journal of Network and Computer Applications 28(1), 1–18 (2005) 9. Lin, Z.H., Fu, L.C.: Multi-user preference model and service provision in a smart home environment. In: Proc. CASE, pp. 759–764 (2007) 10. Strang, T., Linnhoff-Popien, C.: A context modeling survey. In: Proc. UbiComp workshop on Advanced Context Modeling, Reasoning and Management (2004) 11. Qin, W.J., Suo, Y., Shi, Y.C.: Camps: A middleware for providing context-aware services for smart space. In: Proc. GPC, pp. 644–653 (2006) 12. DAML-Time, http://www.cs.rochester.edu/∼ ferguson/daml/ 13. OWL-S: Semantic Markup for Web Services, http://www.daml.org/services/owl-s/1.1/ 14. Carmagnola, F.: The five ws in user model interoperability. In: Proc. IUI 2008 Workshop on Ubiquitous User Modeling (2008) 15. Salton, G., Wong, A., Yang, C.: A vector space model for automatic indexing. Communications of the ACM 18(11), 613–620 (1975)

Ubiquitous Phone System Shan-Yi Tsai, Chiung-Ying Wang, and Ren-Hung Hwang Dept. of Computer Science & Information Engineering, National Chung-Cheng University, 621, Chiayi, Taiwan [email protected], {wjy, rhhwang}@cs.ccu.edu.tw

Abstract. Portable smart devices, such as Pocket PC Phones and Smartphones, have become common devices in our daily life. However, they still provide technology-oriented human interface. In future ubiquitous environment, context-awareness is the key to make the human interface of smart devices become human-centric. In this paper, we design and develop a ubiquitous phone system, referred to as UbiPhone, which provides context-aware, human-centric phone service based on rich context acquired from various sensing sources. The most unique feature of UbiPhone is that a user only needs one click, UbiPhone will automatically choose the right communication channel and device to connect to the callee. UbiPhone also provides intelligent feedback service when the callee is not available, such as when to call back according to callee’s calendar, whether to automatically dial back when the callee becomes available, who to contact with if this is an emergency call and the callee is not reachable. In particular, social network is adopted to reason who would be the most appropriate person to help the caller find the callee immediately. Prototype of UbiPhone has been developed to show its feasibility. Keywords: Ubiquitous computing, smart world, context-awareness, Social network, Intelligent phone system.

1 Introduction As rapid advancement of technology, a revolutionary view of our future life articulated by Mark Weiser [1] is bit by bit coming true. Most of digital equipments, such as smart phone, are equipped with the capabilities of computing and communication in the age of information revolution. Through these advanced technologies, a ubiquitous environment full of human-centered services can be fulfilled soon. Contextawareness is the key feature for human-centered services as it enables services adaptive to human needs. By embedding context aware intelligence into devices of our daily life, many tasks could become simplified and efficient. For example, personal mobile phone has become one the necessary devices for our daily life as many of us are using it for communication. Therefore, making the mobile phone context aware could take the initiative to the ubiquitous small world. Much attention has been paid to integrate context and phone system in recent years, e.g., ContextPhone[2], iCAM system [3], Live Contacts project[4], Enhanced F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 201–215, 2008. © Springer-Verlag Berlin Heidelberg 2008

202

S.-Y. Tsai, C.-Y. Wang, and R.-H. Hwang

Telephony [5], Live Addressbook [6], Context Phonebook [7], and Awarenex [8]. ContextPhone[2] is a context aware smart phone project. The components include ContextLogger for recording user’s location movement, ContextContacts for exchanging context with each other and ContextMedia for sharing users’ media. iCAM system [3] displays location information of the callee on the contact list when the caller is making the call. According to the callee’s location information, iCAM also provides the caller an ordered list of phone numbers of the callee, such as house phone, office phone and cellular phone. Similarly, Live Contacts project[4] integrates various contexts to provide a list of communication ways, such as office phone, cellular phone, SMS, IM and e-mail for the caller. Then the caller can choose his favorite way to communicate with the callee from the list. In the aforementioned projects, although various contexts have been used to help the caller to make a phone call, most of them are not able to provide an automatic call to the callee through the most appropriate communication tool. Furthermore, social context, an important context for intelligent phone service, has not been adopted and analyzed. Social context has been applied to several applications in recent years. In a social network [9, 10], any two individuals (persons) in the network may be connected through a link set of intermediate acquaintances. In [11], the social context was used to find the most appropriate participant to talk with during a conference. The social relationship was discovered by analyzing the contents of research papers. In [12], Alex Pentland proposed a perspective on social dynamic which is based on the signals and behavior of interactions among people. He concluded that the measurements of social interactions can form powerful predictors of behavioral outcome in several important social dynamics, such as to get a raise, to get a job or to get a date. VibeFones [13, 14] gained sophisticated understanding about people’s social lives by acquiring those social signals and phone interactions. Observations concluded that user’s social rank can be discovered by analyzing his ordinary social interactions. In this paper, we propose a ubiquitous phone service, referred to as UbiPhone. Our goal is to provide a human-centric context-aware phone service which makes a phone call as easy as a single click on the callee from the contact list. UbiPhone automatically connects to the callee using the most appropriate phone system based on current context information, such as caller and callee’s location, presence status, network status, available phone systems, calendars, social relationship, etc. In case when the callee is not reachable, UbiPhone will prompt the caller when the callee will become reachable according to callee’s calendar. If it is an emergency call, UbiPhone will provide the caller the most possible person that will be nearby the callee or know how to reach the callee immediately based on the social network model we developed. Moreover, we also integrate IM and SMS (Short Message Service) with UbiPhone to provide short message service when a callee is unable to receive a voice-based communication at that time. Obviously, context management is very important to the implementation of UbiPhone. In UbiPhone, a framework for context management, includes context acquisition, context representation, context retrieval, and context inference, is proposed based on state-of-art technologies. It adopts ontology as the underlying context model. RDF (Resource Description Framework) [15, 16] and OWL (Web Ontology Language) [17] are adopted to implement the ontology model. SPARQL [18] is adopted for query and reason context from the ontology.

Ubiquitous Phone System

203

A prototype of UbiPhone has been implemented to demonstrate the feasibility of UbiPhone. PDA phones are used to carry out the human-centric phone service. In this paper, we also demonstrate that the use of social network model is helpful in finding a callee who is currently unreachable via phone. The rest of this paper is organized as follows. Section 2 presents the architecture and design idea of UbiPhone. A prototyping system implementation details are given in Section 3. Performance evaluation results are shown in Section 4. Finally, Section 5 follows with a concluding remark.

2 Design of UbiPhone In this section, we present the architecture of UbiPhone in section 2.1. Context and social models adopted in UbiPhone are described in section 2.2. The intelligent services provided by UbiPhone to achieve human-centric services are described in section 2.3. 2.1 Architecture of UbiPhone The architecture of UbiPhone is shown in Figure 1. UbiPhone Servers consist of service providing server and context management server. Service providing server provides the intelligent human-centric services by the help of inference engine. Context management server mainly acts as a semantic database server to manage raw data collected from the environment. UbiPhone Clients include users’ smart devices, sensors and other network equipments. Between servers and clients, we also designed a middleware as an interface to transmit all messages and signals. Web Services is adopted for the implementation of the middleware. More specifically, both contexts acquired from client’s device and messages to be sent to client from a server are encapsulated into XML-based SOAP messages. Java Web Service (JWS) framework is adopted to generate WSDL description files which are deployed on the server using Apache Tomcat Servlet Container.

Fig. 1. Architecture of UbiPhone

204


In Figure 1, the agents provide functions for service enquiry, context retrieval and statistics acquisition. When each agent receives certain events from sensing devices, it will provide the appropriate service. Call agent and messaging agent assist to connect to the right communication channel for users at that time. Status agent can automatically change users’ status according to user’s location, calendar and network status. Schedule agent, location agent and network status agent are designed to transmit user’s calendar, current location and available network system to context management server. Statistics agent derives statistics of phone call records and location logs of a user and updates collected statistics to the context management server. Figure 2 shows the flowchart of how context is acquired from various devices or updated to user devices. For example, the networking status (context) of the user device, such as whether it is connected to GSM network, is acquired by the network agent and sent to the context management server whenever there is a change. Figure 3 shows another example of how presence status is maintained. Context Information Acquire & Update User Request

Location Info. Update

Social Info. Update

Context Info. Response

Context Info. Update

Context Info. Push


RFID/GPS Info. Maintain


Updated Social Info. Social Info. Update.

Periodical Update Update Social Info.

Push RFID Reader Info. & GPS Info. For each User

Push GSM/IP Network/Bluetooth Status to Server. For each User

Privacy Rule

GSM/IP Network/Bluetooth Support & Status

Server

Privacy Rule


Server Push Context Info to Client Ex: New Calendar event

Client Push IM Info & New Calendar event to Server

Server Response Data with Privacy Rule. Ex: Joe’ s Calendar

Client Request Data. Ex: Joe’ s Calendar

Client

Statistics Agent.

Location Agent

Network Agent

Context Info. Maintain

Privacy Context Management Server

Network Status Update

Context Data

Client Agent

User

User Request


Update By SPARQL

Fig. 2. Flow chart of context acquisition and update

2.2 Models of UbiPhone We develop context digest model and social network model in UbiPhone to provide human-centric service. 2.2.1 Context Model Context digest model adopts ontology as the underlying technique to manage context. Context digest model acquires context from various agents and smart devices. These


205

Presence Status Maintenance Manual Input

Automatic Status Set By Context Info .

Get PS Contacts Status

Manual Input

Establish a Call Establish a Call

Calling

IM Status BusyΕ Idle etc.

Scheduled

GSM/ IP Network/ Bluetooh Support & Status Location Info.

Call Agent

Privacy Rule

Status Set

Conflict handle & Status Set

Privacy Rule

All Contacts Status

Intelligent Services

Fig. 3. Flow chart of context presence status maintenance

context data can be classified into three types: static, dynamic and statistic. Static context, such as users’ profile and preference, device’s profile seldom changes. Dynamic context, such as user’s presence status and location, updates to context management server frequently and responds to smart device immediately. Statistic context, such as phone call logs, location logs and event logs, is derived from statistics of historic logs. 2.2.2 Social Network Model In UbiPhone, a social network is modeled based on the user’s phone call records. The social network consists of a set of relations between two individuals. The “weight”, ω, of the social relation between any two users is calculated based on Equation (1). ωu ,b = i

Ru ,bi + Rbi ,u

max {Ru , x + R x ,u }

(1)

∀x

where Ra,b is the number of phone calls generated from a to b. The social network model is used by the inference module of service providing server to provide personalized service. Moreover, in order to implement the emergent-contact service, we define Prelation(u,b,t), the possibility to reach u through b at time t, based on social relation as shown in Equation (2).

206


Prelation (u , b, t ) = (1 − βω ) ∗ Pu (b | t ) + βω ∗ ωu ,c

(2)

where βω is the weight of social network model in calculating this possibility; Pu(b|t) is the preference of reaching the user u through contact buddy b at time t. Pu(b|t) is pre-configured by u. βω should be set to 1 if the user does not configure his emergentcontact preference Pu(b|t). 2.3 Intelligent Service of UbiPhone Among many intelligence services of UbiPhone, three most interesting intelligent services are ubiquitous call, anycall and emergent-contact service. They are described in detail in the following. All of these services are implemented based rule-based reasoning engine of the service providing server. 2.3.1 Ubiquitous Call Service Ubiquitous call service is aimed to provide a novel human-centric phone call service. When a user desires to call a buddy from his contact list, he just clicks on the buddy’s name. The call agent on the user’s PDA phone will communicate with the service providing server and provide the most appropriate service to the user, either a VoIP call, a GSM call, or a short message will be suggested if the callee is not available to answer the phone call. Figure 4 shows the flowchart of ubiquitous call. When the user initiates the ubiquitous phone call service, the call agent sends the request to the service providing server. The service providing server first checks the presence status of the callee. If the callee is not busy and can be reached via voice communication tools, the service providing server then checks if VoIP is available for both caller and callee. If yes, then it will instruct the call agent to establish a VoIP call. Otherwise, the callee’s phone number is retrieved based on his location information, e.g., office phone number if he is in office or GSM phone number if he is outdoors, the service providing server then instructs the call agent to make a GSM call to the selected phone number. In case the callee is busy, e.g., he in a meeting or class, appropriate text communication tool is suggested to the user through the message agent. The call agent also prompts callee’s expected available time and his presence status to the callee in the latter case. 2.3.2 AnyCall Service AnyCall is similar to the anycast service in IP networks; it provides the user to call the most appropriate buddy within a group. Consider an emergent scenario where the user needs to call anyone in his family group to inform an accident. In such an emergent situation, any buddy from the family group will be able to help the caller. Thus, considering as a human-centric service, AnyCall provides the user an interface to just a single click on the group name, it will connect the phone call to one of the group members who is available at that time to answer the phone. Figure 5 shows the flowchart of AnyCall. In this case, presence status is used to form a list of candidate callees whose status are free now. The location context is then


Fig. 4. Flowchart of Ubiquitous Call

207

Fig. 5. Flowchart of AnyCall

used to decide who is the nearest buddy geographically. If more than one buddy has the same shortest distance, then Ubiphone selects the buddy with the cheapest phone rate to call from the caller. (In Taiwan, phone rate is different due to more than one cellular phone providers.) Finally, after the callee is determined, a procedure similar to ubiquitous call is adopted to find out the most appropriate communication way (or phone number) to call the callee. 2.3.3 Emergent-Contact Service The emergent-contact service helps the caller contact with a third party from callee’s contact list or a phone at a location when the callee is not reachable. Currently social, calendar and location information are used to reason the most appropriate third party in Ubiphone. Specifically, the function Prelation(u,b,t), defined in equation (2), gives us the possibility to reach the callee based on social context. In equation (3), we define another function, Plocation(u,b,t), which gives us the possibility to reach the callee based on the historic location context. Specifically, the presence location of u and b of the last W weeks are recorded in the context management server. The possibility that u and b are at the same location (defined as the same building) is given by the number of times that they are in the same building at time t (the same day of week and the same time of the day) during the last W weeks over the number of weeks (W):

Plocation (u , b, t ) =

| location u (t ) = location b (t ) | W

(3)

The possibility to reach u through b at time t which considers both social and location context is then defined by the following Equation:

208


Pcontact ( u, b, t ) = α location ∗ Plocation (u , b, t ) + α relation ∗ Prelation ( u, b, t )

where α location + α relation =1

(4)

Note that equations (1)~(4) are defined as possibility not probability as they are indicators of possibility, not necessary follow the law or semantics of probability.

Fig. 6. Flowchart of Emergent-contact Service

Figure 6 shows the flowchart of emergent-contact service. After the call agent sends the request to the service providing server. The service providing server first retrieves the list, b, of buddies of u who are free now from the context management server. It then retrieves u’s schedule (calendar) at this time. If the schedule exists and the location of u could be retrieved from the schedule context, then find a buddy who is at that location or the phone number of that location to call. Otherwise, u’s current location will be predicted based on his location logs in the past. Equations (1)~(4) will be used to compute the possibility of each buddy in the contact list and the one that has the highest possibility will be suggested to call.


209

3 Prototyping and Implementation In this section, we demonstrate the feasibility of the UbiPhone via prototyping a real system. Our demonstration scenario is made up with four users using Dopod PDA phones equipped with WiFi, GSM, GPS, and RFID. Users joined one or more research groups. An example of contexts of four users is shown in Table 1. Users configured their contact list at will, as shown in Figure 7. Table 1. Users’ context Context Name Status Location Group

Hushpuppy

Lunsrot

Zqq

busy library Web 2.0 group

free EA315 Web 2.0 group

busy EA105 Ubiquitous Learning group

The scenario is demonstrated as follows: Lunsrot read an interesting paper and wished to discuss his novel idea with any buddy in the e-Learning project. He invoked the AnyCall service (implemented as “I’m Feeling Lucky” function on the PDA phone) by a click on the “e-Learning 2.0” and selecting the “I’m Feeling Lucky” function. Since only Septhiorth was free at that time and he was on the Internet, so the UbiPhone connected Lunsrot to Septhiorth via a VoIP connection, as shown in Figure 8. After talking to Septhiorth, Lunsrot continued his survey on Web 2.0 literature. Later on, he read an interesting topic and decided to share with Hushpuppy. So, he invoked the Ubiquitous Call service to Hushpuppy. Since Hushpuppy was busy in library, UbiPhone blocked the voice channel and suggested Lunsrot to use messaging service, as shown in Figure 9. Now, consider an emergent scenario where Septhiorth needs to inform an important message to zqq immediately. Assume that zqq is not reachable because he is busy and his mobile phone power worn out at that time. So Septhiorth invokes the

Fig. 7. The Screen of Four Users’ Smart Devices

Fig. 8. The Demonstration of “AnyCall”

210


Emergency-contact service to find out who can help him to reach zqq. Based on the flow chart of Figure 6, Emergency-contact service suggests Septhiorth to call Lunsrot, as shown in Figure 10. Since Lunsrot and zqq is in the same building, so Septhiorth is able to reach zqq through Lunsrot.

Fig. 9. The Demonstration of “Ubiquitous Call”

Fig. 10. The Demonstration of “Emergency-contact Service”

4 Performance Evaluations In this section, we evaluate the performance of emergent-contact service. Consider the scenario where the user masa has been chosen as the callee for this evaluation. Table 2 shows masa’s contact list which includes buddy’s ID, name, and group. Table 3 shows the default preference, Pu(g|t), of each group configured by masa. The experiment is carried out for five weeks. During the first four weeks, masa’s location logs and phone call logs are collected and stored in the context management server. Statistics of the location logs are then derived for each hour based on a weekly cycle. For example, the location logs may show masa was at (EA315, D4304, EA315, EA315) from 8:00 AM to 9:00 AM on Monday during the first four weeks. The possibility that masa will be at room EA315 is then estimated to be 0.75 for the fifth week. On the other hand, masa’s phone call logs are used to build a social network model where a node corresponds to a buddy and the weight of a link between two nodes is computed based on Equation (1). Based on the social network model and statistics aforementioned, the experiment on the fifth week is conducted as follows. For each hour during the fifth week, select a random time and assume masa turned off his PDA phone at that time. Randomly select a buddy to invoke the emergent-contact service and obtain the service from the service providing server. The weighting parameters are setting as follows: αlocation = 0.65, αrelation = 0.35, βω = 0.3. We then evaluate whether the suggested emergentcontact buddy is nearby masa.


211

Table 2. The Experimenters List No. b1 b2 b3 b4 b5 b6 b7 b8 b9 b10 b11 b12 b13 b14 b15 b16 b17 b18 b19 b20 b21 b22

Nickname masa yuwen puzi calais iamshyang bluejam zqq sigwa des33 hopper septhiorth hushpuppy Lunsrot yanmin Nick rhhwang Mei Brian Dad yuan hsg sue

Relation (Group) Myself Graduate Mate Graduate Mate Graduate Mate Graduate Mate Graduate Mate/University Mate/Dorm Mate Graduate Mate/University Mate University Mate University Mate University Mate Younger Graduate Mate Younger Graduate Mate Younger Graduate Mate Younger Graduate Mate Graduate Mate for Doctor's degree Tutorial Professor Family Family Family Friend Friend Friend

Table 3. The Default Preference of Group Defined by Masa Group Name Myself Family Dorm Mate Friend University Mate Graduate Mate Younger Graduate Mate Graduate Mate for Doctor's degree Advisor

Preference 0 1 2 3 4 5 6 7 8

Pu(g|t) 0 1 1/2 1/3 1/4 1/5 1/6 1/7 1/8

Table 4 shows the statistics of location logs for each hour on a weekly cycle. Table 5 shows masa’s location logs of the fifth week. Besides, social network based on masa’s phone call logs of the first four weeks are derived, as shown in Figure 11. Here, the service is considered successful if the emergent-contact buddy is actually nearby masa (in the same building or outdoor location) and can get masa on the phone right away. Table 6 shows the results of our experiment where “s” denotes success and “f” denotes fail. As we can observe that the service is a success for most of the time. In average, the successful rate is 0.7738.

212

S.-Y. Tsai, C.-Y. Wang, and R.-H. Hwang Table 4. The Statistics of Masa’s Location Log During the First Four Weeks

00:00~01:00 01:00~02:00 02:00~03:00 03:00~04:00 04:00~05:00 05:00~06:00 06:00~07:00 07:00~08:00

Mon D4304/ EA315 D4304/ EA315 D4304/ EA315 D4304/ EA315 D4304/ EA315 D4304/ EA315 D4304/ EA315 D4304/ EA315

Tue D4304

Wed D4304/ EA315

D4304

D4304

D4304

D4304

D4304

D4304

D4304

D4304

D4304

D4304

D4304

D4304

D4304

D4304

D4304

D4304

09:00~10:00

D4304

D4304/ EA315

D4304/other s D4304/ others

10:00~11:00

D4304

D4304/ EA501/ EA315

D4304/ EA315/ others

11:00~12:00

D4304

D4304/ EA501/ EA315


12:00~13:00

Meal/ D4304

Meal/ D4304/ EA315

Meal/ D4304/ EA315

13:00~14:00

D4304/ others

D4304/ EA315


14:00~15:00

D4304/ EA315

D4304/ EA315

15:00~16:00

D4304/ EA315

EA315

16:00~17:00

D4304/ S.C.

EA315/ EA105

17:00~18:00

D4304/ S.C.

EA315/ EA105

18:00~19:00

Meal/ D4304

Meal/ EA105

19:00~20:00

D4304

Meal/ EA315

20:00~21:00

D4304

EA315

21:00~22:00

D4304

08:00~09:00

22:00~23:00 23:00~24:00

D4304 D4304

EA315 D4304/ EA315 D4304/ EA315

D4304/ EA315/ others D4304/ EA315/ others D4304/ EA315/ C.Y. D4304/ EA315/ C.Y./S.C D4304/ EA315/ C.Y./S.C D4304/ EA315/ C.Y. D4304/EA3 15/C.Y. D4304/EA3 15 D4304/EA3 15 D4304/ EA315

Thu D4304/ EA315

Fri

Sat D4304/ EA315/K.H. D4304/ D4304 D4304 EA315/K.H. D4304/ D4304 D4304 EA315/K.H. D4304/ D4304 D4304 EA315/K.H. D4304/ D4304 D4304 EA315/K.H. D4304/ D4304 D4304 EA315/K.H. D4304/ D4304 D4304 EA315/K.H. D4304/ D4304 D4304 EA315/K.H. D4304/other D4304/ D4304 s EA315/K.H. D4304/ D4304/ D4304/ others others EA315/K.H. D4304/ D4304/ EA315/ EA315/K.H. EA315/ others / others others D4304/ D4304/ EA315/K.H. EA315/ EA315/ others / others others D4304/ Meal/ EA315/K.H. D4304/ Meal/ / EA315/H.W EA315 others .Room D4304/ D4304/ EA315/K.H. EA315/ EA315/H.W / others .Room others Meal/ D4304/ EA315/ D4304/ EA315/ others S.C./K.H EA315 D4304/ D4304/ EA315/other EA315/ EA315/ s others S.C./K.H D4304

Sun D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H.

D4304/ S.C.

EA315/Driv ing/others

D4304/ EA315/K.H.

D4304/ S.C.

EA315/Driv ing/others

D4304/ EA315/K.H.

Meal/ D4304

EA315/ C.Y./K.H.

D4304/ EA315/K.H.

D4304

EA315/ C.Y./

D4304/ EA315/K.H.

D4304/ EA315/Driv ing D4304/ EA315/Driv ing D4304/ EA315

K.H. D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H.

D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H. D4304/ EA315/K.H.

D4304/ EA315 D4304/ EA315 D4304/ EA315 D4304/ EA315

D4304 D4304 D4304 D4304

D4304/ EA315/K.H.


213

Table 5. The Statistics of Masa’s Location Log During the Fifth Week

00:00~01:00 01:00~02:00 02:00~03:00 03:00~04:00 04:00~05:00 05:00~06:00 06:00~07:00 07:00~08:00 08:00~09:00 09:00~10:00 10:00~11:00 11:00~12:00

Mon

Tue

Wed

D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304

D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 EA501 EA501

EA315 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304

Thu

Fri

D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 CCU campus DonateBlood EA315

12:00~13:00

D4304

EA315

EA315

Driving Range

13:00~14:00

D4304

EA315

EA315

Driving Range

14:00~15:00

D4304

EA315

EA315

D4304

15:00~16:00 16:00~17:00 17:00~18:00

D4304 Squash Court Squash Court

EA315 EA105 EA105 Great NightFair

EA315 EA315 EA315 Great NightFair

D4304 D4304 D4304

18:00~19:00

D4304

19:00~20:00

D4304

EA315

EA315

D4304

20:00~21:00

D4304

EA315

EA315

D4304

21:00~22:00 22:00~23:00 23:00~24:00

D4304 D4304 D4304

D4304 D4304 D4304

EA315 D4304 D4304

D4304 D4304 D4304

D4304

Sat

D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 Ming Hsiung ChengClinic Cheng Clinic EA315 EA315 EA315 Orange Grange Orange Grange Orange Grange D4304 D4304 D4304

Sun

D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304

D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 D4304 EA501 EA501

D4304

EA315

D4304

EA315

D4304

EA315

D4304 Squash Court Squash Court

EA315 EA105 EA105

D4304

Great NightFair

D4304

EA315

D4304

EA315

D4304 D4304 D4304

D4304 D4304 D4304

Table 6. The Experimental Results

00:00~01:00 01:00~02:00 02:00~03:00 03:00~04:00 04:00~05:00 05:00~06:00 06:00~07:00 07:00~08:00 08:00~09:00 09:00~10:00 10:00~11:00 11:00~12:00 12:00~13:00 13:00~14:00 14:00~15:00 15:00~16:00 16:00~17:00 17:00~18:00 18:00~19:00 19:00~20:00 20:00~21:00 21:00~22:00 22:00~23:00 23:00~24:00

Mon

Tue

Wed

Thu

Fri

Sat

Sun

f f f f f s s s s s s s s s s s s s s s s s s s

s s s s s s s s s f s s s s s s s s s s s s s s

s s s s s s s s s f f f s s s s s s s s s s s f

f s s s s s s s s f f s f f f f f f f s s s s s

s s s s s s s s s s f f f f f s s s s f f f s s

f s s s s s s s s s f f f f s s s s f s s s s s

s f f s s s s s s s s s s s s s s s f f s s s s

214


Fig. 11. Social Network

5 Conclusion In this paper, we have proposed a ubiquitous phone service, UbiPhone, which emphasizes on providing human-centric context-aware phone service. In particular, three human-centric services have been shown in this paepr to demonstrate a phone call to a buddy or a group could be done as simple as a single click on the contact list. We have also shown how the social context and location context could be used to reach a buddy in emergency using the emergent-contact service. UbiPhone also adopts a general platform to manipulate context. Specially, ontology is used for context modeling and Web Services is used for client-server communication. Software agents have been used as the middleware to send and receive context between client and server such that applications could provide context-aware services in a unified form. Our experimental results show that emergent-contact service could provide the caller the buddy nearby the callee at most of the times. The prototype of UbiPhone also demonstrates the feasibility of the proposed humancentric concepts. Several issues of the proposed UbiPhone require further study. For example, vertical handoff between heterogeneous networks is necessary to make a phone call uninterrupted when the user moves around.

Acknowledgment The research is supported by the NSC 96-2219-E-194-005 and NSC 96-2219-E-194006, National Science Council, ROC.


215

References 1. Weiser, M.: The Computer for the Twenty-First Century. Scientific American, 94–104 (September 1991); reprinted in IEEE Pervasive Computing, 19–25 (2002) 2. Raento, M., Oulasvirta, A., Petit, R., Toivonen, H.: ContextPhone: A Prototyping Platform for Context-Aware Mobile Applications. IEEE Pervasive Computing 4(2), 51–59 (2005) 3. Nakanishi, Y., Takahashi, K., Tsuji, T., Hakozaki, K.: iCAMS: A Mobile Communication Tool Using Location and Schedule Information. IEEE Pervasive Computing 3(1), 82–88 (2004) 4. Henri ter Hofte, G., Otte, R.A.A., Kruse, H.C.J., Snijders, M.: Context-aware Communication with Live Contacts. In: Proceedings of the Conference on Computer-Supported. Cooperative Work, Chicago, Ill, USA (November 2004) 5. Cadiz, J.J., Narin, A., Jancke, G., Gupta, A., Boyle, M.: Exploring PC-telephone Convergence with the Enhanced Telephony Prototype. In: Proceedings of the International Conference for Human-computer Interaction (CHI 2004), New York, NY, USA, pp. 215–222 (2004) 6. Milewski, A.E., Smith, T.M.: Providing Presence Cues to Telephone Users. In: Proceedings of the Conference on Computer-Supported. Cooperative Work, New York, NY, USA, pp. 89–96 (2001) 7. Schmidt, A., Stuhr, T., Hans, G.: Context-Phonebook - Extending Mobile Phone Applications with Context. In: Proceedings of Mobile HCI 2001, Lille, France (September 2001) 8. Tang, J.C., Yankelovich, N., Begole, J., Van Kleek, M., Li, F., Bhalodia, J.: ConNexus to awarenex: extending awareness to mobile users. In: Proceedings of the International Conference for Human-computer Interaction (CHI 2001), New York, NY, USA, pp. 221–228 (2001) 9. Radcliffe-Brown, A.R.: On Social Structure. Journal of the Royal Anthropological Institute 70, 1–12 (1940) 10. Milgram, S.: The Small World Problem. Psychology Today, 60–67 (May 1967) 11. Matsuo, Y., Tomobe, H., Hasida, K., Ishizuka, M.: Mining Social Network of Conference Participants from the Web. In: Proceedings of IEEE Web Intelligence, pp. 190–193 (2003) 12. Pentland, A.S.: Social Dynamics: Signals and Behavior. In: IEEE International Conference on Developmental Learning, San Diego, CA (October 2004) 13. Pentland, A.S.: Socially aware computation and communication. IEEE Computer Society 38(3), 33–40 (2005) 14. Madan, A., Pentland, A.S.: VibeFones: Socially Aware Mobile Phones. In: Proceedings of the 10th IEEE International Symposium on Wearable Computers, pp. 109–112 (October 2006) 15. RDF Primer. W3C document, http://www.w3.org/TR/rdf-primer/ 16. RDF Schema. W3C document, http://www.w3.org/TR/rdf-schema/ 17. OWL. W3C document, http://www.w3.org/2004/OWL/ 18. SPARQL. W3C document, http://www.w3.org/TR/rdf-sparql-query

Utilizing RFIDs for Location Aware Computing Benjamin Becker1, Manuel Huber2 , and Gudrun Klinker2 1 EADS InnovationWorks, Germany Advanced Industrial Design and Visualization 81663 M¨ unchen [email protected] 2 Technische Universit¨ at M¨ unchen, Germany Institut f¨ ur Informatik Boltzmannstraße 3, 85748 Garching b. M¨ unchen [huberma, klinker]@in.tum.de

Abstract. This paper focuses on the possibilities of utilizing infrastructural RFIDs for location aware computing. A key to automated computer aided work is location and situation awareness. This allows the system to operate independently from direct user input, minimizing the impact on the original process. For demonstration purposes a large area tracking system using existing RFID technology and infrastructure is developed and evaluated. As a first possible application we embed the system into an aircraft cabin. The localization procedure is based on the coordinates of the RFID transponders detected by the RFID reader, as well as the orientation information measured by a gyroscope. Position estimation is accomplished through an iterative algorithm, optimizing the position of the antenna’s readability field with respect to the currently visible tags. The average error is below half a meter. Possible use cases range from maintenance assistance to way finding in commercial shops or underground parking lots.

1

Introduction

In order to automatically adapt to the ever-changing contexts in which a user may perform, situation and especially location awareness emerged as a key concept in ubiquitous computing. A system that is aware of the user’s location is enabled to react to the current demands without explicit input and may assist in various novel means. Unfortunately location information is not always easily accessible. Often tracking either relies on public infrastructure which may not be present everywhere, especially indoors, or on specialized infrastructure which usually has to be deployed at a high cost per area. Furthermore the installation of specialized hardware is likely to be unfeasible in certain environments because it usually is obtrusive and distracting. The consequence of these constraints is the idea to use whatever technology is present as infrastructure for position estimation. F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 216–228, 2008. c Springer-Verlag Berlin Heidelberg 2008

Utilizing RFIDs for Location Aware Computing

217

Fig. 1. Computer Assisted Maintenance Worker

In an industrial scenario it is possible to introduce new technology in a controlled manner and thus it is more accessible in this manner as more diverse technologies are likely to be present. Especially RFID technology, which already has seen some attention in ubiquitous computing, is at the moment predominately used by supply chain management and life-cycle management applications. Therefore RFID technology is expected to become more and more ubiquitous in the near future and thus may be part of our common environments, even possibly enabling its use in domestic environments. We focus on the industrial setup of aircraft cabin maintenance (Figure 1) where we can assume already present RFID tags distributed in the cabin for quality assurance as part of our infrastructure. We will show that using these tags it is possible to perform inside-out tracking and pose estimation. We demonstrate this on a first prototype of an automated maintenance support system. Location awareness herein reduces the need for direct user input since the system is able to detect the current state of the worker. Computer aided maintenance and automated worker support for maintenance tasks in general have already been studied in the past (see for example [1] or [2]) and their benefit has been established. This allows us to mainly focus on the characteristics of the tracking concept including scalability, low impact integration and precision.

218

2

B. Becker, M. Huber, and G. Klinker

Exposition

The following paragraphs will describe the prototype’s overall hardware setup as well as the currently used position estimation algorithm. This is still a prototype and will require further adaption for integration into concrete future scenarios. 2.1

Technical Background

In contrast to existing infrastructural large-area outdoor trackers like GPS and GSM, there is a general lack of scalable and robust large-area trackers working indoors. This paper introduces the possibilities of using RFID technology for localization and tracking in large, dense indoor areas. The prototype illustrates the concept on the example of an aircraft cabin testbed, where RFID tags have been deployed independently for the setup and evaluation of various use cases. As already stated, logistics, quality assurance and life-cycle management are major domains of RFID technology. To match these requirements, tags vary in levels of complexity from simple ID transmitters to smart cards. Passively powered tags use the energy transmitted via the RF field for the communication response whereas active tags depend an additional power source. Currently there are two common physical communication layers [3]: near field (LF at 134.2kHz, HF at 13.56MHz and 27.125MHz ) and far field (UHF at 433MHz or 868MHz (EU) and 915MHz (US), MW 2.45GHz or 5.8GHz ) which especially differ in their operating distance which ranges up to few centimeters in the first case and typically up to 6m in the second case. Active powered tags reach further and often implement additional features like complex encryption - but suffer from a short lifespan and additional maintenance issues due to their battery. Summing up, active tags allow more complex applications at a higher investment per tag. Usually they are used to identify only few but special objects for a short period of time, whereas passively powered tags are often used as a long-term identification of structural parts. For our aircraft cabin scenario we focus on passively powered tags which are being used for long-term quality assurance of structural parts. These support our assumption that RFIDs are massively distributed as part of the infrastructure. Recent use cases show the need for long range readability beyond 1m, from which our proposed tracking method profits much more than it would from short range RFIDs, as these would severely limit the operating volume. 2.2

Related Work

Tracking methodologies are generally classified into outside-in and inside-out tracking. For outside-in tracking a global set of observers determines the positions of all participating entities, where as in inside-out tracking each participant determines its pose independently on its own using landmarks in the environment. Similarly existing approaches for RFID based tracking can be classified into these categories. The inside-out RFID tracking uses distributed tags and a single mobile reader for the position estimation of the reader whereas outsidein uses a certain number of fixed readers to track a set of mobile transponder.


219

Outside-in based systems generally suffer from the need to distribute numerous readers in the tracking area, impeding the possible scalability to large areas, due to complex communication needs and high investments. Although outside-in tracking does not match the general constraints of the aircraft use cases, it does use the same technological basis and therefore our system should be able to compete with it. Even commercial applications exist [4], using angle of arrival for the tracking of proprietary, active ultra-wide-band tags. Another system [5] demonstrates the possibility to enable tracking by distance grouping of transponders by varying the antenna’s power. Inside-out based tracking was already used to update a robot’s position estimation [6] by reading tags with known location. Another promising approach [7] is introducing the Super-Distributed RFID Tag Infrastructure. This scenario uses a small robot vehicle equipped with a very short-range RFID reader pointing downwards. The tags are distributed on the floor using their ID to reference a specific position. 2.3

Prototype Scenario

The prototype demonstrates the possibility to achieve 6DOF localization and tracking of a mobile device in an aircraft cabin using a long-range reader and a cloud of distributed passive tags, taking a gyroscope’s current orientation estimate into account. The goal of this application is to allow automated maintenance support based on location awareness. Maintenance of air crafts is time critical as every minute on ground means lost revenue to the airline. Clearly, the more promising approach for our case is an inside-out based tracking system, having no or little impact on the environment by bundling the required hard- and software on a single mobile device. The system’s design for a single reader and inexpensive passive RFID tags allows us to cover a large area with little investment. Besides, often the required infrastructure is already present from other RFID applications as listed in the prior paragraph. Also a mobile computing device can be easily extended with additional sensors like a gyroscope or WLAN tracking, increasing range, robustness and precision of the localization algorithm. Hardware Setup. Since RFIDs are only just being introduced in the aircraft production and maintenance process the tags in the prototype’s cabin mock-up were added supplementarily. This paragraph describes how the tracking system was embedded into the aircraft cabin and gives additional detail about the hardware in use. It is useful to have a large and reliable reading range to increase robustness as well as the position estimation quality. The RFID tags which had already been deployed in the aircraft cabin testbed are passive ultra-high-frequency transponders, which match these requirements (Figure 2, Rafsec G2 ShortDipole at 868Mhz ). The reader used was a Feig LRU2000 UHF Long Range Reader. Since neither this nor any other investigated reader made any tag quality metrics (such as received signal strength indicators) available, each scan only resulted in

220


Fig. 2. RFID Tags Distributed in Aircraft Cabin

binary “detected” or “not-detected” information per tag. The miniature longrange reading device (Figure 3) is equipped with a directional antenna. The antenna’s gain increases the reading range along a single axis, allowing this system to detect tags up to a distance of 6.0 meters. As illustrated in Figure 2 (right and lower left), the transponders in this scenario are distributed within the aircraft’s cabin, especially attached to the hatrack, the seating and the floor panels. In the final scenario in a real aircraft cabin usually every major structural part has its own identification tag, giving a far denser distribution. For measuring the current orientation, a gyroscope is directly connected to the antenna (Figure 4) with a precalibrated offset transformation. Basic Localization Concept. The localization procedure is based on two sensors: the RFID reader and the gyroscope. The reader’s output is a list of tags read by the antenna within the last 200 milliseconds. Note that these measurements only constitute the existence of the relevant tags in the reading range of the antenna and do not contain further distance information. The developed RFID tracking system has three basic characteristics: – A large number of RFID tags are attached to structural parts of the cabin. For each of them the exact location and ID are known and available to the system as part of the Digital MockUp (DMU) data.

Fig. 3. RFID Reader

Fig. 4. Xsens Gyroscope


221

– A gyroscope measures the current orientation. This is used internally to increase the accuracy of the position estimate and returned as part of the pose estimation. – The reading range of the RFID antenna is taken into consideration. It has to be determined beforehand. The localization is based on the coordinates of the currently detected transponders, which can be derived from the database. The antenna is estimated to be at a position from where it is able to read all currently detected tags best. This is accomplished through an iterative process which imposes requirements on how the detection range is modeled, as seen in the following section. In our case the position of each RFID tag and the corresponding ID was measured in a preprocessing step. Hopefully this can be automated in a future scenario by having predefined IDs and positions of tags on certain constructions or building parts. This assumption is realistic, since RFID technology is in the process of being integrated into the aircraft manufacturing process. The setup of the following algorithm is intended to strongly reduce computational cost of the pose estimation. This is required since mobile devices often lack computational power in favor for mobility and uptime. Antenna Readability Model. Before actually using the system, the detection range of the RFID reader’s antenna has to be determined. Due to missing details on the assembly of the antenna this has to be empirically estimated through experiments. To distinguish the readability field some tests were run. Since the system uses a directional antenna with a rotationally symmetric RF field, the dimension of the testbed was reduced to a 2D plane through the central axis of the antenna. Numerous tags were attached to polystyrene cubes and placed on a grid to evaluate the reading range and probability, resulting in a discrete readability field. For each point p relative to the antenna the position estimation algorithm requires a scaled displacement vector from the antenna model. This vector v(p) indicates in which direction the antenna needs to be moved to increase the chance of detecting a transponder at location p. A scalar d(p), defining the intensity of the displacement vector of a tag at this position is used to increase the influence of a wrongly positioned tag on the pose estimation. Both will be derived from the probability of a tag being read. Since realtime computation of the readability for all tags is computationally expensive, a gradient field was derived by interpolation (Figure 5) which serves as a lookup table. The resulting gradient field is used to scale the displacement, which is shown in Figure 6. Both fields use coordinates in the antenna’s reference frame. The antenna is placed at p = (0, 0) and the point with the maximum reading possibility is located at pmax = (0, 0.85m). Optimization of the Position Estimate. The reader scans the antenna field for visible tags during 100ms slices. Initially the position estimation assumes

222


Fig. 5. Antenna Readability

Fig. 6. Displacement

that the antenna is in the centroid of the visible tag positions. At each step a simple optimization procedure is performed. First of all the visible tags are rotated into the reference frame of the antenna by applying the inverse of the orientation measured by the gyroscope. This relative position is projected onto the 2D plane and used to access the readability as well as the displacement field. In the next step the median over all displacement vectors is computed and added to the current position resulting in a new position estimate. The estimated quality of this position is calculated as the sum over the readability of all detected tags, taken from the readability field. Iteratively this procedure optimizes the position estimate until a lower border has been reached. When finished the position as well as the likeliness with the highest quality rating is returned. In case no tags have been detected, the current position is kept untouched. 2.4

Prototype Analysis

This section focuses on the evaluation of the technical properties of the prototype and compares it with other existing setups. Error Analysis. The setup for the test is shown in Figure 7. For the estimation of the ground truth position an optical infrared tracker is used, allowing the reflective marker on the antenna to be localized within millimeter precision. Also the offset between the infrared reflection marker and antenna was calibrated beforehand. This assures that the entire tracking system used for validation and testing introduces a negligible error compared to the expected precision of the RFID tracking. We present here only a one minute excerpt from our test runs, which exemplifies the typical performance of the system. The mobile device was moved in the cabin with varying orientations. The currently visible tags and the resulting position estimation of the RFID tracker as well as the ground truth position


223

Fig. 7. Antenna with IR-Marker and A.R.T. Tracking System

Fig. 8. Antenna Field (Median Readability) and Cabin Coordinates

were timestamped and logged. In a post-processing step the recorded data was examined to determine the precision and to analyze the current setup. To differentiate the error induced by the distribution of the tags and the antenna model, the coordinate system’s axes are distinguished with a trailing ‘a’ for antenna or ‘c’ for cabin. In Figure 8, the cabin has the xc axis (red) pointing to the side, the yc axis (green) to the front and the zc axis (blue) to the ceiling. The antenna’s coordinate system has the xa axis pointing to the side, the ya along the antenna’s normal and the upward axis za . The overall performance of our approach can be extracted from Figure 9. It shows the error distributed along all three axes of the cabin as well as the average error across the entire testing time. The output is the raw data taken from the position estimator without any filtering or continuous position estimation for jitter reduction. The Euclidean distance between the RFID tracker’s estimate and the ground truth ranges up to 1.5 meters with an average of 0.48 meters. In the beginning of the test run, the antenna was moved while pointed towards the front of the cabin. This results in a small yc axis error since a more homogeneous distribution of the RFID tags (Figure 8, right) allows the antenna model to fit perfectly with its long reading range along its ya axis. Therefore the effect is homogeneous in both reference systems. In the time range between 10 and 40 seconds the antenna was additionally pointed upwards and downwards while still moving through the cabin, increasing the error along the z c axis. The uncertainty along the other two axes as well as

224


Eucl. sum x y z

2.5 error distance in meter

2 1.5 1 0.5 0 -0.5 -1 -1.5 0

10

20

30

40

50

60

time in seconds

Fig. 9. Error in the Test Run in the Cabin’s Coordinate System

the overall error are also increased, since the antenna’s reading field is mainly outside of the cabin’s volume where no tags were placed. Within the final time interval beginning at 40 seconds, the antenna’s position was kept still but was rotated mainly around its za axis and therefore often pointing along the side of the cabin (xc axis). This shifts the error between the yc and the xc axis. The distribution density of the tags towards the side of the cabin is also inappropriate. Not only the distribution and the homogeneity of the tags are important for a proper function of the algorithm, also the number of tags used for the position estimation is of interest. Looking at Figure 10, it is evident that the simple approach to use the centroid of the tag cloud is inferior to the optimization based on the antenna model. The overall median error is reduced from 0.71 to 0.48 meters. It is evident that the possible uncertainty introduced by the antenna’s long reading range in direction of the ya axis could be compensated Initial Centroid

Optimized Position Estimation 1000 x y z

error distance in meter

1.4

x y z

1.2 1

800

600

0.8 400

0.6 0.4

200

0.2 0

number of position estimations

1.6

0 0

1 2 3 4 5 6 7 number of tags for pos. estimate

0

1 2 3 4 5 6 7 number of tags for pos. estimate

Fig. 10. Error in the Antenna’s Reference Frame (x, y, z) and Histogram (gray bars) with Respect to the Number of Detected Tags


225

with the orientation based method. The errors on the xa and za axis could be improved only slightly. For the case of zero detected tags the error is high because without measurement information supplied no further estimation can be performed. In the case of a single tag the algorithm’s estimate induces an even larger error. This indicates that a tracking using a motion model, based on preceding estimates, could achieve an increased performance since a good initial estimate seems superior to the estimation based on a single tag. Single tag discoveries often occur when the reader is pointing directly into an obstacle, reducing the range of the RF field. Although the error should be indirectly proportional to the number of tags this can not be derived from our current test results. As the histogram in Figure 10 illustrates, only about 3% of the frames have more than 4 tags detected. Nevertheless it seems obvious that the error increases in a high detection case. A possible reason for this can be traced back to the reader buffering recently read tags when a quick rotation is performed, resulting in more detected tags but partially from an outdated orientation. Although neither the algorithm nor the implementation can be expected to be optimal, the results clarify four main error sources in the system setup: – – – –

inhomogeneous distribution of tags cause the position algorithm to fail low density of RFID tags yield an inferior pose estimation obstacles changing the antenna’s reading probability field increases the error accumulation of visible tags over a certain period of time introduces an error when quick rotations are performed

These characteristics need to be considered in future optimizations of the system. Error Evaluation and Comparison. Considering the simple algorithm and optimization model used, the achieved quality is encouraging. As the results clarify, the antenna model and the position estimation algorithm are not optimal. Nevertheless it is obvious that an algorithm using a representation of the antenna model in combination with the orientation information of a gyroscope can significantly increase the quality of an RFID tracker. For a comparison with existing RFID based tracking systems, the outside-in based approaches are analyzed first. Although they have a totally different target application and differ largely from our setup it is important to see what results are possible when using RFID technology at ideal conditions. Disregarding the already mentioned disadvantages, the results are only of slightly better quality regarding precision of the position estimate. The tracker Ubisense reaches only slightly better results with 95% of the measurements having an error below 30cm [4]. The passive outside-in system landmarc ([5]) achieves an median error value of about one meter. Both systems highly depend on the number of readers used in the tracking area. Inside-out based approaches, as used in the fields of robotics, can benefit from the advantages of continuous tracking with control inputs from the robot’s steering sensors. Besides this, the localization is reduced to a 2D position estimation.

226


The system presented in Super-Distributed RFID Tag Infrastructure [7] allows a high quality position estimation. When a tag is detected, the short reading range of the antenna provides a high accuracy localization, giving the robot the chance to correct its internal estimate. This still suffers from a major drawback: When carried by a worker and therefore without control input, continuous tracking fails since the short reading range implicates very seldom tag discoveries. 2.5

Application and Further Use Cases

For our maintenance application the tracking is precise enough to embed a location aware assistance software. It is possible to guide the worker to the correct frame of the aircraft and to priorize the tasks according to his current position, so that maintenance tasks in the current working area are favored and thus minimizes time spent between tasks. Also it is possible to reduce user input which would occupy the worker. The use of a tracking system allows extending the existing workflow of a mechanic. Nowadays it is common to have a digital task list on a mobile device to verify that all important maintenance tasks have been performed correctly. With the additional location sensitivity it is possible to guide the worker towards the next task and automatically give additional information regarding the current step. For further scenarios, it will be interesting to see if RFID technology is feasible for standalone tracking, as supporting tracker for initialization of a more precise but local tracking method (e.g. optical model-based tracking)[8] and for largearea tracking. Also it is promising to evaluate RFID technology as an additional source of information for ubiquitous computing or Augmented Reality applications regarding location awareness and object identification. Furthermore RFID based infrastructure can serve as an additional tracking modality in ubiquitous tracking setups[9], extending it to otherwise inaccessible environments. As stated before RFID technology is expected to become more common and enter areas of our daily life. As a consequence this also can be seen as a low cost tracking infrastructure becoming ubiquitously available which can lead to numerous applications. One such envisioned possibility is assisted wayfinding and navigation in supermarkets. This scenario further is supported by the fact that today many consumer products already start to be tagged by RFIDs, as some preliminary field studies show. Another similar scenario could be the installation of tags in underground parking lots to facilitate customer routing and orientation between parking decks and cars.

3

Conclusion and Outlook

Our system extends the existing approaches to a complete 6DOF tracker by combining the 3DOF orientation of the device with the 3DOF position optimized considering the antenna’s alignment.


227

The precision of the position estimate is of similar quality as the existing commercial outside-in tracking systems. Contrary to those approaches, the tracking system presented here allows simple and cost effective scaling for tracking within large areas. A major difference to the existing inside-out tracking systems is that it does not require the control input of a robot to function, allowing it to be applied for tracking mobile devices. This prototype confirms the idea of using RFID technology as a basis for location aware computing. The precision is acceptable for location based applications but leaves room for improvement. Our work shows that there is a high potential in using RFID technology for tracking purposes. For different scenarios it might not be feasible to spend additional money on a gyroscope to reduce the error but use a less directional antenna taking the centroid of the detected tags as pose estimation. Further Improvements. One possible way to improve the application is to examine the readability of tags in detail. Not only the antenna’s characteristic defines the probability of a tag to be detected, but also the orientation of the tag’s dipole in the RF field. This leads to the idea not only to store the position of the distributed RFID tags in a database but extend it to a 6DOF pose. Also different reader hardware setups will be investigated. Other readers allow dynamic reconfiguration of the radio parameters and also do not exhibit the buffering problem observed above. Different antennas with more pronounced directional radiation patterns may also influence the performance of the system. Another influence on the readability is the close environment of the tag, the material it is attached to as well as any material preventing it from beeing read by the reader, in case it is placed in the objects inside. This should also be taken into account when trying to estimate the reader’s position. Currently also ignored, yet of major importance to the position estimation, is the impact of undetected tags on the position estimate. An estimate not only becomes more likely if the detected tags are mapped perfectly to the antenna model but also must decline if many tags within the readability field are invisible. Our next extension will be a Kalman Filter based model that takes a tag list as measurement input and compares this to the readability derived from the antenna model. This allows the algorithm to also exploit undetected tags and to reach an even better estimate. Additionally the Kalman Filter will use an internal motion model for continous tracking.

Acknowledgments This paper profits from the work done by Marcus Fey in his diploma thesis. Our research was supported by the BFS (Bayerische Forschungsstiftung) project trackframe.

228


References 1. Ong, S.K., Nee, A.Y.C. (eds.): Virtual and Augmented Reality Applications in Manufacturing. Springer, Heidelberg (2004) 2. Speckmann, H., Ley, M.: Multi-media NDT procedures and online-maintenance assistance in the frame of the european R&D project INDeT (integration of NDT). In: ECNDT 2006, 9th European Conference on Non-Destructive Testing, Berlin, Berlin, Germany, Deutsche Gesellschaft f¨ ur Zerst¨ orungsfreie Pr¨ ufung e.V. (2006) 3. Finkenzeller, K.: RFID-Handbuch, Grundlagen und praktische Anwendungen induktiver Funkanlagen, Transponder und kontaktloser Chipkarten, 3rd edn. Hanser (2002) 4. Steggles, P.: Ubisense Hardware Datasheet (2006) 5. Ni, L.M., Liu, Y., Lau, Y.C., Patil, A.P.: Landmarc: Indoor location sensing using active RFID. In: First IEEE International Conference on Pervasive Computing and Communications, vol. 1, pp. 407–415 (2003) 6. H¨ ahnel, D., Burgard, W., Fox, D., Fishkin, K., Philipose, M.: Mapping and localization with RFID technology. In: Proc. of the IEEE International Conference on Robotics and Automation (ICRA) (2004) 7. Bohn, J., Mattern, F.: Super-Distributed RFID Tag Infrastructures. In: Markopoulos, P., Eggen, B., Aarts, E., Crowley, J.L. (eds.) EUSAI 2004. LNCS, vol. 3295, pp. 1–12. Springer, Heidelberg (2004) 8. Najafi, H., Navab, N., Klinker, G.: Automated initialization for marker-less tracking: A sensor fusion approach. In: Proc. of International Symposium on Mixed and Augmented Reality, Arlington, VA, USA (November 2004) 9. Newman, J., Wagner, M., Bauer, M., MacWilliams, A., Pintaric, T., Beyer, D., Pustka, D., Strasser, F., Schmalstieg, D., Klinker, G.: Ubiquitous tracking for augmented reality. In: Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2004), Arlington, VA, USA (November 2004)

A Component-Based Ambient Agent Model for Assessment of Driving Behaviour Tibor Bosse, Mark Hoogendoorn, Michel C.A. Klein, and Jan Treur Vrije Universiteit Amsterdam, Department of Artificial Intelligence, de Boelelaan 1081a, 1081 HV Amsterdam, The Netherlands {tbosse, mhoogen, mcaklein, treur}@cs.vu.nl http://www.cs.vu.nl/~{tbosse, mhoogen, mcaklein, treur}

Abstract. This paper presents an ambient agent-based model that addresses the assessment of driving behaviour. In case of negative assessment, cruise control slows down and stops the car. The agent model has been formally specified in a component-based manner in the form of an executable specification that can be used for prototyping. A number of simulation experiments have been conducted. Moreover, dynamic properties of components at different aggregation levels and interlevel relations between them have been specified and verified.

1 Introduction Recent developments within Ambient Intelligence provide new technological possibilities to contribute to personal care for safety, health, performance, and wellbeing; cf. [1], [2], [9]. Applications make use of possibilities to acquire sensor information about humans and their functioning, and knowledge for analysis of such information. Based on this, ambient devices can (re)act by undertaking actions in a knowledgeable manner that improve the human’s, safety, health, performance, and wellbeing. The focus of this paper is on driving behaviour. Circumstances may occur in which a person’s internal state is affected in such a way that driving is no longer safe. For example, when a person has taken drugs, either prescribed by a medical professional, or by own initiative, the driving behaviour may be impaired. For the case of alcohol, specific tests are possible to estimate the alcohol level in the blood, but for many other drugs such tests are not available. Moreover, a bad driver state may have other causes, such as highly emotional events, or being sleepy. Therefore assessment of the driver’s state by monitoring the driving behaviour itself and analysing the monitoring results is a wider applicable option. A component-based ambient agent-based model is presented to assess a person’s driving behaviour, and in case of a negative assessment to let cruise control slow down and stop the car. The approach was inspired by a system that is currently under development by Toyota. This ambient system that in the near future will be incorporated as a safety support system in Toyota’s prestigious Lexus line, uses sensors that can detect the driver’s steering operations, and sensors that can detect the focus of the driver's gaze. F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 229–243, 2008. © Springer-Verlag Berlin Heidelberg 2008

230

T. Bosse et al.

The ambient agent-based system model that is presented and analysed here includes four types of agents: sensoring agents, monitoring agents, a driver assessment agent, and a cruise control agent (see also Figure 1). Models for all of these types of agents have been designed as specialisations of a more general componentbased Ambient Agent Model. Within the model of the driver assessment agent, a model of a driver’s functioning is used expressing that an impaired internal state leads to observable behaviour showing abnormal steering operation and unfocused gaze. The monitor agent model includes facilities to automatically analyse incoming streams of information by verifying them on temporal patterns that are to be monitored (for example, instable steering operation over time). The design has been formally specified in the form of a component-based executable agent-based model that can be used for prototyping. A number of simulation experiments have been conducted. Moreover, dynamic properties of components at different aggregation levels and interlevel relations between them have been formally specified and verified against these traces. The paper is organised as follows. First, the modelling approach is introduced in Section 2. In Section 3 the global structure of the agent-based model is introduced, whereas Section 4 presents a generic ambient agent model. Specialisations of this generic agent model for the specific agents within the system are introduced in Section 5, and in Section 6 simulation results using the entire model are described. Section 7 shows the results of verification of properties against the simulation traces, and finally Section 8 is a discussion.

2 Modelling Approach This section briefly introduces the modelling approach used. To specify the model conceptually and formally, the agent-oriented perspective is a suitable choice. The modelling approach uses the Temporal Trace Language TTL for formal specification and verification of dynamic properties [3], [7]. This predicate logical language supports formal specification and analysis of dynamic properties, covering both qualitative and quantitative aspects. TTL is built on atoms referring to states, time points and traces. A state of a process for (state) ontology Ont is an assignment of truth values to the set of ground atoms in the ontology. The set of all possible states for ontology Ont is denoted by STATES(Ont). To describe sequences of states, a fixed time frame T is assumed which is linearly ordered. A trace γ over state ontology Ont and time frame T is a mapping γ : T → STATES(Ont), i.e., a sequence of states γt (t ∈ T) in STATES(Ont). The set of dynamic properties DYNPROP(Ont) is the set of temporal statements that can be formulated with respect to traces based on the state ontology Ont in the following manner. Given a trace γ over state ontology Ont, the state in γ at time point t is denoted by state(γ, t). These states can be related to state properties via the formally defined satisfaction relation |=, comparable to the Holds-predicate in the Situation Calculus [8]: state(γ, t) |= p denotes that state property p holds in trace γ at time t. Based on these statements, dynamic properties can be formulated in a sorted first-order

A Component-Based Ambient Agent Model

231

predicate logic, using quantifiers over time and traces and the usual first-order logical connectives such as ¬, ∧, ∨, ⇒, ∀, ∃. A special software environment has been developed for TTL, featuring a Property Editor for building TTL properties and a Checking Tool that enables formal verification of such properties against a set of traces. To specify simulation models and to execute these models, the language LEADSTO, an executable sublanguage of TTL, is used (cf. [4]). The basic building blocks of this language are causal relations of the format α → →e, f, g, h β, which means: If then

state property α holds for a certain time interval with duration g, after some delay (between e and f) state property β will hold for a certain time interval of length h.

where α and β are state properties of the form ‘conjunction of literals’ (where a literal is an atom or the negation of an atom), and e, f, g, h non-negative real numbers.

3 Global Structure For the global structure of the model, first a distinction is made between those components that are the subject of the system (e.g., a patient to be taken care of), and those that are ambient, supporting components. Moreover, from an Driver Assessment agent-based perspective (see, agent e.g., [5], [6]), a distinction is made between active, agent Cruise Steering Gaze-focus components(human or artificial), Control Monitoring Monitoring agent agent agent and passive, world components (e.g., part of the physical world or a database). Agents may interact through communication. Interaction between an agent Steering Gaze-focus and a world component can be Sensoring agent Sensoring agent car and environment either observation or action performance; cf. [5]. An action is generated by an agent, and transfers to a world component to have its effect there. An obdriver servation result is generated by Fig. 1. Ambient Driver Support System a world component and transferred to the agent. In Figure 1 an overview of the system is shown. Table 1 shows the structure in terms of different types of components and interactions. 3.1 State Ontologies Used at the Global Level For the information exchanged between components at the global level, ontologies have been specified. This has been done in a universal order-sorted predicate logic

232

T. Bosse et al.

format that easily can be translated into more specific ontology languages. Table 2 provides an overview of sorts and predicates used in interactions at the global level. Table 1. Components and Interactions of the Ambient Driver Support System subject components subject interactions

ambient components

ambient interactions

interactions between subject and ambient components

subject agent subject world component human driver car and environment observation and action by subject agent in subject world component driver observes car and environment driver operates car and gaze ambient agents steering and gaze-focus sensoring agent; steering and gaze-focus monitoring agent; driver assessment agent, cruise control agent communication between ambient agents steering sensoring agent communicates to steering monitoring agent gaze-focus sensoring agent communicates gaze focus to gaze-focus monitoring agent eye-focus monitoring agent reports to driver assessment agent unfocused gaze steering monitoring agent reports to driver assessment agent abnormal steering driver assessment agent communicates to cruise control agent state of driver observation and action by ambient agent in subject world component steering sensoring agent observes steering wheel gaze-focus sensoring agent observes driver gaze cruise control agent slows down or stops engine

Table 2. Ontology for Interaction at the Global Level SORT

Description an action an agent an information element, possibly complex (e.g., a conjunction of other info elements) a world component Predicate Description performing_in(A:ACTION, W:WORLD) action A is performed in W observation_result_from(I:INFO_EL, W:WORLD) observation result from W is I communication_from_to(I:INFO_EL, X:AGENT, information I is communicated by X to Y ACTION AGENT INFO_EL WORLD

Y:AGENT) communicated_from_to(I:INFO_EL,X:AGENT,Y:AGENT)

information I was communicated by X to Y

3.2 Temporal Relations at the Global Level Interaction between global level components is defined by the following specifications. In such specifications, for state properties the prefix input, output or internal is used. This is an indexing of the language elements to indicate that it concerns specific variants of them either present at the input, output or internally within the agent. Action Propagation from Agent to World Component ∀X:AGENT ∀W:WORLD ∀A:ACTION output(X)|performing_in(A, W) → input(W)|performing_in(A, W) Observation Result Propagation from World to Agent ∀X:AGENT ∀W:WORLD ∀I:INFO_EL output(W)|observation_result_from(I, W) → input(X)|observed_result_from(I, W)


233

Communication Propagation Between Agents ∀X,Y:AGENT ∀I:INFO_EL output(X)|communication_from_to(I,X,Y) →input(Y)|communicated_from_to(I,X,Y)

4 Component-Based Ambient Agent Model This section focuses on an Ambient Agent Model (AAM) used for the four types of ambient agents in the system. These agents are assumed to maintain knowledge about certain aspects of human functioning, and information about the current state and history of the world and other agents. Based on this knowledge they are able to have some understanding of the human processes, and can behave accordingly. In Section 5 it is shown how the Ambient Agent Model AAM has been specialised to obtain models for monitor agents, a driver assessment agent, and a cruise control agent. 4.1 Components within the Ambient Agent Model Based on the component-based Generic Agent Model (GAM) presented in [5], a model for ambient agents (AAM) was designed. Within AAM, as in GAM the component World Interaction Management takes care of interaction with the world, the component Agent Interaction Management takes care of communication with other agents. Moreover, the component Maintenance of World Information maintains information about the world, and the component Maintenance of Agent Information maintains information about other agents. In the component Agent Specific Task, specific tasks can be modelled. Adopting this componentbased agent model GAM, the ambient agent model has been obtained as a refinement in the following manner. The component Maintenance of Agent Information has three subcomponents in AAM. The subcomponent Maintenance of a Dynamic Agent Model maintains the causal and temporal relationships for the subject agent’s functioning. The subcomponent Maintenance of an Agent State Model maintains a snapshot of the (current) state of the agent. As an example, this may model the gaze focussing state. The subcomponent Maintenance of an Agent History Model maintains the history of the (current) state of the agent. This may for instance model gaze patterns over time. Similarly, the component Maintenance of World Information has three subcomponents for a dynamic world model, a world state model, and a world history model, respectively. Moreover, the component Agent Specific Task has the following three subcomponents: Simulation Execution extends the information in the agent state model based on the internally represented dynamic agent model for the subject agent’s functioning, Process Analysis assesses the current state of the agent, and Plan Determination determines whether action has to be undertaken, and, if so, which ones (e.g., to determine that the cruise control agent has to be informed). Finally, as in the model GAM, the components World Interaction Management and Agent Interaction Management prepare (based on internally generated information) and receive (and internally forward) interaction with the world and other agents.

234

T. Bosse et al.

4.2 State Ontologies within Agent and World To express the information involved in the agent’s internal processes, the ontology shown in Table 3 was specified. Table 3. Ontology used within the Ambient Agent Model Ontology element belief(I:INFO_EL) world_fact(I:INFO_EL) has_effect(A:ACTION, I:INFO_EL) leads_to_after(I:INFO_EL, J:INFO_EL, D:REAL) at(I:INFO_EL, T:TIME)

Description information I is believed I is fact true in the world action A has effect I state property I leads to state property J after D property I holds at time T

As an example belief(leads_to_after(I:INFO_EL, J:INFO_EL, D:REAL)) is an expression based on this ontology which represents that the agent has the knowledge that state property I leads to state property J with a certain time delay specified by D. 4.3 Generic Temporal Relations within AAM The temporal relations for the functionality within the Ambient Agent Model are: Belief Generation based on Observation and Communication

∀X:AGENT, I:INFO_EL, W:WORLD input(X)|observed_from(I, W) ∧ internal(X)|belief(is_reliable_for(W, I))

→ →

internal(X)|belief(I)

∀X,Y:AGENT, I:INFO_EL input(X)|communicated_from_ to(I,Y,X) ∧ internal(X)|belief(is_reliable_for(X, I))

→ →

internal(X)|belief(I)

Here, the first rule is a generic rule for the component World Interaction Management, and the second for the component Agent Interaction Management. When the sources are assumed always reliable, the conditions on reliability can be left out. Belief Generation based on Simulation

∀X:AGENT ∀I,J:INFO_EL ∀D:REAL ∀T:TIME internal(X)|belief(at(I, T)) ∧ internal(X)|belief(leads_to_after(I, J, D))

→ →

internal(X)|belief(at(J, T+D))

The last generic rule within the agent’s component Simulation Execution specifies how a dynamic model that is explicitly represented as part of the agent’s knowledge (within its component Maintenance of Dynamic Models) can be used to perform simulation, thus extending the agent’s beliefs about the world state at different points in time. This can be considered an internally represented deductive causal reasoning method. Another option is a multiple effect abductive causal reasoning method: Belief Generation based on Multiple Effect Abduction

∀X:AGENT ∀I,J1, J2:INFO_EL ∀D:REAL ∀T:TIME J1≠J2 ∧ internal(X)|belief(at(J1, T)) ∧ internal(X)|belief(leads_to_after(I, J1, D)) ∧ internal(X)|belief(at(J2, T)) ∧ internal(X)|belief(leads_to_after(I, J2, D)) → → internal(X)|belief(at(I, T-D))

4.4 Generic Temporal Relations within a World For World Components the following specifications indicate the actions’ effects and how observations provide their results. Action Execution and Observation Result Generation in a World

∀W:WORLD_COMP ∀A:ACTION ∀I:INFO_EL input(W)|performing_in(A, W) ∧ internal(W)|has_effect(A,I)

A Component-Based Ambient Agent Model → →

235

internal(W)|world_fact(I)

∀W:WORLD_COMP ∀I:INFO_EL input(W)|observation_focus_in(I, W) ∧ internal(W)|world_fact(I)

→ →

output(W)|observation_result_from(I, W)

∀W:WORLD_COMP ∀I:INFO_EL input(W)|observation_focus_in(I, W) ∧ internal(W)|world_fact(not(I)) output(W)|observation_result_from(not(I), W)

→ →

5 Instantiations of the Ambient Agent Model This section provides instantiations of the Ambient Agent Model for, respectively, Ambient Monitor Agents, a Driver Assessment Agent, and a Cruise Control Agent. 5.1 Ambient Monitor Agents As a refinement of the Ambient Agent Model AAM, an Ambient Monitoring Agent Model AMAM has been designed, and instantiated for steering monitoring and gaze monitoring. Table 4 indicates the components within these monitoring agents. These agents relate temporal patterns of gaze, resp. steering to qualifications of abnormality. Table 4. Ambient Monitor Agent Model: Components Maintenance of Agent and World Information maintenance of model of gaze/steering patterns over time history models Agent Specific Task process analysisdetermine whether a gaze/steering pattern is abnormal plan determinationfor abnormality state decide to communicate to driver assessment agent Agent Interaction Management prepare communication to driver assessment agent

A monitor agent receives a stream of information over time, obtained by observation of a world component or by communication from other agents. Typical sources of information are world parts equipped with sensor systems or sensoring agents that interact with such world parts. Any monitoring agent has some properties of the incoming information stream that are to be monitored (monitoring foci), e.g., concerning the value of a variable, or a temporal pattern to be detected in the stream. As output a monitoring agent generates communication that a certain monitoring focus holds. A monitor focus can be a state property or a dynamic property. An example of a simple type of state property to be used as a monitor focus is a state property that expresses that the value of a certain variable X is between two bounds LB and UB: ∃V [ has_value(X, V) ∧ LB ≤ V ∧ V ≤ UB ]. In prefix notation, this can be expressed as follows: exists(V, and(has_value(X, V), and(LB ≤ V, V ≤ UB))). It is possible to obtain abstraction by using (meaningful) names of properties. For example, stable_within(X, LB, UB) can be used as an abstract name for the example property expressed above by specifying: has_expression(stable_within(X, LB, UB), exists(V, and(has_value(X, V), and(LB ≤ V, V ≤ UB))))

The fact that a property stable_within(X, LB, UB) is a monitor focus for the monitor agent is expressed by: monitor_focus(stable_within(X, LB, UB)). An example of a monitor property is:

236

T. Bosse et al.

∀t [ t1≤t ∧ t≤t2 ∧ at(has_value(X, V1), t1) → ∃t’, V2 t≤ t’ ≤ t+D ∧ V2 ≠V1 ∧ at(has_value(X, V2), t’) ]

This property expresses that between t1 and t2 the value of variable X is changing all the time, which can be considered as a type of instability of that variable. This dynamic property is expressed in prefix notation as: forall(t, implies(and(t1≤t, and(t≤t2, at(has_value(X, V1), t))), exists(t’, exists(V2, and(t≤ t’, and(t’ ≤ t+D, and(V2 ≠V1, at(has_value(X, V2), t’))))

This expression can be named, for example, by instable_within_duration(X, D). It is assumed that the monitor focus on which output is expected is an input for the agent, communicated by another agent. This input is represented in the following manner. communicated_from_to(monitor_focus(F), A, B) communicated_from_to(has_expression(F, E), A, B)

Note that it is assumed here that the ontology elements used in the expression E here are elements of the ontology used for the incoming stream of information. Moreover, note that for the sake of simplicity, sometimes a prefix such as input(X)|, which indicates in which agent a state property occurs, is left out. Within AMAM’s World Interaction Management component, observation results get a time label: observed_result_in(I, W) ∧ current_time(T) → → belief(at(I, T)). Similarly, within the Agent Interaction Management component communicated information is labeled: communicated_from_to(I, X, AMAM) ∧ current_time(T) → → belief(at(I, T)). The time-labeled consequent atoms belief(at(I, T)) are transferred to the component Maintenance of Agent History and stored there. Within the component Process Analysis two specific subcomponents are used: Monitoring Foci Determination, and Monitor Foci Verification. Monitoring Foci Determination. In this component the monitor agent’s monitoring foci are determined and maintained: properties that are the focus of the agent’s monitoring task. The overall monitoring foci are received by communication and stored in this component. However, to support the monitoring process, it is useful when an overall monitor focus is decomposed into more refined foci: its constituents are determined (the subformulas) in a top-down manner, following the nested structure. This decomposition process was specified in the following manner: monitor_focus(F)

→ →

in_focus(F)

in_focus(E) ∧ is_composed_of (E, C, E1, E2)

→ →

in_focus(E1) ∧ in_focus(E2)

Here is_composed_of(E, C, E1, E2) indicates that E is an expression obtained from subexpressions E1 and E2 by a logical operator C (i.e., and, or, implies, not, forall, exists). Monitoring Foci Verification. The process to verify whether a monitoring focus holds, makes use of the time-labeled beliefs that are maintained. If the monitoring focus is an atomic property at(I, T) of the state of the agent and/or world at some time point, beliefs about these state properties are involved in the verification process: in_focus(E) ∧ belief(E)

→ →

verification(E, pos)

Verification of more complex formulae is done by combining the verification results of the subformulae following the nested structure in a bottom-up manner: in_focus(and(E1, E2)) ∧ verification(E1, pos) ∧ verification(E2, pos) in_focus(or(E1, E2)) ∧ verification(E1, pos)

→ →

→ →

verification(and(E1, E2) , pos)

verification(or(E1, E2) , pos)

A Component-Based Ambient Agent Model in_focus(or(E1, E2)) ∧ verification(E2, pos)

→ →

237

verification(or(E1, E2) , pos)

→ →

in_focus(implies(E1, E2)) ∧ verification(E2, pos)

in_focus(implies(E1, E2)) ∧ not verification(E1, pos)

verification(implies(E1, E2), pos)

→ →

verification(implies(E1, E2), pos)

→ → verification(not(E), pos) in_focus(exists(V, E)) ∧ verification(E, pos) → → verification(exists(V, E), pos) in_focus(forall(V, E)) ∧ not verification(exists (V, not(E), pos) → → verification(forall(V, E), pos) in_focus(not(E)) ∧ not verification(E, pos)

The negative outcomes not verification(E, pos) of verification can be obtained by a Closed World Assumption on the verification(E, pos) atoms. If needed, from these negations, explicit negative verification outcomes can be derived: not verification(E, pos)

→ →

verification(E, neg)

The following relates verification of an expression to its name: verification(E, S) ∧ has_expression(F, E)

→ →

verification(F, S)

Eventually, when a monitoring property E has been satisfied that is an indication for a certain type of abnormal behaviour of the driver, the Monitoring agent will indeed believe this; for example, for the Steering Monitoring Agent: verification(E, pos) ∧ internal(monitoring_agent)|belief(is_indication_for(E, I))

→ →

internal(monitoring_agent)|belief(I)

5.2 Driver Assessment Agent As another refinement of the Ambient Agent Model AAM, the Driver Assessment Agent Model DAAM; see Table 5 for an overview of the different components. For the Driver Assessment Agent, a number of domain-specific rules have been identified in addition to the generic rules specified for the Ambient Agent Model presented in Section 4. Some of the key rules are expressed below. First of all, within the Driver Assessment Agent an explicit representation is present of a dynamic model of the driver’s functioning. In this model it is represented how an impaired state has behavioural consequences: abnormal steering operation and gaze focusing. Table 5. Driver Assessment Agent Model: Components Maintenance of Agent and World Information maintenance of dynamic models model relating impaired state to abnormal steering behaviour and gaze focussing maintenance of model of internal state, abnormality of gaze of driver, and of steering state models wheel Agent Specific Task process analysis determine impaired driver state by multiple effect abduction plan determination for impaired driver state decide to communicate negative assessment to cruise control agent Agent Interaction Management receive and prepare communication (from monitor agents, to cruise control agent)

The dynamic model is represented in component Maintenance of Dynamic Models by: internal(driver_assessment_agent)|belief(leads_to_after(impaired_state, abnormal_steering_operation, D)) internal(driver_assessment_agent)|belief(leads_to_after(impaired_stste, unfocused_gaze, D))

238

T. Bosse et al.

The Driver Assessment Agent receives information about abnormality of steering and gaze from the two monitoring agents. When relevant, by the multiple effect abductive reasoning method specified by the generic temporal rule in Section 4, the Driver Assessment Agent derives a belief that the driver has an impaired internal state. This is stored as a belief in the component Maintenance of an Agent State Model. Next, it is communicated to the Cruise Control Agent that the driver assessment is negative. 5.3 Cruise Control Agent The Cruise Control Agent Model CCAM is another agent model obtained by specialisation of the Ambient Agent Model AAM. It takes the appropriate measures, whenever needed. Within its Plan Determination component, the first temporal rule specifies that if it believes that the driver assessment is negative, and the car is not driving, then the ignition of the car is blocked: internal(cruise_control_agent)|belief(driver_assessment(negative)) ∧ internal(cruise_control_agent)|belief(car_is_not_driving)

→ → output(cruise_control_agent)|performing_in(block_ignition, car_and_environment)

If the car is already driving, the car is slowed down: internal(cruise_control_agent)|belief(driver_assessment(negative)) ∧ internal(cruise_control_agent)|belief(car_is_driving)

→ → output(cruise_control_agent)|performing_in(slow_down_car, car_and_environment)

6 Simulation Results Based upon temporal rules as described in previous section, a specification within the LEADSTO software environment (cf. [4]) has been made and simulation runs of the system have been generated, of which an example trace is shown in Figure 2. In the figure, the left side indicates the atoms that occur during the simulation whereas the right side indicates a time line where a dark box indicates the atom is true at that time point and a grey box indicates false. Note that in the trace merely the outputs and internal states of the various components are shown for the sake of clarity. The driver starts the car and accelerates, resulting in a driving car. internal(car_and_environment)|world_fact(car_driving)

After a short time, between time points 10 and 20, the driver shows signs of inadequate behaviour: the gaze becomes unfocused and steering instable. Over short time intervals an alternation occurs of: output(driver)|performing_in(steer_position(centre), car_and_environment) output(driver)|performing_in(steer_position(left), car_and_environment) output(driver)|performing_in(steer_position(right), car_and_environment)

On the other hand, the gaze focus becomes fixed for long time intervals: output(driver)|observation_result_from(gaze_focus(far_away), driver)

The temporal sequences of these observed steering positions and gaze focus are communicated moment by moment by the respective sensoring agent to the

A Component-Based Ambient Agent Model internal(car_and_environment)|world_fact(car_not_driving) output(driver)|performing_in(start_engine, car_and_environment) output(driver)|performing_in(steer_position(centre), car_and_environment) output(car_and_environment)|observation_result_from(car_not_driving, car_and_environment) internal(car_and_environment)|world_fact(steer_position(centre)) internal(car_and_environment)|world_fact(engine_running) output(car_and_environment)|observation_result_from(steer_position(centre), car_and_environment) output(car_and_environment)|observation_result_from(engine_running, car_and_environment) output(driver)|performing_in(accelerate, car_and_environment) output(steering_sensoring_agent)|communication_from_to(steer_position(centre), steering_sensoring_agent, steering_monitoring_agent) internal(car_and_environment)|world_fact(car_driving) output(car_and_environment)|observation_result_from(car_driving, car_and_environment) internal(driver)|world_fact(gaze_focus(far_away)) output(driver)|performing_in(steer_position(left), car_and_environment) output(driver)|observation_result_from(gaze_focus(far_away), driver) verification(gp(1), pos) output(driver)|performing_in(steer_position(right), car_and_environment) internal(car_and_environment)|world_fact(steer_position(left)) verification(gp(15), pos) output(car_and_environment)|observation_result_from(steer_position(left), car_and_environment) output(gaze_focus_sensoring_agent)|communication_from_to(gaze_focus(far_away), gaze_focus_sensoring_agent, gaze_focus_monitoring_agent) internal(car_and_environment)|world_fact(steer_position(right)) output(car_and_environment)|observation_result_from(steer_position(right), car_and_environment) output(steering_sensoring_agent)|communication_from_to(steer_position(left), steering_sensoring_agent, steering_monitoring_agent) output(steering_sensoring_agent)|communication_from_to(steer_position(right), steering_sensoring_agent, steering_monitoring_agent) output(steering_monitoring_agent)|communication_from_to(abnormal_steering_operation, steering_monitoring_agent, driver_assessment_agent) output(gaze_focus_monitoring_agent)|communication_from_to(unfocused_gaze, gaze_focus_monitoring_agent, driver_assessment_agent) output(driver_assessment_agent)|communication_from_to(driver_assessment(negative), driver_assessment_agent, cruise_control_agent) output(cruise_control_agent)|performing_in(slow_down_car, car_and_environment) output(cruise_control_agent)|performing_in(block_ignition, car_and_environment) internal(car_and_environment)|world_fact(engine_always_off) time

0

10

20

30

239

40

50

60

Fig. 2. Example Simulation Trace

corresponding monitoring agent. The following dynamic monitor property is used as monitor focus within the Steering Monitoring Agent: ∀t [ t1≤t ∧ t≤t2 ∧ belief(at(steer_position(centre), t)) → ∃t’ t≤ t’ ≤ t+D ∧ not belief(at(steer_position(centre), t’))

This property expresses that between t1 and t2, whenever the steer is in a central position, there is a slightly later time point at which it is not in a central position (in other words, the driver keeps on moving the steer). This dynamic property is expressed in prefix notation as: forall(t, implies(and(t1 ≤ t, and(t ≤ t2, belief(at(steer_position(centre), t)))), exists(t’, and(t ≤ t’, and(t’ ≤ t+D, not(belief(at(steer_position(centre), t’))))

In LEADSTO this property was expressed as: is_composed_of(gp(1), forall, t, gp(2, t)) is_composed_of(gp(2, t), implies, gp(3, t), gp(8, t)) is_composed_of(gp(3, t), and, gp(4, t), gp(5, t)) has_expression(gp(4, t), t1≤t) is_composed_of(gp(5, t), and, gp(6, t), gp(7, t)) has_expression(gp(6, t), t≤t2) has_expression(gp(7, t), belief(at(steer_position(centre), t))) is_composed_of(gp(8, t), exists, t’, gp(9, t, t’))

is_composed_of(gp(9, t, t’), and, gp(10, t, t’), gp(11, t, t’)) has_expression(gp(10, t, t’), t≤t’) is_composed_of(gp(11, t, t’), and, gp(12, t, t’), gp(13, t, t’)) has_expression(gp(12, t, t’), t’≤sum(t, D)) is_composed_of(gp(13, t, t’), not, gp(14, t, t’), gp(14, t, t’)) has_expression(gp(14, t, t’), belief(at(steer_position(centre), t’)))

Note that during the process within the Steering Monitoring Agent the overall monitoring focus given by this dynamic property is decomposed into a number of smaller expressions (using the predicate is_composed_of). The top level expression (that is checked by the Steering Monitoring Agent) is called gp(1). The atomic expressions

240

T. Bosse et al.

have the form of a belief that a state property holds at a certain time point (e.g., belief(at(steer_position(centre), t))), or of an inequality (e.g., t≤t2). The following dynamic monitor property is used as monitor focus within the Gaze Focus Monitoring Agent: ∃t ∀t’ [ t ≤ t’ ≤ t+D → belief(at(gaze_focus(far_away), t’)) ]. This property expresses that there is a time period from t to t+D in which the gaze of the driver is focused at a point far away. It is expressed in prefix notation as: exists(t, forall(t’, implies(and(t≤t’, t’≤t+D), belief(at(gaze_focus(far_away), t’))))). Within the LEADSTO model, this property was expressed as: is_composed_of(gp(15), exists, t, gp(16, t)) is_composed_of(gp(16, t), forall, t’, gp(17, t, t’)) is_composed_of(gp(17, t, t’), implies, gp(18, t, t’), gp(21, t, t’)) is_composed_of(gp(18, t, t’),

and, gp(19, t, t’), gp(20, t, t’)) has_expression(gp(19, t, t’), t≤t’) has_expression(gp(20, t, t’), t’≤sum(t, D)) has_expression(gp(21, t, t’), belief(at(gaze_focus(far_away), t’)))

Here, the top level expression (that is checked by the Gaze Focus Monitoring Agent) is called gp(15). Given these monitoring foci, the monitoring agents detect the patterns in this sensor information, classify them as abnormal, and communicate this to the Driver Assessment Agent. By the multiple effect abductive reasoning method, this agent generates the belief that the driver is having an impaired state, upon which a negative driver assessment is communicated to the Cruise Control Agent. The Cruise Control Agent first slows down the car, and after it stopped, blocks the ignition: output(cruise_control_agent)|performing_in(slow_down_car, car_and_environment) output(cruise_control_agent)|performing_in(block_ignition, car_and_environment)

7 Verification of Dynamic Properties This section addresses specification and verification of relevant dynamic properties of the cases considered, for example, requirements imposed on these systems. 7.1 Properties of the System as a Whole A natural property of the Ambient Driver Support System is that a driver with impaired driving behaviour cannot continue driving. The global property is: GP1 No driving when symptoms of impaired driving occur If the driver exposes symptoms that indicate that it is not safe to drive anymore then within 30 seconds the car will not drive and the engine will be off ∀γ:TRACE, t:TIME, R:REAL (unfocused_gaze(t, γ) ∧ abnormal_steering_behaviour(t, γ)) ⇒ ∃t2:TIME < t:TIME + 30 [state(γ, t2, internal(car_and_environment))|= world_fact(car_not_driving)]

This property makes use of two other properties: UG Unfocussed gaze has occurred for some time. In trace γ, during the time period D just before t, the gaze of the driver was focussed at a far distance. ∀t2:TIME ((t2 = t-D)) ⇒ [state(γ, t2, internal(driver)|= world_fact(gaze_focus(far_away))]

AS Abnormal steering behaviour has occurred In trace γ, during a time period P just before t, whenever the steer is in a central position, there is time point within D time steps at which the steer is not in a central position.


241

∀t:TIME ((t-P-D < t2) ∧ (t2 < t-D) ∧ [state(γ, t2, internal(car_and_environment)|= world_fact(steer_position(centre))]) ⇒ ∃t3:TIME, ((t t}. Fig. 4(b) depicts an example for a region Ω(0.9). Considering Ω(t) it is possible to calculate the probability of taking a correct decision for that region using equation Ω(t) 4. Now the problem is reduced to find one value of t that maximize P r(Cd ) conΩ(t) strained to P r(Cd ) >= ρ0 . If regions of confidence cannot be find that meet the constrain, it is necessary to substitute the decision rule with a more robust one. Another alternative is to include new observable aspects to the system. Specific details for our experiments will be discussed in the next section. When the training phase ends, we have, for each position Ofkix a confidence region Ωkρ0 . During the system-run phase, a mobile device running a self-localization software is allowed to move within the building. Periodically, the software issues one or more inquiry commands, depending on the decision rule adopted in the training phase. Because the mobile device only issues inquiries, there is no effective connections or change of personal informations. Such feature assures the privacy of the node realizing the location. Additionally, we just require the fixed nodes to be able to answer to inquiries. That has a twofold advantage. First, the hardware used to implement a fixed node may be very simple and cost effective. It may implement only the Bluetooth functions related to the inquiry command. Second, it consumes very low power, because the inquiry command is issued only on demand and very rapidly. In our experiments, the fixed node (Bluetooth chip) consumed only 3.3 mW when in inquiry scan mode and 29 mW when answering to an inquiry command, which takes only 625 microseconds. After acquiring information about the environment, the mobile device runs the decision rule to decide in which regions it is to be found. The software uses the information about regions to identify the sector where the mobile device is located. Additionally, each sector is correlated with the spatial information about the building. Though, it is possible to present the localization result as a region on the building floorplan map on the mobile device’s display. The next sections show the realization of a self-localization and personal navigation system using the concept explained here.

4

Experiments

The region-based localization methodology, explained in the last section, was applied to realize an indoor localization and personal navigation system. We describe our experiments considering the two phases of the implementation: training and system-run phases.

264

J.O. Filho et al.

4.1

Training Phase

A section of the Wilhelm-Schickard-Institut building, at the University T¨ ubingen was used as a real scenario for our experiments. Its basic floorplan can be seen in Fig. 5. A grid of points with one meter displacement among each other was set up over the area. During the training phase, the mobile devices were allowed to be positioned only at those points. Four points in this grid, depicted in Fig. 5 as Off1ix , Off2ix , Ofr1ix and Ofr2ix , were chosen to permanently accommodate one Bluetooth Smart Node (BTnodes), developed at the Swiss Federal Institute of Technology (ETH). Those points compose the set of fixed nodes. Each fixed node was placed considering aspects of the building topology, because we wanted to investigate how it later could affect the shape of the respective region of confidence. Off1ix was positioned at the middle of a straight corridor, while Off2ix was placed at a bent corridor corner. The regions of confidence for Off1ix and Off2ix are expected to spread along the corridor, with a smaller coverage in the second case, due to additional reflexions on the corner. Ofr1ix and Ofr2ix where placed inside a small and a big room, respectively. The idea was to investigate how the shape of the region of confidence would spread out of the room.

Fig. 5. Experimental set up. Points in the grid are spaced by 1 meter. Fixed nodes Of ix answer to inquiry commands issued by a cell phone placed at the indicated measuring points on the grid.

Self-Localization in a Low Cost Bluetooth Environment

265

Fig. 6. Probability maps for the answered inquiry commands

First task in the training phase is to measure the probability of the inquiry event for each point in the grid. That was accomplished executing the following procedure at each point. First, a BTnode and a Sony Ericsson V600i mobile devices were placed at the point. We used two different devices to keep track on the possible variation of using diverse equipments. Each device executed one hundred inquiry commands. In each round, it is logged which fixed nodes answered the command. When all the commands were issued, the rate of successful inquiries may be determined individually for each fixed node. Such rate is used as an estimator for Ok

the probability pωif ix , explained in section 3. The result of this procedure can be seen in Fig. 6, considering each fixed point individually. The variation on the rate of successful inquiries was very small between the two applied devices. The mobile devices obtained a high rate (nearly 100%) of answered inquiries from Off1ix and Off2ix , when they were located in the corridor. That is because they were in the line-of-sight of the fixed BTnodes. When leaving the line-of-sight or when in presence of obstacles, a fast drop-off appears and the mobile devices can hardly reach the fixed BTnodes. We assumed that the fixed nodes Ofr1ix and Ofr2ix would answer to all the inquiries done within the room they are placed with probability 1. In both cases there is a region in the corridor with a radius of about 10 meters where the probability to get an inquiry response is still high. Outside these bounds this probability vanishes very rapidly. During our experiments, the doors were kept open to avoid attenuation of the inquiry signal. However, we stated that doors and metalic obstacles do affect the measured probability and a detailed study about their effects is part of our on going work.

266

J.O. Filho et al.

Regions of Confidence. The regions of confidence Ωkρ0 for each fixed node Ofkix should now be determined. In order to do that, it is necessary to fixate the desired degree of confidence ρ0 and adopt one decision rule. For the remaining of this work, we arbitrarily set ρ0 = 0.9, which guarantees decisions with at least 90% of confidence for all regions. Additionally, we investigate two possible decision rules. The first decision rule is given in equation 5. It considers that the mobile device is in region Ωkρ0 if the fixed node Ofkix answers one inquiry command issued by the mobile device. 1 , if Ofkix answered the inquiry. d1 (one inquiry) = (5) 0 , if Ofkix did not answered the inquiry. The second decision rule considers the last three inquiry commands issued by the mobile device. It decides that such mobile device is in region Ωkρ0 if the fixed node Ofkix answered to at least two of the three commands. It can be described as d2 (three inquiries) =

1 0

, if Ofkix answered the at least two inquiries. , if Ofkix answered less than two inquiries.

(6) We are now able to determine the regions of confidence for one given fixed node Of ix . As an example, we consider the fixed node Off1ix . We grouped the points using a threshold value t as explained in section 3. Fig. 7 shows the obtained value for the degree of confidence ρ as a function of t ∈ [0, 1] considering d2 as the decision rule. The maximum probability of making a correct decision is 96.9%. It is achieved when t = 56. Our region includes every point of the measured grid 1 0,9 0,8

degree of confidence

0,7 0,6 0,5 0,4 0,3 0,2 0,1 0

threshold t

Fig. 7. Degree of confidence for Ω(t) of Off1ix


267

where the number of answered inquiry commands lies between 100 and 56. For this region, we can take a correct decision with a probability 96.6%. Interestingly, that result also stands for regions defined with threshold t between 56 and 31. We keep using t = 56, which corresponds to the smallest regions with the higher degree of confidence. Smaller regions lead to greater accuracy. The same procedure was taken on determining the regions for the other fixed nodes. Table 1 shows the maximal value for the degree of confidence reached when using decision rules d1 and d2 . Decision rule d1 fails to provide regions of confidence with ρ >= ρ0 = 0.9 in two cases. Therefore, it should be rejected. Decision rule d2 , however, was able to provide adequate regions for all fixed nodes. It was adopted to be used in the system-run phase. The threshold value t used to build the region of each fixed node is also indicated on table 1. The regions defined for Off1ix , Off2ix , Ofr1ix and Ofr2ix may be seen in Fig. 8. Those regions have degree of confidence ρ equal to 0.969, 0.907, 0.907 and 0.941, respectively. Though, they are all regions of confidence Ω 0.9 . Table 1. Maximal degree of confidence according to decision rule Fixed Node

d1

d2

Final Threshold (t) when using d2

Off1ix Off2ix Ofr1ix Ofr2ix

94.5% 87.7% 85.7% 90.4%

96.9% 90.7% 90.7% 94.1%

56% 51% 51% 52%

Fig. 8. Regions of confidence calculated for Off1ix (top left), Off2ix (top right), Ofr1ix (bottom left) and Ofr2ix (bottom right)

268

J.O. Filho et al.

After the training phase, the environment is prepared for running the selflocalization and navigation system. The fixed nodes are installed and the regions were already determined. The system-run phase for our scenario is explained in details in the next section.

5

Running the Localization and Navigation System

The information for each region of confidence, obtained in the training phase, were merged under one unique building floorplan map. The result may be seen in Fig. 9(a). The floor plan view allows us now to split the space into sectors. Each sector corresponds to an area on the space ”covered” by one or more regions of confidence. Therefore, it is possible to relate each sector with the corresponding regions that cover it. Such information is tabulated and will be available to be used by the self-localization software. As an application example, a self-localization software was implemented using Java 2 Micro Edition and JSR-82 Bluetooth API. The implementation in Java allows the software to run on nearly every mobile device like a mobile phone or a PDA. The Java application has got two main functionalities. First, a localization mode enables the user to find out his location in the building. Second, a navigation mode allows the user to choose a destination. The system will navigate him to the selected place. When the localization mode is active, the software starts the Bluetooth service at the mobile device and issues three inquiry commands. The system analyzes the inquiry responses to find out which fixed BTnodes have answered and how often. Due to our decision rules a BTnode must answer at least two times to get a valid predicate. Answers of different regions are combined to determine in which sector the mobile device is to be found. After this evaluation, the resulting sector

(a) Regions overlap to form sectors on the building plan.

(b) Software output screen for the localization mode.

Fig. 9. Example system for localization and navigation


269

is displayed. It takes about 30 seconds from starting the application to obtaining the actual location. Fig. 9(b) shows the output screen for the localization mode. Up to three mobile devices were allowed to start inquiry commands simultaneously in order to investigate concurrency issues in the system. However, there were not observed colision or delay problems. The navigation functionality is an extended localization mode, where the software on the mobile device periodically issues inquiry commands. After each round, it executes the decision rule d2 and refreshes the output screen accordingly. The whole process takes about 5 seconds. Note that all the operation is executed by the software on the mobile device. With the small, battery operated Bluetooth landmarks a minimal infrastructure hardware is required. Even though, the system is reliable enough to support localization and navigation services. Accuracy in the localization may be customized for each building by introducing new fixed nodes.

6

Conclusion and Future Work

We presented a low cost, low power, Bluetooth based self-localization and navigation system. The proposed region-based localization concept is suitable to be completely executed on the mobile device to be localized. Nevertheless, a high degree of confidence on the localization is achieved. Self-localization, without the need of changing personal information with the environment, makes our approach a safe and privacy aware solution. Our experience shows, that it is possible to build reliable, broad usable and low cost personal navigation systems using Bluetooth. The system was implemented in a real scenario at the University T¨ ubingen. It uses a Java software, suitable to be executed in any Java capable mobile device, such as mobile phones and PDAs. Accuracy is yet a point to be improved. That may be achieved optimizing the position of the fixed Bluetooth nodes that compose the environment infrastructure. A methodology to optimally place these nodes, investigating the influence of obstacles such as doors, and how the system behaviour when simultaneously operated by several mobile devices is part of our actual ongoing work.

References 1. Sommer, J., Hoene, C.: Indoor positioning applications. localisation in wireless environments using bluetooth and wlan. In: Third Heidelberg Innovation Forum (2006) 2. Aitenbichler, E., M¨ uhlh¨ auser, M.: An ir local positioning system for smart items and devices. In: ICDCSW 2003: Proceedings of the 23rd International Conference on Distributed Computing Systems, Washington, DC, USA, p. 334. IEEE Computer Society, Los Alamitos (2003) 3. Fukuju, Y., Minami, M., Morikawa, H., Aoyama, T.: Dolphin: An autonomous indoor positioning system in ubiquitous computing environment. In: WSTFES 2003: Proceedings of the IEEE Workshop on Software Technologies for Future Embedded Systems, Washington, DC, USA, p. 53. IEEE Computer Society, Los Alamitos (2003)

270

J.O. Filho et al.

4. Teker, U.: Realisierung und Evaluation eines Indoor-Lokalisierungssystems mittels WLAN. PhD thesis, Universit¨ at Bremen, Diploma thesis (2005), www.uni-bremen.de 5. Otsason, V., Varshavsky, A., LaMarca, A., de Lara, E.: Accurate gsm indoor localization. In: Beigl, M., Intille, S.S., Rekimoto, J., Tokuda, H. (eds.) UbiComp 2005. LNCS, vol. 3660, pp. 141–158. Springer, Heidelberg (2005) 6. Nilsson, M., Hallberg, J., Synnes, K.: Positioning with Bluetooth. In: 10th International Conference on Telecommunications ICT 2003, pp. 954–958 (2003) 7. Forno, F., Malnati, G., Portelli, G.: Design and implementation of a bluetooth ad hoc network for indoor positioning. IEE Proceedings - Software 152(5), 223–228 (2005) 8. Anastasi, G., Bandelloni, R., Conti, M., Delmastro, F., Gregori, E., Mainetto, G.: Experimenting an indoor bluetooth-based positioning service. In: ICDCSW 2003: Proceedings of the 23rd International Conference on Distributed Computing Systems, Washington, DC, USA, p. 480. IEEE Computer Society, Los Alamitos (2003) 9. di Flora, C., Ficco, M., Russo, S., Vecchio, V.: Indoor and outdoor location based services for portable wireless devices. In: ICDCSW 2005: Proceedings of the First International Workshop on Services and Infrastructure for the Ubiquitous and Mobile Internet (SIUMI), Washington, DC, USA, pp. 244–250. IEEE Computer Society, Los Alamitos (2005) 10. Madhavapeddy, A., Tse, A.: A study of bluetooth propagation using accurate indoor location mapping. In: Beigl, M., Intille, S.S., Rekimoto, J., Tokuda, H. (eds.) UbiComp 2005. LNCS, vol. 3660, pp. 105–122. Springer, Heidelberg (2005) 11. Wendlandt, K., Robertson, P., Berbig, M.: Indoor localization with probability density functions based on bluetooth. In: IEEE (ed.) PIMRC 2005, pp. 2040–2044. VDE Verlag (2005) 12. Subramanian, S.P., Sommer, J., Schmitt, S., Rosenstiel, W.: SBIL: Scalable indoor localization and navigation service. In: WCSN 2007: Third International Conference on Wireless Communication and Sensor Networks, pp. 27–30 (2007) 13. Bahl, P., Padmanabhan, V.N.: RADAR: An in-building RF-based user location and tracking system. In: INFOCOM, vol. (2), pp. 775–784 (2000) 14. Niculescu, D., Nath, B.: Vor base stations for indoor 802.11 positioning. In: MobiCom 2004: Proceedings of the 10th annual international conference on Mobile computing and networking, pp. 58–69. ACM Press, New York (2004) 15. G¨ unther, A., Hoene, C.: Measuring round trip times to determine the distance between wlan nodes. Networking, 768–779 (2005) 16. Vorst, P., Sommer, J., Hoene, C., Schneider, P., Weiss, C., Schairer, T., Rosenstiel, W., Zell, A., Carle, G.: Indoor positioning via three different rf technologies. In: 4th European Workshop on RFID Systems and Technologies (RFID SysTech 2008), Freiburg, Germany (2008) 17. Galstyan, A., Krishnamachari, B., Lerman, K., Pattem, S.: Distributed online localization in sensor networks using a moving target. In: IPSN 2004: Proceedings of the third international symposium on Information processing in sensor networks, pp. 61–70. ACM Press, New York (2004) 18. Guha, S., Murty, R., Sirer, E.G.: Sextant: a unified node and event localization framework using non-convex constraints. In: MobiHoc 2005: Proceedings of the 6th ACM international symposium on Mobile ad hoc networking and computing, pp. 205–216. ACM Press, New York (2005)

Penetration Testing of OPC as Part of Process Control Systems Maria B. Line1, Martin Gilje Jaatun1, Zi Bin Cheah2, A.B.M. Omar Faruk2, Håvard Husevåg Garnes3, and Petter Wedum3 1 SINTEF ICT, N-7465 Trondheim, Norway {maria.b.line,martin.g.jaatun}@sintef.no 2 Kungliga Tekniska Högskolan, Stockholm, Sweden {zbcheah,aofaruk}@kth.se 3 Google, Trondheim, Norway {hgarnes,wedum}@google.com

Abstract. We have performed penetration testing on OPC, which is a central component in process control systems on oil installations. We have shown how a malicious user with different privileges – outside the network, access to the signalling path and physical access to the OPC server – can fairly easily compromise the integrity, availability and confidentiality of the system. Our tentative tests demonstrate that full-scale penetration testing of process control systems in offshore installations is necessary in order to sensitise the oil and gas industry to the evolving threats. Keywords: Information Security, Process Control, Penetration Testing, OPC.

1 Introduction Process control systems (PCS) are used to support and control a specific industrial process. The setting for the kind of system discussed in this paper is oil production on offshore installations. Traditionally, hardware and software components in such systems were proprietary; designed for use in a specific context. The current trend is heading towards commercial off-the-shelf technologies because of the goal of integrated operations, which means extensive cooperation between onshore and offshore staff, and the possibility of controlling installations completely from onshore. This transition results in exposure to an entirely new set of security threats. Many components and applications in use were developed before this transition without any thought of making them intrusion-proof, because the access used to be more limited, both in the physical and the logical way. With respect to security, the ideal solution would be to build all parts of a process control system from scratch, taking all new threats into account. In reality this is not achievable; economically it would be a disaster for the operators, and a set of threats relevant today may change during the period of development. Also, there is the fact that new security flaws are usually introduced when developing new systems. Hence the brand new systems would not be perfect anyway. F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 271–283, 2008. © Springer-Verlag Berlin Heidelberg 2008

272

M.B. Line et al.

The more realistic solution is to add security functionality or security measures where they are needed. In order to identify where the needs are, several components and applications already in use must be analyzed in detail. This paper presents how a test network was set up to simulate a process control system, and how intrusion tests were performed towards OPC. The concept of OPC is described in the following section. 1.1 OPC – OLE for Process Control OLE1 for Process Control (OPC) [1] is a mechanism for exchanging process control data among numerous data sources in process control system. OPC is implemented as a client-server-architecture. The OPC server aggregates data from available control units in the system. It is capable of reading and writing values from/to these units, and offers mechanisms for interaction with these values to an OPC client. OPC is widely used in process control systems due to its stability and reliability. Also, it allows transmission of double precision real numbers, which other protocols usually do not allow without rewriting [2]. The OPC interface is defined on Microsoft’s OLE/COM (where COM stands for Component Object Model) [3], and most OPC servers run on MS Windows servers, even though there exist some implementations for other operating systems, such as OpenSCADA. The OPC interface is implemented as a set of function calls to COM objects defined by the OPC server, and can be accessed by DCOM (Distributed COM) from other computers. Hence, OPC does not implement security, but makes use of the security offered by DCOM. We have used OPC Tunneller and OPC Simulation Server from Matrikon2 in our simulated process control system. The reason for choosing Matrikon is that it is a freely available and fully functioning OPC-server for use in a non-production setting. The simulation server is normally used for testing OPC connections and services before production servers are put to use. It is therefore considered to be a suitable alternative for a real OPC server. 1.2 Penetration Testing Penetration testing is a way of evaluating the security of a system by simulating attacks by a malicious user against the system. Such testing can be performed in different ways; by running known vulnerability scanners like Nessus3 and Nmap4, or more as research and testing of new and unknown vulnerabilities. Nessus is well-suited for testing systems in production, and it is low-cost. It is continuously updated with new modules for newly discovered vulnerabilities, and running Nessus gives a good indication for how secure a system is, without giving a 100% complete result, which is quite impossible to achieve in any ways. Testing of unknown vulnerabilities requires extensive knowledge of the system in focus and is hence out of the question for our tests this time, as time and resources 1

OLE stands for Object Linking and Embedding. http://www.matrikonopc.com 3 http://www.nessus.org 4 http://insecure.org/nmap/ 2

Penetration Testing of OPC as Part of Process Control Systems

273

puts limitations on our work. In the future we might be able to move on to deeper analyses of OPC including testing of unknown vulnerabilities. 1.3 Our Tests in Brief Our aim is to show how a malicious user with varying privileges – outside the network, access to the signalling path and physical access to the OPC server – can fairly easily compromise the integrity, availability and confidentiality of the system. Resources available to us include a lab where we can set up a test network that simulates a process control system, and software freely available on the Internet. We define a blue team and a red team, where the blue team is responsible for setting up the test network and detecting intrusions, and the red team will act as malicious hackers and run different sets of exploits/attacks. 1.4 Paper Outline Section 2 describes how the test network was setup. Section 3 is about vulnerabilities that apply to OPC, which will be exploited. Section 4 describes the performed attacks together with achieved results, while section 5 discusses our findings. We offer concluding remarks and directions for further work in section 6.

2 Test Network Setup Blue team set up a simulated process control system in a test lab [4]. The topology reflects a typical offshore network inspired by the SeSa method [5]. The network is divided into three subnets, as follows: • DMZ subnet • Admin subnet; with an OPC client • Process Network subnet; with an OPC server DMZ is an abbreviation for demilitarized zone. In common computer networks, a DMZ contains an organization's services that are available for access from untrusted networks, typically the Internet. Services in a DMZ are usually needed by external hosts in order to communicate successfully with hosts inside the network and vice versa. The purpose of the DMZ is to add an additional layer of security to an organization's network by having services that are frequently accessed housed in this layer. The Admin layer models a typical process control system setup where hosts in the Admin Network are privileged users that usually have access to the Process Network layer. In order for users in the Admin network to access the Process Layer services, they should have the proper credentials to do so. For example, a manager in the Admin layer might use OPC client software such as a Matrikon OPC Client to access information from an OPC server (which is placed in the Process Layer). In order for the manager to successfully do this, he has to have the proper login/password credentials.

274

M.B. Line et al.

The Process Network layer is the deepest layer of the system and houses the most critical services in an offshore network. If an attacker manages to take control over this network, he or she has succeeded in compromising the entire network. The specifications of the computers used in the network are listed in Table 1 below, and the topology is illustrated in Fig. 1.

Fig. 1. The network topology as set up by the blue team Table 1. Computer hardware specification Host Name

Operating System

# NICs

Gateway Honeywall DMZ Gateway DMZ Host Admin Gateway OPC Server

Linux Fedora Core 6 Linux Fedora Core 6 Linux Fedora Core 6 Linux Fedora Core 6 Linux Fedora Core 6 Windows XP SP2

3 3 2 1 2 1


275

The honeywall and the router are for administrative and monitoring purposes. The OPC client is placed on the Admin subnet, while the OPC server resides on the process network subnet. 2.1 The Honeywall A honeypot is a host set up to detect, monitor and/or trap attackers, and a honeywall is a host that is usually placed between honeypot(s) and non-honeypot components in a network. It is regarded as a bridge between honeypots and regular networks segments. The two main tasks for a honeywall are traffic monitoring and intrusion prevention. Our honeywall has an IP address on only one interface, and we use the gateway to block anyone trying to access it. This interface is used as a remote management interface, while the other two interfaces (without IP addresses) act as the bridge. The bridge operates like a hub; passing traffic along from one interface to the next. As a honeywall, it also reads, copies, and analyzes all the passing traffic. A honeywall should be “non-existent” from a logical view. No one will know there is a honeywall host present unless that person enters the laboratory and sees our network. 2.2 The Gateway All traffic that enters or exits the network has to go through the gateway. If the gateway crashes, it means that the connection between our test network and the Internet is broken. In addition, the gateway performs firewall filtering and Network Address Translation (NAT). The purposes of the main gateway are: • • •

Internet entry-exit point Firewall packet filtering Firewall NAT

Our network provides an open port (21379) for the OPC client to connect to our OPC server using OPC tunnelling. Furthermore, the services SSH (port 22) and HTTPS (port 443) are also open for remote connection. An attacker might wish to exploit such open services and try to attack remotely from the Internet. We selected iptables5 as our choice of firewall implementation due to its flexibility. The firewall not only blocks traffic, but shapes traffic, too. Traffic can be denied or allowed based on port numbers, host names and IP addresses. For our case we use the filter table and NAT table in iptables. In the filter table of iptables, we used ALLOW ALL. Any traffic may come in and out of the network. The only exception is that we have configured our gateway to deny access to the private IP address of 10.0.0.X. The 10.0.0.X is the private network between the gateway and the honeywall; we want this subnet to be logically invisible so that no one can access this subnet. Usually, ICMP (Internet Control Message Protocol) is used to ping a host to determine if it is alive. We have configured the firewall to reply with “icmp port unreacheable” when anyone tries to ping the 10.0.0.X subnet. This will let others think that the subnet does not exist. The only way 5

http://www.netfilter.org

276

M.B. Line et al.

to access this subnet is via HTTPS from the Internet. In this case, the outside host will not know that the subnet exists because NAT has already translated the addressing. The NAT table in iptables allows us to forward traffic to certain ports to ports that we decide. Since we only have one public IP address, NAT directs the traffic to the appropriate destinations. In our configuration we forward traffic meant for the ports 3389 (RDP)6, 21379 (OPC Tunneller) and 135 (RPC)7 to the OPC server, and 443 (HTTPS)8 to the honeywall. 2.3 Intrusion Prevention We do not want an attacker that has successfully compromised our network to launch attacks from our network against others. However, we do not want to deny all outbound traffic, only malicious traffic. Outbound informational queries, such as ICMP ping, finger query, or a simple HTTP GET command should be allowed. To realize this we deploy a special snort function called snort_inline9 that is integrated in the Honeywall. This function lets us limit outbound, but welcome inbound, malicious traffic. One way to limit outbound traffic is to drop all outbound malicious packets; another is to limit the amount of outbound traffic per time unit – we decided to do both. Limiting outbound traffic is especially important regarding preventing Denial-ofService attacks from our network towards others. We do not set the outbound traffic to zero, as this will arouse the attacker's suspicion as to why there is no outbound connections allowed. 2.4 OPC By default, Matrikon OPC Tunneller does not use encryption or authentication to connect with another OPC Tunneller. We can, however, use a shared secret to encrypt and authenticate communication between client and server side OPC Tunnellers. This option was not employed in our tests. Throughout the penetration testing we have assumed that the IP-address of the OPC server is known to the attackers (red team). 2.5 Red Team’s Equipment Red team set up a Matrikon OPC Simulation Server with Windows XP SP2 and all available updates. The server was set up as described in the guide published by CERN [6]. The password for this server was known by the red team, as the login service was not to be tested. This server is in addition to the blue team simulation OPC-network. Besides this test server, red team operated with two regular computers and a switch that connected their units with “the Internet”, where “the Internet” is represented by a switched Ethernet LAN.

6

3389: Needed for remote desktop management of the OPC server. 21379 and 135: Needed to run the OPC server. 8 443: Needed for configuration of the honeywall via a browser. 9 http://snort-inline.sourceforge.net 7


277

3 Vulnerabilities in OPC In order to map vulnerabilities related to OPC, we have to consider vulnerabilities related to DCOM and RPC. This is because OPC is based on DCOM, which uses RPC, as mentioned in the Introduction. Frequently used vulnerability databases like the National Vulnerability Database (NVD)10, the Open Source Vulnerability Database (OSVD)11, and the database provided by United States Computer Emergency Readiness Team (US-CERT)12 reveal 555 vulnerabilities in RPC. 71 vulnerabilities related to DCOM are listed, and 40 OPC-specific vulnerabilities are reported. Here are some examples of vulnerabilities: − NETxAutomation NETxEIB OPC Server fails to properly validate OPC server handles − MS Windows RPC DCOM Interface Overflow − MS Windows DCOM RPC Object Identity Information Disclosure − MS Windows DNS RPC buffer overflow − MS Windows RPC service vulnerable to denial of service − MS Windows RPCSS Service contains heap overflow in DCOM request filename handling − MS Windows 2000 RPC Authentication Unspecified Information Disclosure These vulnerabilities open for buffer overflow, Denial-of-Service, and information disclosure; especially in the case of RPC. Carter et al [7] describe OPC-specific vulnerabilities in detail, and the main points of their discussion are summarized below. Lack of Authentication in OPC Server Browser: Configuration guidance from many vendors recommends allowing remote Anonymous Login so that OPCEnum will work when DCOM Authentication is sent to “None”. If a buffer overflow is discovered in the OPC Server Browser code, the result could be arbitrary code execution or Denial-of-Service attack against any computer running the OPC Server Browser. Fortunately, such an overflow has not been discovered yet. Lack of Integrity in OPC Communications: The default DCOM settings do not provide message integrity for OPC communication. If the underlying network is compromised and the attacker can sniff and insert traffic, it is likely that rogue messages could be injected once the client and server are authenticated during the initial connection establishment. A number of “Man-in-the-Middle” tools and techniques are available, and it is likely that these could be modified or enhanced to conduct attacks against OPC communication. Lack of Confidentiality in OPC Traffic: Although DCOM supports message encryption, most OPC vendors do not recommend enabling Packet Privacy for their OPC Server or the OPC Server Browser. Some vendors recommend VPN tunnelling as a means of providing secure remote access. Matrikon uses client- and server-side tunnelling component with encryption for this purpose. 10

http://nvd.nist.gov/ http://osvdb.org/ 12 http://www.kb.cert.org/vuls/ 11

278

M.B. Line et al.

In a different article, Lluis Mora [8] discusses the vulnerabilities of OPC servers. These include attacks using invalid server handle, invalid or crafted configuration file, resource starvation, etc. They have developed the tool “OPC Security Tester”/OPCTester13 for testing these vulnerabilities.

4 Exploits and Results Red team’s task was to act as attackers towards the test network and the OPC server specifically. Their initial knowledge included the IP address of the OPC server. In the following, the different types of mappings and attacks they performed are described, together with the results they achieved. More details about their work can be found in a separate report [9]. 4.1 Initial Network Mapping – Scanning and Probing the Network Many tools are available for gathering information about networks, operating systems and services offered by a network. Network administrators use such tools to manage network services, monitor hosts, service uptime etc. But as such tools are freely available; attackers can also use them for their own purposes. Nmap and Nessus can be used for scanning a process control system. Nessus also supports testing a list of vulnerabilities on a specific remote host. This can be thought of as the first step for any attacker in order to explore the list of services and ports exposed by a network. The introduction of a packet sniffer in the network clearly showed that in the case of setting the DCOM security level for the OPC server to "connect", all the OPC traffic is sent over the network in plain text, both from our test server and from the simulation network. Without any knowledge of the packet layout, we could still easily read out string values of the packets. Closer inspection and cross-checking with a proper OPC client lead us to also be able to identify other types of values in the packets, especially numerical values. This experience indicates that CERN does not value confidentiality of OPC data, as they have recommended this configuration setting. Further examinations of the network were done with the network mapper Nmap and the vulnerability scanner Nessus. Neither of these gave any indications of easily exploitable vulnerabilities in either our test server or in the test network. The information that was obtained was correct from both scanners on the operating system and other information known for our test server, and indicated Linux Fedora as the operating system running on the front end of the simulation network. However, these tools do not test OPC in particular, but rather the general setup of the servers with their operating systems and services. OPCTester performs automated tests of around 30 known vulnerabilities and access faults. We ran OPCTester to scan both the test server and the simulation network and did not get any useful results for either of the OPC servers. Run locally on our test server, OPCTester showed all available tests as passed without known vulnerabilities. 13

http://www.neutralbit.com/en/rd/opctest/


279

To sum up, network mapping did not yield any information on exploitable technical issues on the simulation network nor on our test server. Network sniffing did clearly show that the confidentiality of a standard set up OPC-server is void. 4.2 Entering the Signalling Path As “the Internet” in our setup is a switched Ethernet, we utilized the well known technique ARP-spoofing in order to be able to realistically perform network sniffing. We used the tool ARPspoof for this purpose, which is included in the Linux software package dsniff14. With ARPspoof we continuously sent out gratuitous ARP-packets to two given IP addresses informing them that the other IP-address is located at the attacker’s MAC address. This way all traffic between the two hosts is switched onto the attacker’s port on the switch, effectively putting us (the attacker) in the middle. 4.3 Packet Sniffing A network packet analyzer, e.g. Wireshark15, can be used to capture network packets and analyze them for future attacks. In the case of OPC traffic communication between the server and client side tunnellers is by default not encrypted. Information about OPC server and groups and items added or updated in OPC server can hence be read. A Man-in-the-Middle attack can be used due to the lack of access control mechanisms. An ARP spoofing attack allows an attacker to spoof the MAC address to sniff, modify, redirect or stop the traffic to the IP address of the target system. By entering the signalling path we were able to read the packets sent between the client and the server. By monitoring the authentication process of the client we were able to read in clear text the user name and machine name of the client and the machine name of the server. The client response to the server challenge includes both the NTLM version 1 response and the LM response. There is no need for both of the responses which only leads to further reduction of the security. Both of these two protocols use DES, which is known to be a weak encryption algorithm. NTLM version 2 has been accepted as a de-facto internet standard as of 2000 [10], is widely used, and is deemed much more secure than NTLM version 1. 4.4 Denial-of-Service Attacks We used SYN flooding and ARP spoofing to launch a Denial-of-Service attack. TCP uses three-way handshake (SYN, SYN-ACK, ACK) to establish a session. In the case of SYN flooding the attacker sends several packets, but does not send the last ACK back to the server, or the attacker spoofs the source IP address of the SYN message, server sends SYN-ACK to the false IP address and never receives the last ACK. ARP spoofing can stop the traffic to the spoofed target system. With the previously mentioned middleman-status we performed a Denial-of-Service attack by simply dropping all the packets destined for both the spoofed hosts, and thereby acting as a black hole in the signal path. This attack totally destroyed the communication between the test server and the client as expected. 14 15

http://www.monkey.org/~dugsong/dsniff/ http://www.wireshark.org/

280

M.B. Line et al.

A second successful Denial-of-Service attack was performed as a SYN-flood attack run by a single attacker. On the test server, this attack not only disabled incoming communication from a client, but also slowed down the system so much that the server was practically unresponsive during the attack. The effect of the attack lasted until about one minute after the attack was called off by the attacker. As this attack used fake source addresses in the packets, this attack can potentially be very difficult to distinguish from genuine heavy load. As seen, both a black-hole method and a SYN-flood method of attacks were able to destroy the availability of the test system; the SYN-flood even resulted in lost local availability. 4.5 Man-in-the-Middle Attack A tool was written to do string replacement and forwarding of packets in the ARP spoofed network setup between the OPC client and the OPC test server. As the recommended setup of DCOM did not seem to contain any packet integrity mechanisms, we expected this attack to be able to compromise the integrity of the entire OPC network setup.

Fig. 2. Man-in-the-middle attack on OPC; packets between server and client are modified


281

The tool we wrote simply replaced the string "Simulation" as sent from the OPC server A with the string "Real" and forwarded the modified packets to the OPC client B. Likewise the tool changed the string "Real" in the packets from the client to the server into the string "Simulation" and again forwarded these packets. Our tool was written as a proof-of-concept in the sense that the strings replaced were arbitrary. Any value and/or string can be replaced in the packets in the same way. The same tool can also be easily modified to shield the client from the server and in effect hijack the session. The resulting setup is described in Fig. 2. Table 2. Excerpt from printouts from OPC client from two different OPC sessions Without packet modification 2007-12-05 14:00:05,061 [main] DEBUG org.openscada.opc.lib.da. browser.BaseBrowser – Browsing with a batch size of 10 Leaf: Clients [Clients] Branch: Simulation Items Branch: Bucket Brigade Leaf: ArrayOfReal8 [Bucket Brigade.ArrayOfReal8] Leaf: ArrayOfString [Bucket Brigade.ArrayOfString]

Man-in-the-middle attack 2007-12-05 14:02:25,345 [main] DEBUG org.openscada.opc.lib.da. browser.BaseBrowser – Browsing with a batch size of 10 Leaf: Clients [Clients] Branch: Real Items Branch: Bucket Brigade Leaf: ArrayOfReal8 [Bucket Brigade.ArrayOfReal8] Leaf: ArrayOfString [Bucket Brigade.ArrayOfString]

As we can see from the sampled output in Table 2, we have a situation that to the eye looks genuine, but in reality is completely false. With the Man-in-the-Middle attack we were able to gain complete control of the system with the user privileges and access rights of the user in session. We were able to read, drop, change, and create packets as we saw fit in both directions, to the server and to the client. 4.6 Configuration Errors In the process of setting up the simulation network, the blue team installed an OPC Tunneller. As we connected to this OPC Tunneller, there were no access control mechanisms in place se we had complete access to the OPC server in the simulation network. The blue team also gave us instructions on how to set up access control on the OPC client, but these instructions led to the setup screen for the Windows XP DCOM run-as-user configuration management. We assume the instructions for the server setup was misunderstood such that the blue team in reality had set their OPC Tunneller to run as a specific user on the server rather than setting access control credentials needed for a client to log onto it. We can see that a small misunderstanding in the setup of the server led the red team, and in reality anyone, to take complete control of the simulation OPC server while the blue team was certain that access control was in place.

282

M.B. Line et al.

5 Discussion Process control networks in the oil and gas industry have traditionally had a very strong focus on safety, but safety-related risks are quite different from security-related risks. One way of differentiating between security and safety is to say that safety consists of protecting the world from the system, while security is protecting the system from the world [11]. Another aspect is that safety-related incidents normally have clearly defined statistical properties that relate to empirically established values for e.g. Mean Time Between Failures (MTBF) for components and systems. The same thing cannot be said for security-related incidents, since they predominately involve a human attacker – and we have yet to develop a useful statistical model for Homo Sapiens. In the early days of the internet, there was a Usenet newsgroup known as alt.folklore.urban16, devoted to the topic of urban legends. A recurring joke (or mantra) in this newsgroup was: “It could have happened, so it must be true.” Ludicrous as this statement may be in “real life,” this is exactly how we must think in the realm of computer security. The professional paranoia this reflects also testifies to a distinct difference in mindset compared to the safety domain. Our main contribution in this paper is to highlight the “could have happened” aspect of computer security in the Integrated Operations context. We have no empirical evidence that the specific attacks in our laboratory tests have been successfully wielded in a real PCS environment, although there are certainly news reports of other attacks against PCS’s in the past [12]. In our contact with representatives from the Norwegian oil & gas industry, we have been confronted with the sentiment “We don’t care what they say they can do over in the States; unless you can show us something that is directly applicable to our setup, we’re not interested.” This tells us that there is a need for full-scale penetration testing activities – and our tests indicate that such activities would yield interesting results. The most conclusive security breach that we successfully performed was achieved due to configuration error, which may at first glance seem to be an insignificant result. However, it is clear that the human factor plays a major role in the defence of systems as well as in their attack, and the safety concept of “fail safe” should clearly be applied to security settings, It may be argued that also the other attacks successfully implemented in our tests have limited value, since they all require physical access to the communications medium on some level, and these networks are all protected by multiple layers of firewalls and other protective mechanisms [5]. However, previous breaches tell us that not only is the insider threat an important factor in PCS environments [12], but the increasing integration of offshore and land-based networks also mean that the attack surface is increasing. This, in turn, implies that it is important to follow the doctrine of defence in depth, ensuring that there are sufficient (and independent) security mechanisms every step of the way from the external barrier to the heart of the PCS. Clearly, a protocol that uses outdated cryptography for authentication, and transmits data in plain text is not a good match for our doctrine. It is our opinion that the OPC implementation should implement NTLM version 2 or other more secure authentication protocols like Kerberos. 16

It’s still out there – see http://tafkac.org/


283

6 Conclusion We have shown that confidentiality in OPC networks is non-existent in the default setup as recommended by CERN. Furthermore, the authentication process of the client reveals confidential information in clear text and gives a weak encryption of the client password. We have seen that DoS attacks can easily be accomplished, not only in making the server unavailable over the network, but also leading to denial of local control over the server. We have demonstrated a to the eye perfect breach of the integrity of the OPC server as a consequence of lacking DCOM packet integrity measures in the setup recommended by CERN, and we have demonstrated how fragile an OPC network is to configuration errors.

Acknowledgements The research presented in this paper was performed as part of project assignments at the Norwegian University of Science and Technology (NTNU).

References 1. OPC Overview 1.0, OPC Foundation 2008-01-17 (1998), http://www.opcfoundation.org/Downloads. aspx?CM=1&CN=KEY&CI=282 2. Understanding OPC and How it is deployed, Byres Research 2008-01-17 (2007), http://csrp.inl.gov/Recommended_Practices.html 3. DCOM Technical Overview, Microsoft Developer Network 2008-01-17 (1996), http://msdn2.microsoft.com/en-us/library/ms809340.aspx 4. Cheah, Z.b., Faruk, A.B.M.O.: Identifying and Responding to External Threats in a PCS Network. Norwegian University of Science and Technology Project Assignment, Trondheim (December 2007), http://sislab.no/blueteam.pdf 5. Grøtan, T.O., et al.: The SeSa Method for Assessing Secure Remote Access to Safety Instrumented Systems, SINTEF Report A1626, Trondheim (June 2007), http://www.sintef.no/content/page1_16321.aspx 6. Puget, M.B.J.-P., Barillere, R.: IT-CO recommended DCOM settings for OPC, CERN, Geneva (2005) 7. Carter, J., et al.: OPC Security. Digital Bond (2007) 8. Mora, L.: OPC Server Security Considerations. In: SCADA Security Scientific Symposium 2007, Miami, FL (2007) 9. Garnes, H.H., Wedum, P.: Innbruddstesting på prosesskontrollsystemer på oljeplattform, Norwegian University of Science and Technology Project Assignment, Trondheim (December 2007) 10. Zorn, G.: RFC 2759: Microsoft PPP CHAP Extensions, Version 2, The Internet Society (2000) 11. Line, M.B., et al.: Safety vs security? In: Eighth International Conference on Probabilistic Safety Assessment and Management, New Orleans, USA (2006) 12. GAO-07-1036 Critical Infrastructure Protection: Multiple Efforts to Secure Control Systems Are Under Way, but Challenges Remain, United States Government Accountability Office (2007), http://www.gao.gov/htext/d071036.html

Intersection Location Service for Vehicular Ad Hoc Networks with Cars in Manhattan Style Movement Patterns Yao-Jen Chang and Shang-Yao Wu Department of Electronics Engineering Chung Yuan Christian University Taoyuan, Taiwan 320 {yjchang,g9476025}@cycu.edu.tw

Abstract. For inter-vehicular communications that happen in metropolitan areas, moving cars travel on restricted directions along streets and make frequent stops over intersections. In this paper, we present Intersection Location Service (ILS), a distributed hashing-based location service algorithm that makes use of the features of street intersections and the Chord algorithm as the location query and fault tolerant mechanism. Performance comparisons of ILS with two well-known location service algorithms, Grid Location Service (GLS) and Hierarchical Location Service (HLS) are demonstrated in the ns-2 simulations of moving cars in various city environments. We have shown by means of simulation that ILS achieves good results in terms of query success ratios and remarkable scalability with respect to network size.

1 Introduction We assume that the nodes are able to determine their own position, e.g. by means of GPS. The task of locating the destination is a communication session is then fulfilled by a location service, which employs a distributed algorithm maintained by all the participating nodes in the Vehicular Ad Hoc Networks (VANETs). Most research results of location services have been focused on nodes whose motions follow patterns of random waypoints. However, we are looking at some protocol that introduces novel mechanisms to exploit the features of the urban scenario affecting the vehicular mobility such as the presence of intersections. For car-to-car communications that occur in metropolitan areas, moving cars obviously travel on restricted directions along streets and make frequent stops over intersections to gain right of ways according to local traffic rules. Several location services protocols including DREAM [10], Homezone [11] and GHLS [8] have been proposed in literature, by exploiting different approach for tracking the nodes' movements such as flooding-based, rendezvous-based, and hash-based. The solution proposed in the paper, called Intersection Location Service, is a variant of the hash-based approach. As main difference from the other schemes proposed in literature, the ILS algorithm is tailored for a vehicular environment. It does not use F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 284–296, 2008. © Springer-Verlag Berlin Heidelberg 2008

Intersection Location Service for Vehicular Ad Hoc Networks with Cars

285

flooding of information, and it includes fault-tolerant mechanisms. In the ILS scheme, each node is assigned a single location server, by hashing the node's unique identifier into an intersection identifier. Only vehicles moving in the intersections act as location servers, although every node is assumed to be capable of relaying packets. The duties of cars at temporarily empty intersections are transferred to cars at other nonempty intersections according to the proposed scheme. The location information cached by cars at a previously nonempty intersection is taken care of in a way that the subsequent queries on such information can be directed to the cars at a certain nonempty intersection that now takes over the responsibilities as location servers. The simulation analysis confirms the effectiveness of the ILS scheme in terms of query success ratios under multiple network scenarios. In addition, ILS performs better in street environments than two other well known location service protocols (GLS [7] and HLS [5]) designed without taking into account street environments. The remainder of this paper is structured as follows: In Section 2 we give an overview of related work. The ILS model is described in detail in Section 3. In Section 4 we present how the Chord algorithm is applied in the protocol of ILS. Section 5 contains the results of the simulation with ns-2. A comparison with two select location algorithms is also presented. The paper is concluded with a summary and future work in Section 6.

2 Related Work DREAM, the Distance Routing Effect Algorithm for Mobility [10], is a location service which uses flooding to spread position information. The capacity of the network is substantially decreased as a result of message flooding, especially in situations where the nodes make frequent position changes. A location algorithm that does not require flooding is Homezone [11]. Each node is assigned an area called Homezone via a hash function well known to all the nodes in the network. Location updates that reflect the changes nodes make to positions are sent to all the nodes in the designated homezone. Problems may arise when the homezone becomes empty, and some cars may get into situations where their default location servers are temporarily unavailable. The Grid Location Service (GLS) [7] divides the area of the entire ad hoc network into a hierarchy of squares forming a quad-tree. Each node selects one node in each element of every levels of the quad-tree as a location server. Therefore, the density of location servers for a particular node is high in areas close to the node and become exponentially sparse as the distance to the node increases. GLS has demonstrated good scalability in the random waypoint mobility model. Hierarchy Location Service (HLS) [5] partitions the area of the ad hoc network into cells. The cells are grouped into regions level by level. Therefore, a node uses a hierarchy of location servers to store its location information. The problem of empty cells where location updates or requests may run into is solved by temporary servers through a handover mechanism to bridge the gap. DREAM, Homezone, GHLS, GLS, and HLS are all designed with random waypoint motions in mind. The location services apply in MANETs where nodes can change directions at will; there are no fixed pathways for mobile nodes to follow.

286

Y.-J. Chang and S.-Y. Wu

However, there are increased needs to provide location services for MANETs that consist of vehicles moving along streets within metropolitan areas. ILS is proposed in this paper to address those needs. HLS and GLS, although not designed for street conditions, are selected for comparisons due to their good performance results for nodes in random waypoint motions.

3 Sketch of Intersection Location Service Protocol In this section, we briefly sketch the basic underlying idea of ILS, which will be detailed in the next section. In ILS, the street intersections covered by the network is numbered from 0 to N. In the ITS scheme, each node is assigned a single location server, by hashing the node's unique identifier into an intersection identifier. Cars within the radius r of the intersection M cache the location information of all the vehicles K for which M=Hash(K) where Hash is a hash function that can distribute cars evenly among the intersections. In an ideal case where all the street intersections are nonempty, the queries of location information about Car K where Hash(K)=M will be directed to Intersection M, the default location server for Car K. Due to the highly dynamic nature of VANETs, it is possible that some intersections are empty for certain periods of time. In case there is no designs of fault tolerance, location queries that are directed to temporarily empty intersections will lead to failures. A priori selection of backup location servers is of little use because any intersection that acts as a location server also suffers from becoming temporarily empty. Using the Chord algorithm [1,9]. Once a location server is determined for a certain query, the query issued by a car will be routed to the target location server with a position-based routing protocol like GPSR [4]. The position-based routing in ILS is done by the cars that serve as intermediate relays to overcome the multiple-hop delivery problems typical in MANETs.

4 Protocol of Intersection Location Service This section describes the ILS protocol which specifies how to find the locations of cars moving along the streets, how the unavailability of intersections can be overcome with relatively small overhead. 4.1 Consistent Hashing The consistent hash function assigns each intersection an m-bit identifier using SHA-1 [2] as a base hash function. Vehicles moving in the radius r of intersections are responsible as location servers to cars by caching their location information. Consistent hashing assigns Car K to a default location server M if M=SHA1(K), where SHA1 stands for SHA-1. However, a location server M can be nonexistent in the 2m space or M goes out of service because the intersection has no cars for a period of time and none of the location information can be cached in any one of the cars within the radius r of the center of intersection M. Therefore, a successor location server for car K, denoted as successor(K), is chosen to be the first location server whose identifier is equal to or follows SHA1(K). If identifiers are represented as an imaginary circle of

Intersection Location Service for Vehicular Ad Hoc Networks with Cars

287

numbers from 0 to 2m -1, then the successor(K) is the first node clockwise from M if M=SHA1(K). The circle is referred to as the Chord ring [1,9]. As an illustrative example, Fig. 1 shows a Chord ring with m=3. The Chord ring has 3 location servers and 4 cars. The successor of car 2 is location server 3, so location server 3 will store location information about car 2 while location server 2 is out of service.

7 7 3 6 2

5 4

Fig. 1. A Chord ring with 4 nodes and 3 intersections where cars with radius r act as location servers

4.2 Query for Car Location When a car requests for location service to acquire the location of other cars before car-to-car communication can be set up, it first submits its query to a nearby intersection. ILS is based on the Chord protocol that works by pruning candidate location servers by half at each iteration and the binary search continues until the exact location server is determined. It is not without price; ILS maintains additional information to facilitate binary search. Additional information is kept by each location server in a data structure, called the finger table [1,9]. As before, let m be the number of bits in the location server identifiers. A location server M maintains its finger table with up to m entries. The i-th entry in the table at location server M contains the identity of the first node S that succeeds M by at least 2i-1 on the identifier circle, i.e., S = successor(M+2i-1), where 1≤i≤m and all arithmetic is modulo 2 m. S is called the i-th finger of M, and is denoted by M.finger[i]. A finger table entry includes both the identifier of location server and its coordinates. Note that the first finger of M is the immediate successor of M on the circle and is referred to as the successor for convenience. Figure 2 is an example of finger tables maintained by vehicles with radius r of nonempty intersections. For a network with N location servers, the finger table at each location server has O(Log(N)) entries relating to information about only a small number of other location servers. The Chord algorithm to find a successor with the use of finger tables operates is as follows.

288

Y.-J. Chang and S.-Y. Wu

Finger 7+2i-1 =8, i =1 7+2i-1 =9, i =2 7+2i-1 =11, i =3

Finger 6+2i-1 =7, i =1 6+2i-1 =8, i =2 6+2i-1 =10, i =3

Successor 1 3 3

Successor 7 1 3

7

1

6

3

Finger 1+2i-1 =2, i =1 1+2i-1 =3, i =2 1+2i-1 =5, i =3

Successor 3 3 6

Finger 3+2i-1 =4, i =1 3+2i-1 =5, i =2 3+2i-1 =7, i =3

Successor 6 6 7

Fig. 2. Finger tables as cached by vehicles within the radius r of intersections

//For a location server M to find the successor of car K M.find_successor(K) { if( M <event name="go to movie theater by subway" /> <event name="watching movie" /> <event name="take a bus" /> <event name="shopping in mart" /> Fig. 7. XML representation for story

544

Y.-S. Lee and S.-B. Cho Table 5. An example of cartoons generated

Going to Movie

Watching movie

Taking a bus

Shopping

4 Experimental Results 4.1 Environment In this paper, we use landmarks and mobile contexts collected from two female undergraduate students for two weeks [15]. They are used to select episodes and generate a story. Table 6 shows the summary of topics in 32 episodes. If prior episode and posterior episode are the same, a story includes the episode only one time and increases the influence of result of episodes. Also, a series of episodes without result are ignored. Subjects bring with smart phone connected to GPS device. They report visited places, activities, and emotion as the summary of daily life. Table 6. Topics of Episodes used in experiment

No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Name Viewing Using means of transportation Traffic jam Study at a late hour Dance hall Meeting Taking a photo of goods Take a walk Washing up Take a photo of myself Shopping Taking lessons Spam SMS Meal Indoor exercise Eating out

No. 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

Name Cooking Exercise Taking a photo of food Study while listening to music Using subway Doze Taking a photo with joy Conversation by telephone Annoyance SMS Having tea Using computer Scenery place Take a photo of scene Leisure hours Alone Study alone

Petri Net-Based Episode Detection and Story Generation

545

4.2 Results Table 7 is comparison between subject’s report and generated story on a specific day. That day, the subject takes an express bus and went to hometown. Time, place, and experience are extracted from the subject’s report. It shows generated cartoons and corresponding experience. Most trivial events such as conversation by telephone, short-message-service and listening music are not included in the user’s report. Subjects tend to ignore them. They have more interests about event related to user’s activity or emotion. Table 7. Comparison between report and cartoon

Time 00:40 12:00 16:30 17:10

Place Home Home Bus station 1 Restaurant

18:30

Bus station 2

19:00

Subway station 1

19:50

Experience Go to bed Wake up Go to restaurant by bus Meal with family

Generated Cartoon -

Go to subway station

Go to station near terminal Arrive at the subway station Subway station 2 2

20:05

Express terminal Go to hometown by express

20:40

Express terminal

Traffic jam

22:20

Hometown

Arrive at hometown

-

-

546

Y.-S. Lee and S.-B. Cho

On the whole, generated cartoons are fewer than events in the report. Generated cartoons miss many events in report. It is due to insufficient contexts and lack of diversity of events in episodes. If user adaptive episodes are included, generated story has more events related to report and mirrors user’s daily life better. We calculate precision for checking coincidence between cartoon and user report. Cartoon consists of character’s activity and background place. If both activity and place of cartoon are coincident with the report, accuracies of the cartoon are evaluated as 1 point. If only one activity or place is accurate, we consider its accuracy as 0.5 point. precision = sum of accuracy of cartoons / number of cartoons Subject 1’s precision is 27% and subject 2’s precision is 43%. If meaningless events such as SMS or telephone call are included, the precisions increase to 59% and 67%, respectively.

5 Conclusion and Future Work In this paper, we attempt to create story and generate cartoons from mobile contexts. Story is a sequence of events with temporal or causal relationship. We propose a method that selects episodes and organizes them to a story. The experiments show the potential of the proposed method to generate story in real life log. Generated cartoons are compared with user’s report in precision. The method needs further improvements such as increase of types of context, accuracy of landmark detection, and the number of episodes supported. Also, we would enhance the episode composition algorithm. More contexts and episodes will improve the precision and diversity of generated cartoons. If user can design his own episode, it can help to generate a story appropriate to user’s preference. Finally, user modeling can also help user’s story.

Acknowledgements This research was supported by MKE, Korea under ITRC IITA-2008-(C1090-08010046).

References 1. Raento, M., Oulasvirta, A., Petit, R., Toivonen, H.: ContextPhone - A prototyping platform for context-aware mobile applications. IEEE Pervasive Computing 4(2), 51–59 (2005) 2. Panu, K., Jani, M., Juha, K., Heikki, K., Esko-Juhani, M.: Managing context information in mobile devices. IEEE Pervasive Computing 2(3), 42–51 (2003) 3. Aizawa, K., Hori, T.: Context-based video retrieval system for the life-log applications. In: Proc. of MIR 2003, pp. 31–38. ACM, New York (2003) 4. Gemmell, J., Bell, G., Lueder, R.: MyLifeBits: A personal database for everything. communications of the ACM 49(1), 88–95 (2006)

Petri Net-Based Episode Detection and Story Generation

547

5. Smeaton, A.F., O’Connor, N., Jones, G., Gaughan, G., Lee, H., Gurrin, C.: SenseCam visual diaries generating memories for life, Poster presented at the Memories for Life Colloquium 2006 (2006) 6. Cho, S.-B., Kim, K.-J., Hwang, K.-S., Song, I.-J.: AniDiary: Daily cartoon-style diary exploits Bayesian networks. IEEE Pervasive Computing 6(3), 66–75 (2007) 7. Korpipää, P., Mantyjarvi, J., Kela, J., Keranen, H., Malm, E.-J.: Managing context information in mobile devices. IEEE Pervasive Computing 2(3), 42–51 (2003) 8. Mantyjarvi, J., Seppanen, T.: Adapting application in handheld devices using fuzzy context representation. Interacting with Computers 15(4), 521–538 (2003) 9. Siewiorek, D., Smailagic, A., Furukawa, J., Krause, A., Moraveji, N., Reiger, K., Shaffer, J., Wong, F.: Sensay: A context-aware mobile phone. In: IEEE International Symposium on Wearable Computers, pp. 248–249 (2003) 10. Choi, K.-S.: CoreNet: Chinese-Japanese-Korean wordnet with shared semantic hierarchy. Natural Language Processing and Knowledge Engineering, 767–770 (2003) 11. Peterson, L.J.: Petri Net Theory and The Modeling of Systems. Prentice-Hall (1981) 12. Propp, V.: Morphology of the Folktale, University of Texas Press (1968) 13. Correira, A.: Computing story trees. American Journal of Computational Linguistics 6(34), 135–149 (1980) 14. Riedl, O.M., Young, M.R.: From linear story generation to branching story graphs. IEEE Journal of Computer Graphics and Animation 26(3), 23–31 (2006) 15. Hwang, K.-S., Cho, S.-B.: Modular Bayesian networks for inferring landmarks on mobile daily life. In: The 19th Australian Joint Conference on Artificial Intelligence, pp. 929–933 (2006)

Protection Techniques of Secret Information in Non-tamper Proof Devices of Smart Home Network Abedelaziz Mohaisen1 , YoungJae Maeng2 , Joenil Kang2 , DaeHun Nyang2 , KyungHee Lee3 , Dowon Hong1 , and JongWook Han1 1

3

Electronics and Telecommunication Research Institute, Daejeon 305-700, Korea {a.mohaisen,dwhong,hanjw}@etri.re.kr 2 Information Security Research Laboratory, INHA University, Incheon, Korea {brendig,dreamx}@seclab.inha.ac.kr, [email protected] Electrical and Computer Engineering Department, Suwon University, Suwon, Korea [email protected]

Abstract. The problem of revealing secret information in home network becomes critical when a physical capture of single device or more happen where the attacker can statically analyze the entire device’s memory. While the trusted platform module that assumes a tamper proof chip in each device is not a choice, we investigate other software-based solutions. This paper introduces several mechanisms and schemes in varying scenarios for secret information protection in non-tamper proof devices of the smart home environment. The mechanisms provided herein utilize the existence of several algorithms and techniques and building blocks that do not require an extra hardware while they are computation efficient on the typical home network devices. To demonstrate the value of contributions, an extensive analysis for different scenarios including security and cost estimations are provided. Keywords: home network, secret information, authentication, code attestation, secret sharing, security and system integration.

1

Introduction

Home Network (HN) is composed of diversity of devices which have the ability to communicate with each other internally and with other devices in other networks externally through a control point (CP). In the communication-enabled HN devices, the user’s input affects devices behavior in the HN model for ease of accessibility and to provide a living-convenient environment [1]. In such environment, secret information stored and maintained in each device is used for several security purposes including device authentication, encryption and/or decryption of the internally or externally exchanged data via the CP. Generally speaking, revealing the secret information in the HN may include critical damages and threats for the privacy of part or the whole of the network. Thus, HN designers and operators carefully need to maintain the stored secret information in HN F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 548–562, 2008. c Springer-Verlag Berlin Heidelberg 2008

Protection Techniques of Secret Information

549

devices under several expected and unexpected attacking scenarios including the physical attack. One of the solutions to prevent physical attack is performed using the Trusted Platform Module (TPM) [2]. The usage of the TPM requires integrated hardware and operating system’s supports which both complicate the overall design and structure of the HN device. Technically, once the TPM is used on an HN device the device price will be greatly increased. In addition, even though the TPM provides no chance for any successful malicious attack to be performed, it is not always guaranteed and sometimes undesirable to provide a TPM for each single device in an HN with diversity of devices of ranging prices and capabilities. Based on that, we assume the TPM is not an option in HN environment. In an HN with non-TPM devices, an attacker does not have any difficulty to extract secret information from these devices. Even though secret information is encrypted and the key for decryption is stored outside of the device, once we the key stored externally needed to be restored for a decryption operation, the device itself need to be authenticated by the external key storing device. To overcome this reiterated problem, as the TPM scenario is out of question, we introduce a solution based on secure cryptographic functions which are already in use. In this paper, we introduce a scheme to protect secret information in a software fashion. We revise several existing building blocks (such like RSA, secret sharing, code attestation) for protecting the secret information without any additional hardware components. Our schemes utilize these blocks in cooperation with other novel schemes as well and show a satisfactory performance and resources consumption as will be shown. On the side of the paper structure, section 2 introduces the assumptions, terms elucidations and contribution followed by our scheme in section 3 and section 4. A comparison between our introduces schemes demonstrating their advantages and limits followed by a hybrid scheme that merges both of the schemes is shown in section 5. Security evaluation and cost estimation are discussed in section 6 Finally, we conclude by concluding remarks in section 7.

2 2.1

Definitions, Assumptions and Contributions Definitions and Building Blocks

Definition 1 (Secret Information - SI). is a manipulated data that provides benefit to someone where revealing this data may cause damage to its owner. Also, the indirect influence of the secret information exists where it is difficult to identify the scale of potential benefit or damage. As an example, a party’s communication leads other party’s benefit or damage [3] therefore the exchanged data between parties is secret information at most. SI in this paper means information which has a direct impact on some entity when exposed. This SI can be any of the following: a private key which is used for public key infrastructure, secret key of symmetric cryptography, bio-informatics

550

A. Mohaisen et al.

key, or authentication key for a HN’s device. Certainly, valuable information for only its existence such like private information is also SI. Definition 2 (Software-Based protection [4]). is a method used for twisting the the secret information by modifying or manipulating this information through statically changed software. In untrusted environment, an attacker can physical capture a device with full control of its resources including the memory and thus reveal any SI. Based on that, we cannot protect SI statically stored in local memory under software-based method [4]. For the same reason, we extend software-based method in which not only the locally stored information on the same device but also other networks components may affect the overall security of the locally stored SI based on the network status. For example, a device may delete the SI to minimize the possible damage if it is informed of a possible attack. Definition 3 (RSA Cryptosystem [5]). In RSA, let p and q be two large enough prime numbers, n = pq and φ(n) = (p − 1)(q − 1), SKc is chosen such that 1 < PKc < φ(n) and SKc is chosen to satisfy SKc PKc ≡ 1 mod (φ(n)) where SKc is CP’s private key and PKc is CP’s public key. Definition 4 (Secret Sharing [6]). Shamir’s secret sharing is defined as follows: Let f (x) be a polynomial of degree t − 1 defined as f (x) = s0 + a1 x1 + a2 x2 + · · · + at−1 xt−1 where s0 is the initial secret. Partial secrets are computed as f (s1 ), . . . , f (sn ) and distributed for different parties. The recovery of the initial secret s0 (and therefore the polynomial f (x)) requires a collusion between a group of n nodes where n ≥ t. That is, f (x) can be reconstructed by the linear regression iff (if and only if ) the number of linearly independent components from the construction of f (x) known to a single user is as big as t or greater. Definition 5 (Diffie-Hellman key exchange [7]). Let g ∈ Zp be a generator where p is a large enough prime. A shared key between two parties P1 and P2 is established according to the following interaction: P1 g, p a ← Zp A = g a mod p

P2 g, p b ← Zp B = g b mod p A

−→ B

←− KA = B a mod p

KB = Ab mod p

Note that KA = KB = (A)b = (g a )b = (B)a = (g b )a = g ab mod p. Definition 6 (Code Attestation). A code is verified in code attestation by comparing the code (or representative code of the code) to other copy on a trusted party. That is performed through the CP by sending an attestation command that enforce the execution of attestation code on the remote device. Examples of these attestation methods are detailed in [8,9,10].


2.2

551

Assumption

Throughout this paper, the following assumptions are used. Note that these assumptions are based on the architecture of home network in Fig. 1 representing the part of unit household. 1) The devices in the home network can communicate between each other and pass operation commands to devices within the same network or other devices that are in another network. The communication with devices in other networks is carried out through the control point. The control point (CP) is a special device (or combination of devices) which is connected community network via a 10/100 Ethernet infrastructure (see Fig. 1) 1 [11,1]. 2) The different devices within the same HN can communicate between each other in a single hop communication method without getting through the CP. Thus, modifying messages between two communicating parties is hard to be carried out unless one of the two parties is compromised. This assumption is rationalized according to several other networks when communicating in a single hop (e.g., wireless sensor networks [12]). 3) An adversary needs at least a period of time longer than a threshold (say, τ seconds) to compromise a device successfully [13]. Also, to analyze the contents of physically compromised HN device, the adversary need to perform this out of the scope of the HN itself (offline analysis). 4) In most cases, the CP is trustworthy but it does not hold any secret information since we cannot guarantee the CP’s safety against unknown attacks. This makes the information secure even when the CP is compromised by an attacker2. 5) The communication traffic between two devices is encrypted using session key which is obtained by Diffie-Hellman Key Exchange (def. 5) except for few specific messages as will be shown later. 6) The CP and other devices have certificates based on RSA public key cryptosystem (def. 3). 2.3

Contributions

We investigate SI protection schemes for HN considering devices without TPM. Our contribution includes four parts which are detailed as follows – First, we introduce a novel scheme based in which CP’s broadcast periodical messages to identify the different devices in the network as safe devices or not (based on whether the network is under attack or not). – Second, we introduce a scheme based on Shamir’s secret sharing [6] and the code attestation method [9] combining their security levels for distributing SI into shares among different HN devices and authenticating each share in the different devices by authenticating the holding device itself using the code attestation method [9]. 1 2

The CP in our case is a server connected to a gateway. Indeed, the compromise of the CP will affect the communication links to other devices in other networks passed through the CP.

552

A. Mohaisen et al.

Community network

PC

Mobile

Interface

Community common section

Streaming SVR

Network Camera

Community health care solution

Unit household

HomeNet PDP

Door security camera

DTV portal HA System

10/100 Ethernet Community network

Community healthcare solution Iris recognition/ Common entrance Management office Parking management

Community server

HomeNet Appliances

Others!

Automatic door IP Phone Remote Unmanned meter home reading delievery

Sensors

Gateway HomeNet Server Magic Mirror

HomeNet Pad (wireless)

Others Household health checker (wireless)

Emergency Solutions

Fig. 1. The typical architecture of home network representing the premium HomeNet solution from LG ElectrnonicsTM . The control point (CP) represents the combination of the HomeNet server and the gateway. Our proposed schemes are concerned with the unit household part.

– Third, as several advantages in each scheme comes out over the other scheme, we introduce a comparison between the two proposed schemes considering their different characteristics, advantages and disadvantages. – Fourth and last, we merge both of the schemes mentioned above to provide a hybrid scheme to enhance the overall security of SI protection and introduce a scheme that uses both schemes merit to overcome their limits.

3

Safe Umbrella: Time-Based Scheme

The safe umbrella is our first method to protect the SI. When using the safe umbrella for protecting SI, we store the SI in volatile memory (VM) in each device (or a group of devices) in the network. Keeping SI in the memory is determined depending on the status of the device and the network with respect to attacking-probability. That means, a device erases the SI stored in its own volatile memory when it detects a possible critical attack (e.g. if the device is informed that a number of compromised devices is greater than a threshold value in a trustworthy manner). However, it is not easy to discover attacker’s malicious behavior. That is, even though other devices are under attack some other devices might be unaware of this possible attack. 3.1

Overview

In the safe umbrella, we assume that each device can be informed about the network status by receiving regular message. To perform this, two scenarios are introduced. In the first scenario, a message that reports the existence of a possible attacker is generated by an HN device that informs each device to delete SI which

Protection Techniques of Secret Information SGM HN Devices

HN Devices

Control Point

SGM

553

Fig. 2. Illustration of safe umbrella’s

is stored in the corresponding device’s memory. In the second scenario, the CP periodically broadcasts messages to inform the devices in the network about the status of the network that determines whether or not there are attackers in the network. Obviously, the first scenario is cheatable since a device under control may deviate in sending the proper signal for other devices. Other than that, there is a high probability that a device under attack cannot send the warning message on time. Thus, we use the second scenario of the solution for estimating and examining the conditions of the network. In this scenario, we assume that the CP periodically broadcasts Safe Guarantee Message (SGM) determining the status of the network as shown in Fig. 2. Upon receiving the SGM, the different devices in the network check safety of the network indicated by checking the SGM following one of these scenarios: – Each device deletes the SI from its volatile memory when the received SGM is not correct. – Each device may delete the SI according to exception error generated from the execution of codes (except the case of errors related to hardware configurations). In this case, the different devices should be provided the ability to distinguish between the exception errors and relate them either to the attack possibility (malicious code running) or to normal running error (hardware exceptions). To determine the status of the network, a time constrained verification process is held in what we call the “signed time-stamp” phase which is shown in the following section. 3.2

Signed Time-Stamp

Each device in the network can confirm its security status by the time synchronization for signed time-stamp broadcast reply. This procedure is illustrated as follows: 1. CP broadcasts signed time-stamp tcenter signed by CP’s private key to devices that CP is controlling within an estimated attack time τ where this τ period helps each device to decide whether it received the message or not. This part of procedure is performed as follows: broadcast

CP −→ : DSKc (tcenter )

554

A. Mohaisen et al.

where DSKc (tcenter ) is the signed (tcenter ) using the CP’s private key and the decryption algorithm D. 2. Each device checks the received time stamp tcenter – which is decrypted by CP’s public key – by comparing the decrypted one with device’s own timestamp (tdevice ) as follows (before all, the message is ignored if the decrypted tcenter is not a valid type of time stamp): 1: 2: 3: 4:

tcenter = EPKc (DSKc (tcenter )) ε = |tcenter − tdevice | if ε ≤ εt , network status=safe. else, network status=unsafe. erase SI.

3. If SGM is not received within the estimated time (i.e., εt ), each device erases SI as a high probability of attack exists and the SGM generator is compromised. Step 2 in the above procedure is clarified as follows: each device decides whether it is in safe status or not by checking whether the time difference between the device’s local time stamp (tdevice ) and the CP’s time stamp (tcenter ) is less than εt or not. εt considers the transmission delay and an additional time drift between the CP and each device. If the time difference is greater than εt , which indicates that an attacker probably intercepted the time packet or the CP is compromised, the device erases SI from its memory to avoid its possible reveal to the attacker.

4

Secret Sharing with Code Attestation

The above scheme, however, is vulnerable to the two possible deviation factors. First, though the CP is compromised, it may not be able to send the SGM on time. The other deviation is related to the node itself in that some possible time drift might not be considered in the design may lead to the deletion of SI though the network is not under attack. To overcome the deviation exposed in the above time-based scheme, we introduce the secret sharing with authentication via the code attestation as solution. The scheme is detailed in the following sections. 4.1

Motivation: Necessity of Secret Sharing

Assuming that the HN devices are in an untrusted environment, we further believe that each attacker has the ability to extract the whole memory contents of any device including the SI. In the previous scheme, we store SI only in volatile memory which requires much care to maintain. However, SI is generally not for “one-time use” so that recovering the SI from a recovery remote storage system will be required. We overcome this dilemma by diffusing (i.e., distributing) SI into several devices using secret sharing to reduce the impact of single or several devices compromise. In the following, we introduce the secret sharing scheme based on the early introduced HN scenario and architecture.


555

Secret Sharing. In secret sharing scheme (def. 4), the SI is divided into partial secrets and the resulting shares are distributed through different devices [6]. More precisely, the SI is divided into n ≥ t number of partial secrets by assigning f (i) to the i-th device where i is evaluated in f (x) = s + a1 x + a2 x2 + · · · + at−1 xt−1 where f (x) is a polynomial with random coefficients and of degree t − 1. Devices can recover and then use the SI by recovering the coefficients of the polynomial after gathering t number of partial secrets (out of the n). Threshold Cryptography. The main goal of the threshold cryptography is the protection of information by fault-tolerant distributing of this information among devices in the network [14]. A device can use the SI when the received shares are more than a threshold value t out of n where n is the number of the devices in the HN. Since recovering the whole secrets is possible for an attacker once he compromises a number of nodes, recovering secrets may lead to some security problems. Therefore, we use the threshold cryptography which makes usage of SI from its shares recovering it. Even though, by adapting secret sharing or threshold cryptography at the HN as shown in this procedure, we cannot eliminate all of the uprising problems. For example, since we use those methods to distribute the SI at a point, we cannot verify the device which holds the share used for recovering the partial or the whole secret. In the following, we review a direction for method that may be beneficial for the devices authentication. 4.2

Code Attestation for Shares’ Authentication

Generally, authentication between two parties is performed using information that each [15] or at least one [16] of the two parties knows. However, this kind of authentication cannot be trusted in an untrusted environment because the adversary can perform authentication illegally after obtaining SI from the device which is captured physically. Therefore, received value (keys or whatsoever) for authentication

Device n

Device 3

Code Attestation

Device 2

Control Point

Device 1

Initiating home network device

Notification

Partial Request

Partial Response

Fig. 3. Illustration of secret information recovery process in which the initiating device is authenticated via the CP which confirms the authenticity to other devices in the network. Based on the authenticity of the device, other devices may send their partial secrets to it.

556

A. Mohaisen et al.

might be delivered by an attacker and need to be verified (i.e., authenticated) to confirm that the value is generated by a uncompromised device. Code Attestation: To solve the above problem, the code attestation is used (def. 6). In the code attestation, we verify a device’s whole memory to detect malicious code that may exist. Therefore, this scheme requires an authenticator to have a copy of the same code of the device to emulate and to generate same value required for authentication. If the result of code attestation discovers an abnormality of modified code, the device is considered as unavailable. Code attestation is performed through the CP by sending attestation code to enforce its execution or with code that already stored in the device. It is important to exclude any probability that an attacker may inject malicious code to avoid the attestation itself. SWATT [9], PIONEER [8] and Remote Code Attestation [10] are well-known code attestation methods and can be used as a building blocks for any attestation-based scheme including our system. 4.3

Integration: Secret Sharing for Home Network

Once a device joins in to the network, the device obtains public key certificate from CP after it is verified by the CP through code attestation. The device generates public and private key and then sends the private key to CP. To reduce the computation at the side of the device, CP may generate the keys and send them to the device. The device deletes the private key after sending the key. Upon that, the following is performed: 1. CP which has SI of device generates a polynomial f (x) of degree t − 1 with t random coefficients (i.e., a0 . . . at−1 ). 2. CP erases all of the information related to SI from the memory after distributing f (i) to each devices. This guarantees that the CP is not a bottleneck anymore after the initiation phase. 3. When required, HN device requests other devices which holds partial secret to send them to recover the SI (as in Fig. 3). 4. Before the devices send the partial secrets to the initiating device, the CP verify the authenticity of the that devices and inform devices holding the partial secrets by the authentication result. 5. If the initiating device is authenticated by the CP, the CP confirms to other devices by sending response messages. The response messages are encrypted by CP’s private key so that other devices in the network confirm the message by CP’s public key within the validation time τ . 6. Other devices can verify the authentication response and determine whether to reply to the initiating device’s request or not. The responses for the initiated requests are encrypted by session key (see def. 5). 7. The initiating device recovers the secret information from its shares (see def. 4) after decrypting the shares using the shared session key (def. 5). Otherwise, a device requests service to CP where CP can perform the service by assembling results from the devices as threshold cryptography as in Fig. 4. CP


Control Point Device n

Device 3

Service Request

Device 1

Code Attestation

Device 2

Initiating home network device

557

Notification

Partial Request

Partial Response Service Response

Fig. 4. Illustration of SI Merging. Unlike the recovery process, all secret merging operations from its partial shares are performed at the CP.

requests partial service requests to other devices instead of the initiating device and sends back the service response to that device. This solution, however, suffers from the trustworthiness of the CP. That is, the CP is the bottleneck for such design.

5

Comparison and Extension

5.1

Comparison

Both of Safe umbrella and secret sharing have their own characteristics which determine their advantages and disadvantages. These characteristics range from hardware implementations (i.e., memory type) to the functionality (authentication, rejoining, secret recovery, etc). Table 1 shows a brief comparison. One of the interesting advantages of the safe umbrella over the secret sharing is the ease of the secret key usage while the secret sharing requires gathering partial secrets that require multiple message request and response as shown earlier. However, the security of the safe umbrella is heuristic in that it is determined by the attacker’s ability related to the time slot available for the authentication. Table 1. Comparison of Safe Umbrella and Secret Sharing. VM refers to volatile memory and NVM refers to the non-volatile memory. Feature Device Authentication Storage of SI Rejoining Network Restoring Secret key Usage of Secret key

Safe Umbrella public key of device VM of device impossible impossible direct

Secret Sharing code of device NVM of other devices possible possible indirect

558

A. Mohaisen et al.

Volatile Memory SK

Non-Volatile Memory S1

S2

S3

S4

Sn

Public Key Crypto

Secret Sharing

Device n

Safe Umbrella

Device 2

Device 1

Control Point

Threshold Crypto Code Attestation Challenge-Response

Home Device

Fig. 5. Architectural view and building blocks of the hybrid scheme that makes use of both schemes’ advantages

The obvious disadvantage of the same scheme, however, is that it is impossible to restore the secret key (for further use) or to rejoin to the network. On the other hand, secret sharing is provided with the rejoining feature via the code attestation and to restore the SI which makes a better solution for long-living network. However, those two features are provided on the account of using the CP which could be a bottleneck in the design. As shown above, the safe umbrella and secret sharing has several advantageous characteristics over each other. To make use advantages of both, we briefly introduce a hybrid scheme that uses both schemes as building blocks. 5.2

Hybrid Scheme

Based on different characteristics of two introduced schemes, we provide a hybrid scheme by maintaining SI in the device’s volatile memory as safe umbrella does and by enabling rejoining the network using secret sharing with code attestation. Thus, the overall procedure is as following: 1. When a device attempts to enter the network, CP performs code attestation to the device 2. If the device is authenticated successfully, the device requests from CP to restore his own secret key for the future use. 3. The device save restored secret key in its own volatile memory for future usage. 4. The CP broadcasts SGM for devices to recognize themselves as safe. 5. In case SGM is delayed over estimated time, device erases the secret key from the volatile memory. By doing so, the different device will be re-keyed always upon joining the network. Later when the device gets valid SGM, it can again restore the secret key after attesting its memory contents. An illustration of the architecture and the building blocks of the hybrid scheme’s structure is shown in Fig. 5.


6

559

Security Evaluation and Cost Estimation

6.1

Security Evaluation

In this section we overview and estimate the security from different points of views including the restoring chances of the SI, attack on running devices, control point’s criticality, and the number of devices and that impact of that on the security. Restoring SI. Our scheme considers the deletion of the SI stored in volatile memory when the device senses an abnormal condition (regardless to whether the revealed information is the designated SI or not)3 . SI can be restored through secret sharing which is available only for a device which is authenticated by CP through code attestation. Code attestation is not perfect against powerful attacker who has powerful hardware, but we believe it is hard to fetch this kind of powerful attacking machine at home environment and to deceive CP for communication (with the attacker’s own CP and pretending as a legal device). Finally, malicious code in a compromised device that behave as a legal device for obtaining SI will be detected by code attestation when the device tries to join in the network. Attack on Running Device. To obtain SI on running device, an adversary must make the device sure that networking and safe umbrella are working correctly until obtaining the secret. Otherwise, adversary needs to obtain the right of execution to incapacitate safe umbrella which is impossible to be performed in the HN considering the hardness of the physical attack. Control Point: Even though the attack on CP is difficult to be performed, an adversary can take out the information when he can control the CP. Because CP does not have SI anymore after the initiation phase, to obtain SI an attacker must disguise itself as a legal device for making a request to collect partial secrets which is hard to be done in practice is shown earlier. A number of Devices. An adversary can restore the SI by attacking as many devices as possible in the HN. The purpose of secret sharing is dispersing attacker’s target. Based on that, this attacking way is related to the cost of attack. In other words, the more devices are used for secret sharing the more security is guaranteed. 6.2

Cost Estimation

In this section we consider the overhead in terms of the computation only determined through the different building blocks of our schemes. 3

We assume that the device can delete the SI before its physical capture due to any abnormal condition. Furthermore, we believe that reveal of the SI, in whole or in parts, to an attacker will affect security of the whole system in a way or another.

560

A. Mohaisen et al.

Validating SGM: RSA [5] requires higher resources compared to the symmetric key cryptography, however by choosing a small number as in many other applications (e.g., RFID and sensor networks), its operation can be faster than the normal case. Verifying message procedure is same like encrypting plain text which is decrypted by private key using public key. Therefore, having limited ability of operation, device can verify the message signed by CP’s private key with a relatively small cost. To sum up, the required overhead per operation is 1 sign (at the CP and n verifications at the devices’ side which means that a single verification is required per device. Secret Sharing and Threshold Cryptography: Threshold cryptography requires many exponent operations. Fortunately in the proposed scheme, threshold cryptography is performed through CP with request from a device so that only the device’s costs are represented in the cost of the request and response for the service. For secret sharing used to decentralize the secret key, devices require operations such like Lagrange interpolation (for polynomial). As the polynomial degree is related to the security parameter and the size of the network, the overall cost therefore increases in proportion to the number of devices. Session Key Generation: The session key generation is required for secure exchange of the shares. The computation of the DH based key requires two exponentiation operations as shown (in def. 5). Code Attestation. A number of operations such like hash and checksum are needed to perform code attestation. Code attestation is performed only when a device try to join the network. Even though cost for code attestation is bigger than our expectation, code attestation is rarely performed. Thus, we believe that the cost for code attestation is marginally acceptable for most of the common HN devices 4 . Since some of HN devices operate only when they are needed or scheduled by the network operator or owner, the CP may not realize whether the device is just powered off or is under attack. For this reason, CP may performs code attestation to the device for authentication when the device attempts to join in the network. The code attestation which detects malicious code on the device as an essential functionality can also be used for our authentication goal.

7

Concluding Remarks

We introduced several schemes for protecting SI in HN devices. Our schemes are based on several existing technologies and algorithms adapted into the home network environment with its specifications. Motivated by the special communication pattern in the home network, we introduced the safe umbrella which, to some extent, solves the problem of the SI protection. To overcome limits of 4

In fact, code attestation is very applicable to sensor nodes which are essential components in the home network environment.


561

the safe umbrella, we introduced the secret sharing scheme with code attestation in which the secrets are maintained for long living network. To show the value of our proposed schemes, we showed detailed security and cost analysis in addition to the discussion of several network scenarios. In a future work, we will investigate other method to substitute SGM-time stamp and finding other applications for the hybrid scheme. Also, it will be beneficial to consider other attacking scenarios rather than the physical capture.

Acknowledgment The authors would like to thank the anonymous reviewers for their valuable comments. This research was supported by the MKE(Ministry of Knowledge Economy), Korea, under the ITRC(Information Technology Research Center) support program supervised by the IITA(Institute of Information Technology Advancement) (IITA-2008-C1090-0801-0028)

References 1. LG electronics: HomeNet (2007), http://www.lge.com/products/homenetwork/homenetwork.jsp 2. Group, T.C.: Trusted computing platform alliance main specification version 1.1b (2003) 3. Sengodan, S., Edlund, R.Z.L.: On securing home networks. In: INET (2001) 4. Wallach, D.S., Balfanz, D., Dean, D., Felten, E.W.: Extensible security architectures for Java. In: 16th Symposium on Operating Systems Principles, pp. 116–128 (1997) 5. Rivest, R.L., Shamir, A., Adleman, L.M.: A method for obtaining digital signatures and public-key cryptosystems. Commun. ACM 21, 120–126 (1978) 6. Shamir, A.: How to share a secret. Commun. ACM 22, 612–613 (1979) 7. Diffie, W., Hellman, M.E.: New directions in cryptography. IEEE Transactions on Information Theory 22, 644 (1976) 8. Seshadri, A., Luk, M., Shi, E., Perrig, A., van Doorn, L., Khosla, P.K.: Pioneer: verifying code integrity and enforcing untampered code execution on legacy systems. In: Herbert, A., Birman, K.P. (eds.) SOSP, pp. 1–16. ACM (2005) 9. Seshadri, A., Perrig, A., van Doorn, L., Khosla, P.K.: Swatt: Software-based attestation for embedded devices. In: IEEE Symposium on Security and Privacy, p. 272. IEEE Computer Society (2004) 10. Shaneck, M., Mahadevan, K., Kher, V., Kim, Y.: Remote software-based attestation for wireless sensors. In: Molva, R., Tsudik, G., Westhoff, D. (eds.) ESAS 2005. LNCS, vol. 3813, pp. 27–41. Springer, Heidelberg (2005) 11. Nakamura, M., Tanaka, A., Igaki, H., Tamada, H.: Adapting legacy home appliances to home network systems usingweb services. In: ICWS, pp. 849–858 (2006) 12. Perrig, A., Stankovic, J.A., Wagner, D.: Security in wireless sensor networks. Commun. ACM 47, 53–57 (2004)

562

A. Mohaisen et al.

13. Perrig, A., Szewczyk, R., Tygar, J.D., Wen, V., Culler, D.E.: Spins: Security protocols for sensor networks. Wireless Networks 8, 521–534 (2002) 14. Desmedt, Y., Frankel, Y.: Threshold cryptosystems. In: Brassard, G. (ed.) CRYPTO 1989. LNCS, vol. 435, pp. 307–315. Springer, Heidelberg (1990) 15. Bellovin, S.M., Merritt, M.: Augmented encrypted key exchange: A password-based protocol secure against dictionary attacks and password file compromise. In: ACM Conference on Computer and Communications Security, pp. 244–250 (1993) 16. Feige, U., Fiat, A., Shamir, A.: Zero-knowledge proofs of identity. J. Cryptology 1, 77–94 (1988)

Universal Remote Control for the Smart World Jukka Riekki1, Ivan Sanchez1, and Mikko Pyykkönen2 1

University of Oulu Department of Electrical and Information Engineering and Infotech Oulu P.O. Box 4500, 90014 University of Oulu, Finland {jukka.riekki,ivan.milara}@ee.oulu.fi 2 University of Lapland Department of Industrial Design P.O. Box 122, 96101 Rovaniemi, Finland [email protected]

Abstract. In this paper, we discuss how to build user friendly user interfaces to the smart world. We present the REACHeS architecture for controlling Internet services through physical user interfaces, using a mobile terminal and icons placed in the environment. An icon advertises a service that can be started by touching the icon with a mobile terminal. This service activation configures the mobile terminal as a remote control for the service. We have implemented this architecture and designed an icon set. The physical user interface is based on RFID technology: the terminals are equipped with RFID readers and RFID tags are placed under the icons. We present the first prototype applications and the first usability tests that we have carried out.

1 Introduction The number of devices that provide services to us in our daily environment increases at an accelerating pace. Hence, the effort required from us, the users, to learn to use all these services is increasing as well. Furthermore, using the services most often requires us to interrupt our daily activities and to focus on the service’s user interface. How could we change this situation? How could we develop our environment into a smart world that allows us to focus on our daily life and just use the services when we need them, without studying their use beforehand and without distracting our focus too much from the everyday activities? In this paper, we discuss how this challenge can be tackled with users’ personal mobile terminals and physical user interfaces. We present the REACHeS system that changes mobile terminals into universal remote controls for the smart world. When a user wants to use a service, she/he first scans visually the local environment for icons advertising services locally available. Then, the user touches the icon advertising the desired service with her/his mobile terminal. The action of touching generates an event that is received by the REACHeS system. The event is generated by the mobile terminal’s RFID reader when it reads the RFID tag placed under the icon. REACHeS delivers the event to the server implementing the requested service. The service replies by creating a user interface into the mobile terminal and taking the necessary F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 563–577, 2008. © Springer-Verlag Berlin Heidelberg 2008

564

J. Riekki, I. Sanchez, and M. Pyykkönen

local resources into use. As a result, the user controls the service, and the resources used by it, with the mobile terminal. The REACHeS system transmits all communication between the service, the mobile terminal, and the local resources. It is also in charge of allocating resources. The main contributions of this work are the icon set and the REACHeS architecture. These contributions enable a wide variety of Internet services to be controlled by touching icons in the environment with mobile terminals. In addition, the architecture facilitates controlling local resources like wall displays. A further contribution is building physical user interfaces from off-the-self components. Icons advertising RFID tags have not been studied much, some suggestions for visualising RFID tags can be found but not any comparable icon set [1,2,3]. Earlier versions of our icons can be found from [4,5]. Here, we present the largest uniform icon set so far and elaborate further different possibilities for placing the icons in the environment. Remote controls and systems resembling the REACHeS have been studied more, see for example [6,7,8,9,10,11]. However, we are not aware of any other architecture that integrates mobile terminals, services, and local resources into a single system, using a comparable set of widely-used and robust technologies. We have presented the REACHeS architecture and the first prototypes in [12]. Here, we focus on the communication between system components and report performance measurements, user interfaces that are consistent with the icons, and the first usability studies. The rest of the paper is organized as follows: in the next two sections, we present the physical user interface and the REACHeS system. The fourth section describes two prototypes and the fifth section the performance measurements and the usability study. The sixth section contains the discussion and comparison with related work. The seventh section concludes the paper.

2 Physical User Interface We see user friendliness as one central characteristic of the smart world. Why should we develop more technology into our daily environment if it is difficult to use and it takes time from our everyday activities? New technology should make our life easier and free time for the activities that we value! Our approach in improving user friendliness is to offer a physical user interface for the user. The physical user interface consists of icons and a mobile terminal. The icons are placed in the environment and they advertise the services that are available locally. A service is activated by touching the corresponding icon with a mobile terminal. Although also other technologies can be used to implement the physical user interface, we focus on RFID technology in this paper. A technology having a short reading distance (less than 5 cm) is used; so the user needs to intentionally bring the terminal near the icon to initiate the reading of an RFID tag. Hence, the reading events can be interpreted as commands to the system serving the user, and it is natural to describe the selection as "touching" an icon. An icon advertises a point in the environment that can be touched with a terminal and a service that is started when the icon is touched. An icon forms, together with the RFID tag placed behind the icon, a two-sided interface between the physical and digital worlds. For the user, this interface advertises a service available and for the system this interface contains the data needed to start the service.

Universal Remote Control for the Smart World

565

The main challenge in icon design is to communicate the icons’ meaning clearly to all potential users. Basically, an icon has to communicate enough information for the user to decide whether to touch the icon or not. If users do not recognize all icons to belong to the physical UI, some services will not be activated. On the other hand, if a user recognizes an icon but interprets it incorrectly, the activated service is not the expected one. Moreover, if the user touches an icon that is not part of the physical UI, the user might reason that the system is broken. Icon design is clearly a challenging task, especially when the goal is that the icons can be placed anywhere in our everyday environment where users might potentially use the advertised services. To tackle this challenge, we have divided icons into two parts: the outer part is a general icon that communicates to the user a point that can be touched. The inner part, in its turn, is a special icon that advertises a service. Figure 1 presents our icon set. The first icon is the general icon. The next five icons advertise services performing simple actions: print something, show some entity’s calendar, send a message or call some entity, and locate some entity. The first icon on the second row, Play Slideshow, starts a specific application. The following five icons have been designed to advertise the corresponding place on some document, for example on a map.

Fig. 1. The icon set

566


The first icon on the third row, Remote control, advertises that some resource can be controlled remotely by touching here. The next four icons advertise services that are related to an information storage, that is, to a container. The following icon, Drop to wall display, advertises a service that drops a document from the mobile terminal to a wall display. The next icon, Pick from wall display, advertises a document transfer to the opposite direction. The second icon on the fourth row, Save, saves a document when touched and the third icon, Play audio, plays an audio file. The next icon, Blog, brings a specific blog page to the mobile terminal’s display. The Join the game icon configures the user’s mobile terminal as a game pad for a game played in the local environment. The last icon, Stop, might be used to stop a service. All information about the service can not be encoded in the icon, no matter how well it is designed. For example, the Drop to wall display icon specifies the drop action. But which document is to be dropped and to which wall display? A shared understanding is clearly required between the user and the system; they both have to interpret the situation in the same way. Icon placement can be used in building this understanding. For example, if we place the Drop to wall display icon on the wall display (or next to the display), it is obvious that the document is to be dropped on just that wall display. Placing a Drop to wall display icon on a wall display is an example of a direct tag placement. In this case, an icon is placed on an entity that has a role in the service. Here, it is clear that the action is "Drop a document to this wall display". However, all icons can not be placed directly on an entity that has a role in the service. In such cases, indirect placement offers a solution: the icon is placed on a document that describes an entity having a role in the service. Documents advertising music might contain Play audio icons. The icons advertising places (Attraction, Beach, etc) can be placed, for example, on a map or on a tourist guide. Furthermore, as this technology is new for the majority of the users, we might start the usage by placing icons on posters that instruct the users. For example, a poster might describe a local resource and inform the user that by touching the Remote control icon (included in the poster) brings a remote control UI on the mobile terminal’s display. In weak placement, the entity on which the icon is placed has no role in the service. A better alternative is to use indirect placement; to place the icon on a poster describing the service, at least till the icon is widely known. Moreover, in embedded placement, an RFID tag is installed under an icon that is already present in the environment, for example, under a name tag in an office corridor. Finally, an icon can be placed on top of an RFID reader. In this case, the icon is not bound to one service but the system can change dynamically the data that is read into a terminal when the icon is touched – the icon needs to advertise this to the users. This approach allows any mobile terminal to be used (having a network connection and able to receive data that is pushed from a server). The terminal just needs to be equipped with an RFID tag. When the user brings the terminal near the reader, data identifying the terminal is read from the tag and delivered to a server.

3 REACHeS We suggest that the user’s personal mobile terminal is the main interaction device in the smart world. The REACHeS (Remotely Enabling and Controlling Heterogeneous Services) system provides for services a simple interface for creating a user interface


567

Fig. 2. The REACHeS architecture

into the mobile terminal and communicating with it (Figure 2). The system also controls the local resources (wall displays, etc) and offers for the services interfaces for using these resources. When a user wants to use some service, she/he browses the environment, selects the corresponding icon, and touches it with a mobile terminal. The RFID reader installed in the mobile terminal reads data from the RFID tag that is placed under the icon (number 1 in the Figure). The data read from the tag is delivered as an event to the REACHeS system (2). The REACHeS passes the data to the server that is responsible for the service in question (3). The service replies with a message that determines the user interface to be created to the mobile terminal (4). REACHeS adapts and passes this information to the mobile terminal (5). As a result, the user sees the remote control interface on the mobile terminal display and can start to control the service. During operation, REACHeS transmits the messages that are sent from the mobile terminal to the service and vice versa (as indicated by 2-5). The service can also request the control of local resources (e.g. a wall display) from REACHeS (6). In such a case, REACHeS redirects events from the service to the local resources, and vice versa. From the service’s point of view, the events flow directly to the resource, thus REACHeS establishes a kind of virtual path between the service and the resource (7). In a typical message sequence, a command given by the user is first delivered through REACHeS to the service and then the service responds by sending a response to the mobile terminal and a command to a local resource. The communication proceeds like this till the service is stopped.

568


The main components of the REACHeS are the User Interface Gateway (UIG) and the System Display Control (SDC). UIG is the core component that delivers the messages sent to REACHeS to the intended destinations. When a service component sends a message to a local resource, UIG directs the message to the component controlling the resource. Thus far, we have defined only the component controlling displays (SDC), but components controlling other resource types are straightforward to add. REACHeS contains also adaptor components for client (i.e. mobile terminal) GUIs, interface components for the external services, an administration component, and a database for storing information about local resources, registered services and relations between them. The communicated messages deliver events. Messaging is implemented using HTTP protocol. An event is represented either directly as message parameters or as simple XML fragment like <event> event parameter1 … parametern , where the name of the event, event, is followed by the parameters of the event. The first alternative is used in communication between clients and services, while the second one is used to transmit messages from the service to a resource. REACHeS is implemented using servlets and runs in a Tomcat 5.5. The mobile phones (Nokia 6131 NFC) are equipped with NFC compliant RFID readers. The phones implement the JSR 257 (Contactless Communication API) for communicating with the RFID reader. Near Field Communication (NFC) [13] is a standard-based, short range wireless connectivity technology. Data is communicated as NFC Data Exchange Format (NDEF) messages that are composed of records, which can contain, for example, a URL or location data [14]. Applications use record types to identify the semantics and structure of the record content. We have two versions of the client application. The first one is simply the phone’s Web browser; the second client version is a MIDlet application. The browser client is started when a tag containing a URL is read. An HTTP request is sent (message number 2 in Figure 2) and the received webpage (5) is shown on the display. The MIDlet is started when an NDEF record associated with the MIDlet is read from a tag. The MIDlet sends an HTTP request to the REACHeS system (2) and receives a response (5). In both cases, the communication proceeds as described above. Wall displays are the first local resources that we have integrated into the REACHeS system. A service controls a wall display by sending HTTP Post requests to the REACHeS which passes the requests to the wall display’s browser. A request contains one or more events. The name of an event is a method name from Table 1 and the event parameters are that method’s parameters. Flash is used to play multimedia content. The multimedia player (JWMediaPlayer, [15]) is a Flash application that is loaded in a webpage using the insertFlashObject() method. When a service receives a command from the user (play, pause, stop, etc), it processes the command and sends the corresponding sendCommandToObject() method call as an event to the wall display’s browser. The external services have to be able to communicate with the REACHeS using the HTTP protocol. A service implementation contains a service component, tags for starting the service, the icons, and the client application. In addition, data can be stored in files in XML format. For example, when a user activates a photo album application, the photographs to be shown can be specified in an XML file. The location of this file can be given as a parameter in the URL that is stored in the tag. This approach allows the same application to be activated with different parameters easily.


569

Table 1. The interface for controlling wall displays Method insertBody(source) insertWebPage(url) changeAttribute(element, attribute, value) playAudio(file) stopAudioFile() insertFlashObject(flashFile, id, destination, filelist, extra) sendCommandToObject(id, command, extra)

Description Shows the source webpage on the wall display. Shows the url webpage on the wall display. Changes from the current document the value of the element's attribute to value. This method can be used, for example, to change the image that is shown on the webpage. Starts playing the audio file file. Stops playing the currently played audio file. Inserts flashFile object in the element destination. Id is used as an identifier when events are sent to the object. Filelist parameter passes a list of files to be played by the Flash player. Also extra parameters such as width, height, autostart, etc can be delivered. Sends the command to the object id. Extra parameters can be given as well.

4 Prototypes The Product Browser application lets the user browse advertisements on the wall display using a mobile terminal as a remote control. The wall display might be placed on a shop window or on the wall of a shopping mall, for example. A Remote Control icon is bound to the Product Browser (i.e. the RFID tag contains a set of NDEF Records defining the service and its parameters). Another option would be to have a special Product Browser icon. The wall display and the service need to be registered into REACHeS before the application is used. When a user touches a tag (Figure 3), the remote control sends to the REACHeS the HTTP GET request that specifies the Start event (2 in Figure 2). This event is listed on the first row of Table 2; the actual server address and port number have been replaced with “server” and “port”. The file playlist.xml is a file following the RSS 2.0 specification. The file specifies the information (images and text) that is shown on the wall display. The REACHeS responds to the request by reserving the display and sending a request to the Product Browser service (3 in Figure 2). This request resembles the one received from the mobile terminal. The

Fig. 3. User starts the Product Browser application

570

J. Riekki, I. Sanchez, and M. Pyykkönen Table 2. Examples of communication Message Start Update display Next ad request Next ad event Close

Data server:port/reaches?service=show_products&event=start&display=10.20.41.8& playlist=server:port/productBrowser/playlist.xml <event>insertWebPage server:port/reaches/productBrowser/main.html server:port/reaches?service=show_products&event=next <event>changeAttribute mainPhoto src server:port/…/productBrowser/images/ image1.jpg server:port/reaches?service=show_products&event=close

Fig. 4. The Product Browser user interface on the mobile phone’s display

Fig. 5. The first advertisement in the Product Browser

service accesses the Internet if necessary and responds by updating both the mobile terminal’s display and the wall display. The second row of Table 2 shows the event the REACHeS receives from the service and redirects to the display. Now, the wall


571

display presents the title page and the mobile phone’s display presents the user interface for controlling the Product Browser (Figure 4). The user can browse the advertisements by pressing the buttons at the lower part of the display. The small "i" button opens a webpage presenting additional information. When the user selects for example the Next command, the REACHeS passes this command to the service as the HTTP request listed at the third row of Table 2. The service replies by sending and event determining that the next advertisement should be shown on the wall display (fourth row in Table 2). REACHeS sends the event to the corresponding display. As a result, the first advertisement is shown on the wall display (Figure 5). The commands given by the user are processed like this till the user closes the service with the HTTP request that causes all resources associated to this service to be released (fifth row in Table 2). Other prototypes such as the Multimedia Player and the SlideShow Viewer are described in [12].

5 Testing 5.1 Performance We performed the measurements listed in Table 3. Due to space constraints, we do not report the detailed test setup or sequence but focus on the results. The aim of these measurements was to verify our assumption that the mobile network delay dominates the overall latency (EGPRS was used), to measure the significance of the service communication in overall latency, and to study the communication between the service and the display. We measured latencies for both internal services (running in the same machine with the REACHeS) and external services (running in different machines). Table 4 shows the results. The times are slightly longer if we use external services but the difference is small. Start events require much time because both the REACHeS and the service must allocate necessary resources. For the other events, the latency is (on average) 1.2 seconds for external services. The mobile data network is clearly the slowest component, as the REACHeS Execution Time is 100 times shorter than the Total Latency for common events. The last three experiments indicate that all the other system components have acceptable performance. Table 3. Performance measurements 1. Total Latency: the time from a button press that generates an event till the phone receives a message informing that the event has been processed by the service. 2. REACHeS Execution Time: the time from the moment the REACHeS receives a request from the mobile phone till it sends a response back. 3. Service Execution Time: the time from the moment the REACHeS sends a request to the service till it receives a response back. 4. Display Update Time: the time from the moment the service sends a request to the REACHeS till the display performs the required action.

572

J. Riekki, I. Sanchez, and M. Pyykkönen Table 4. The performance of the REACHeS system Service location

Experiment

1. Total Latency 2. REACHeS Execution time 3. Service Execution Time 4. Display Update Time

Internal External Internal External Internal External Internal External

Measured time [ms] Start event Other events 3300 1200 4500 1100 1265 1461 1095 1076 1398 1887

10 13 2 5 171 202

When a wall display is used, the Total Latency is not as important as the latency from the moment the user sends an event till the moment the effect is shown on the wall display. This Effective Time can be approximated from the measured times as follows: EffectiveT ime =

Total _ Latency + Display _ Update _ Time 2

We assume that the HTTP request made from the mobile phone and the response sent from the REACHeS have the same network delay. The Display Update Time has no effect on the Total Latency because the service does not wait a response from the display before it sends a response back to the REACHeS. Table 5 shows that Effective Time for common events is always under one second. Table 5. The Effective Time [ms]

Internal Service External Service

Start 3040 4130

Other 771 752

5.2 Usability Study We have performed a usability study to compare controlling a wall display using: (1) the display’s touch screen, (2) gestures that are recognized by a hand-held sensor device [16], and (3) a mobile terminal as a remote control. The last method utilized the REACHeS system. The wall display is presented in Figure 6. The test group consisted of 11 participants with the average age of 28 years. We tested two applications: a Photo Album reminding the Product Browser and a remote browser control that allows a user to control a wall display’s browser. Each command available was associated with a button in the remote control’s UI and had an own gesture as well. The normal browser GUI was used directly to control the wall display. First, we tested how intuitive the different methods were. The participants used both applications with all three control methods without knowing how each command was performed. The majority of the participants succeeded in relating the commands with the buttons of the mobile terminal (five out of eight commands on average), but only two out of eight gestures were guessed.


573

Fig. 6. The wall display

At the second stage, we explained to the participants how to perform each command. Then, the participants had to achieve a small task and the time to accomplish the task was measured. The touch screen was clearly faster (average time 82 seconds) than our remote control system (average time 139 seconds), or gesture control (average time 144 seconds); most probably because the participants used the familiar Windows style GUI directly and there were no wireless links or sensor data processing increasing the latency. On the other hand, both the gesture control and the remote control GUI were new for the participants and their development was in quite an early stage when this test was carried out (an earlier version of the mobile phone GUI was used). This immaturity probably affected also to the low rate of relating the commands to the buttons and gestures. Furthermore, the participants had to share their attention between the mobile GUI and the wall display. We noticed that the participants mainly focused on the feedback shown on the wall display. Selecting hyperlinks was easiest with the touch screen as the correct link could be selected directly. When gestures and the remote control were used, the participants had to move through the hyperlinks in the order they were included in the webpage. The participants evaluated the touch screen to be otherwise the best interaction device but the remote control was valued to be equally reliable. The remote control was evaluated to be better than the gesture control. When the remote control was used, the participants needed to touch an icon to activate the service. The majority of the participants (64%) considered the icons intuitive or very intuitive and that no extra textual information is required to explain them. The participants commented on the need to be instructed that an icon needs to be touched to start a service. For the majority (82%) of users, the system worked as they expected. The REACHeS was evaluated by the majority to be reliable and easy to use. Over half (56%) of the

574


participants felt comfortable or very comfortable using the keypad to send commands to the service and most (91%) felt comfortable or very comfortable using the mobile phone display as a remote control GUI. Many participants remarked as a positive aspect that the mobile phone is a familiar device. We also noticed that the cognitive load was not very high as we could establish short conversations with the participants.

6 Discussion We presented an architecture for controlling Internet services through physical user interfaces and an icon set designed for the physical user interfaces. A user activates a service by touching the corresponding icon with a mobile terminal. This user interface is physical, as mobile terminals are used as physical objects rather than as traditional I/O devices. This kind of user interfaces are also known as tangible user interfaces [17]. Ailisto et al. [18] classify this approach of service activation as physical selection and present concepts for selection by touching, pointing, and scanning. Our approach is similar to their concept of selection by touching. An icon forms, together with the RFID tag placed behind the icon, a two-sided interface between the physical and digital worlds. Such interfaces have been suggested by many others, for example Want et al. [19] published their often referenced work in late 90’s. A central characteristic of our approach is the emphasis on advertising services by icons. Icons of this kind have not been studied much. Want et al. [19] do not discuss specific icons for advertising RFID tags but focus more on unobtrusive tagging of physical objects. We classify this as embedded tag placement. Tungare et al. [2] emphasize that the user interacts with objects and present few examples of iconic representations. Ailisto et al. [18] do not suggest any visual appearance for RFID tags. Arnall [3] explores the visual link between information and physical things. Välkkynen et al. [1] present a few suggestions for visualising RFID tags. We have presented our earlier icon sets in [4,5]. But although there is not much work on the visual appearance of RFID tags, icons indicating RFID tags are already approaching our everyday life. For most of us, the first touch to RFID icons might be the icon that is used to mark passports that contain RFID tags [20]. The main difference between the physical user interface presented in this paper and the graphical user interfaces of the commercial mobile terminals (e.g. iPhone) is that the icons of the physical user interface are placed in the local environment, whereas the icons of a GUI are presented on the mobile terminal’s display. Placing the icons (and RFID tags) in the environment leads to easy interaction with a large number of services. The detailed information required to start a locally available service is hidden from the user (into an RFID tag). The amount of tags that can be placed in our daily environment – and hence the number of services that can be started by touching a tag – is large. When a service is started by selecting an icon from the mobile phone’s display, a compromise has to be made between the number of icons and the amount of information the user needs to enter. One alternative is to discover automatically the locally available services and present only the icons corresponding with these services on the mobile phone’s display. However, such a solution requires a considerably more complex implementation than the one presented in this paper.


575

We discussed the challenges for designing and placing icons. The icons need to be recognized as part of the physical user interface and they need to be interpreted correctly. As all information cannot be encoded in the icons nor stored in the tags, the placement can be used to provide information about the service. Additional information can be expressed in the form of rules shared by the users and the system. We suggested that user’s personal mobile terminal is the main interaction device in the smart world. The implemented REACHeS system illustrates how well-known, widely used technologies like HTTP, XML, and Flash suffice to build novel contextaware applications. Also, the mobile terminals are commercial, off-the-self mobile phones. Extra applications do not have to be installed to the computers controlling the displays nor to the mobile phone when the browser client is used. The potential of using mobile devices as universal control devices has been reported earlier [8,6,7,11]. However, these works do not cover a system that integrates the mobile terminals, services, and local resources into a single system. Raghunath et al. [9] propose controlling external displays using mobile phones and Broll et al. [21] present a system that allows Web Services to manipulate mobile phone displays. Both systems are more complex than REACHeS, but at the same time offer less functionality. Elope middleware [10] uses mobile phones with embedded RFID readers to start interoperation between components in the smart space. The middleware is in charge of setting up the parameters to start the communication between agents and services. The REACHeS controls the communication also between resources and services. Furthermore, a mobile phone is not used as a remote control in Elope. The REACHeS system functions in a reliable fashion. The performance measurements indicate that the EGPRS connection is the bottleneck, as can be expected. WLAN connection would without any doubt decrease the latency – but unfortunately terminals offering both a WLAN connection and an integrated RFID reader are not yet available. The current version is usable in many different applications. The latency is too big for real-time applications but small enough for browsing content. We are currently working with wall displays. However, numerous useful services do not require a wall display but the mobile phone’s UI capabilities suffice. Physical user interfaces can produce more fluent user interaction even for the common simple services such as making a phone call or sending a short message. Our earlier usability studies [4] indicate that the icons are easy to learn and that special icons are preferred over a general icon that brings a list of choices to the terminal’s display. The study reported here indicates that a mobile terminal is a reliable and easy to use remote control. A wall display’s own touch screen offers the fastest interaction but is not always the most suitable interaction device. A remote control is a viable option when a user-specific UI is justified and when touching the wall display is not possible. Furthermore, the overwhelming performance of the touch screen can be partly explained by the immaturity of the remote control prototype. However, the remote control has one characteristic shortcoming, namely a user has to share her/his focus of attention between two user interfaces: the remote control UI and the wall display UI. The usability study indicated that users pay more attention to the wall display, hence all important messages should be shown on the wall display In this paper, we focused on starting services by touching. Icons can also be used to control an already running service, for example to pause a video. Another

576


interesting research topic is combining several touches, to first pick a document from a container and then to drop it into a wall display, for example. These topics are in our research agenda. In the near future, we will also develop the icons further, build more prototypes and organize more usability studies. After having built and tested several prototypes, we can analyze the merits and drawbacks of REACHeS in more detail. We have started to study the security issues as well. Finally, developing REACHeS to communicate with mobile phones that contain an RFID tag instead of a reader clearly offers great potential, as this improvement would increase considerably the amount of mobile phones that can be used to access the services.

7 Conclusions We presented an architecture for controlling Internet services through physical user interfaces. We presented an icon set and the first prototypes and usability tests. Although the usability tests indicate that we can achieve good usability, our research on this kind of physical user interfaces is in its early stage. We have implemented some prototypes and used some icons in these prototypes, but most of the suggested services currently exist only on paper. We will continue this research by building more prototypes. We will test these prototypes in real life settings. We will clearly have a considerable amount of work ahead, but we argue that this is the only way to gain the deep understanding that is required to build truly user friendly services for the forthcoming smart world.

Acknowledgments This study was funded by the Finnish Funding Organization for Technology and Innovation and companies. Marta Cortés is acknowledged for taking photographs and collaborating in the prototype implementation. The test group is acknowledged for participation to the usability study.

References [1] Välkkynen, P., Tuomisto, T., Korhonen, I.: Suggestions for Visualising Physical Hyperlinks. In: Proc. Pervasive Mobile Interaction Devices, Dublin, Ireland, May 2006, pp. 245–254 (2006) [2] Tungare, M., Pyla, P.S., Bafna, P., Glina, V., Zheng, W., Yu, X., Balli, U., Harrison, S.: Embodied data objects: tangible interfaces to information appliances. In: 44th ACM Southeast Conference ACM SE 2006, Florida, USA, March 10–12, pp. 359–364 (2006) [3] Arnall, T.: A graphic language for touch-based interactions. Proc. Mobile Interaction with the Real World. In: Conj with the 8th Intl Conf on Human Computer Interaction with Mobile Devices and Services (MobileHCI 2006), Espoo, Finland, September 2006, pp. 18–22 (2006) [4] Riekki, J., Salminen, T., Alakärppä, I.: Requesting Pervasive Services by Touching RFID Tags. IEEE Pervasive Computing 5(1), 40–46 (2006)


577

[5] Riekki, J.: RFID and smart spaces. Int. J. Internet Protocol Technology 2(3/4), 143–152 (2007) [6] Myers, B.A.: Mobile Devices for Control. In: Paternó, F. (ed.) Mobile HCI 2002. LNCS, vol. 2411, pp. 1–8. Springer, Heidelberg (2002) [7] Christof, R.: The Mobile Phone as a Universal Interaction Device – Are There Limits? In: Proc of the Workshop Mobile Interaction with the Real World, 8th Intl Conf on Human Computer Interaction with Mobile Devices and Services Espoo, Finland (2006) [8] Nichols, J., Myers, B.: Controlling Home and Office Appliances with Smart Phones. IEEE Pervasive Computing 5(3), 60–67 (2006) [9] Raghunath, M., Ravi, N., Rosu, M.-C., Narayanaswami, C.: Inverted browser: a novel approach towards display symbiosis. In: proceedings of Fourth Annual IEEE International Conference on Pervasive Computing and Communications, Pisa, Italy, pp. 71–76 (2006) [10] Pering, T., Ballagas, R., Want, R.: Spontaneous marriages of mobile devices and interactive spaces. Communications of the ACM 48(9) (September 2005) [11] Ballagas, R., Borchers, J., Rohs, M., Sheridan, J.G.: The Smart Phone:A Ubiquitous Input Device. Pervasive Computing 5(1), 70–77 (2006) [12] Sánchez, I., Cortés, M., Riekki, J.: Controlling Multimedia Players using NFC Enabled mobile phones. In: Proceedings of 6th International Conference on Mobile and Ubiquitous Multimedia (MUM 2007), Oulu, Finland, December 12-14 (2007) [13] NFC Technology Architecture Specification. NFC Forum (2006) [14] NFC Data Exchange Format (NDEF) Technical Specification. NFC Forum (2006) [15] ABOUT JEROENWIJERING.COM (17.1.2008), http://www.joerenwijering.com [16] Kauppila, M., Pirttikangas, S., Su, X., Riekki, J.: Accelerometer Based Gestural Control of Browser Applications. In: Proc. Int. Workshop on Real Field Identification (RFId2007), UCS 2007, Tokyo, Japan, November 25, 2007, pp. 2–17 (2007) [17] Ullmer, B., Ishii, H.: Emerging frameworks for tangible user interfaces. IBM Systems Journal 39(3-4), 915–931 (2000) [18] Ailisto, H., Pohjanheimo, L., Välkkynen, P., Strömmer, E., Tuomisto, T., Korhonen, I.: Bridging the physical and virtual worlds by local connectivity-based physical selection. Personal Ubiquitous Computing 10(6), 333–344 (2006) [19] Want, R., Fishkin, K.P., Gujar, A., Harrison, B.L.: Bridging Physical and Virtual Worlds with Electronic Tags. In: Proc SIGCHI conference on Human factors in computing systems, Pittsburgh, Pennsylvania, USA, pp. 370–377 (1999) [20] Biometric passport (16.1. 2008), http://en.wikipedia.org/wiki/Biometric_passport [21] Broll, G., Siorpaes, S., Rukzio, E., Paolucci, M., Haamard, J., Wagner, M., Schmidt, A.: Supporting Mobile Services Usage through Physical Mobile Interaction. In: Proc 5th IEEE Intl Conf on Pervasive Computing and Communications, White Plains, NY, USA, pp. 262–271 (2007)

Mobile Navigation System for the Elderly – Preliminary Experiment and Evaluation Takahiro Kawamura1, Keisuke Umezu1,2 , and Akihiko Ohsuga1 1

Graduate School of Information Systems, University of Electro-Communications 2 NEC Corporation Abstract. The aging of society is emerging as a global problem. Participation of elderly people in social activities is highly desirable in several perspectives of the society. Thus, we propose a mobile service that supports elderly people wishing to go out and promotes their participation in social activities. This system collects information on the existence of nearby barriers and provides timely information to users via a mobile phone equipped with GPS. The nearby barriers include not only permanent barriers such as stairs, but also temporary ones such as crowds and parked bicycles. Further, we propose functionality that selects and informs barrier information suitable for each user. This paper illustrates the public experiment that we conducted in Tokyo, and confirms the accuracy of the information filtering.

1

Introduction

Recently, the rapid aging of society has emerged as a global problem. In Japan, elderly people (over 65 years of age) accounted for more than 20This proportion is expected to rise to 26% in 2015 [1]. Elderly people have several problems and anxieties in contemporary society. For example, 30% of elderly people in Japan have no friends [1]. Furthermore, people with disabilities have to make a great deal of effort in order to move around outside their homes. Therefore, many people rarely leave their homes. However, the engagement of the elderly in social activities, including not only work but also in voluntary or local activities, is a key to vitalization of the entire society. In this paper, we propose a mobile navigation system for the elderly that employs cellular phones equipped with GPS. It automatically selects barrier information in an urban environment according to the user’s location and profile, and then notifies it to the user via the cellular phone. We have evaluated the usability regarding the burden of operating the system and the accuracy of barrier notification corresponding to the user profile. The remainder of this paper is organized as follows. Section 2 defines requirements of the navigation system for the elderly on the basis of the results of our preliminary survey. Then, the system overview is described in section 3, and the barrier notification mechanism in section 4. Sections 5 and 6 present the experiments and results with regard to the usability and the notification accuracy, respectively. In section 7, which is the final section, we refer to related works and present our conclusions. F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 578–590, 2008. c Springer-Verlag Berlin Heidelberg 2008

Mobile Navigation System

2 2.1

579

Requirements of Mobile Navigation for the Elderly Survey Results

The survey was conducted orally and the subjects were the authors’ neighbors and relatives and members of the Japan National Council of Social Welfare. We collected mainly problems and complaints concerning daily life, and have decided to focus on the following three problems. 1. Decrease of the willingness to leave the home The main factors decreasing the willingness of the elderly to leave the home are barriers such as pedestrian ways with steps and stairs, illegally parked cars and bicycles, and lack of sidewalks. 2. Desire to be independent The elderly are grateful for the support from their families and volunteers, but most of them are reluctant to accept it. 3. Difficulty in using new machines The elderly tend not be good at using appliances and computer systems or are unfamiliar with them. 2.2

System Requirements for Elderly People

To solve these problems, we extracted the following three system requirements. 1. Barrier / useful information notification Although the ultimate solution is to remove the barriers, we propose to ICT technology to provide users who go out with advanced notification of the existence of barriers, and then indicate alternative routes in order to avoid barriers and reduce anxiety. Of course, the notified content is not restricted to barriers. Useful information such as the location of public rest rooms, slopes and places of interest would motivate the elderly to go out. 2. Easy-to-use interface The system is required to be easily operated by the elderly. Therefore, when operation is necessary, interfaces such as buttons should be of an appropriate size and well designed. Further, icons would be useful. 3. Autonomous system behavior Autonomous behavior is especially important for the elderly who tend to find information systems difficult to use. The current navigation systems for pedestrians such as NAVITIME [2] require active operation by the user, i.e., goal input, route choice, and symbol selection, etc. In a system for the elderly, however, an “agent” should actively provide information to passive users. 2.3

Proposal of Mobile Navigation System for the Elderly

To satisfy the above requirements and promote participation of the elderly in social activities, we have developed a mobile navigation system using a GPS

580

T. Kawamura, K. Umezu, and A. Ohsuga Table 1. Handicap and Will to Go

Go out actively Go out if necessary or stay at home

No disability Possible to go out alone Need Support to go out 73.5% 32.7% 0.0% 27.5%

67.3%

100.0%

cellular phone to support the elderly so that they can go out in a relaxed manner. It corresponds to a personal agent that resides in the cellular phone and automatically notifies the necessary content such as barriers to the user in the street. Also, the system provides a content-submission mechanism. It is like a social bookmarking system for the real world. Needless to say, it is practically impossible to input all content in the world and trace the changes everyday. A social mechanism for the individual users is necessary for content collection. We selected the cellular phone equipped with GPS as the system client as it will be available for widespread use in the near future, although some navigation systems are based on proprietary gadgets. In Japan, 3G cellular phones are required to be equipped with GPS from 2007 onward, and some phones customized for the elderly have sold well. Note that our target users are people who are physically able to go out, but tend to stay at home [1](Table 1). Disabled people who need physical support are outside the scope of this paper. Also, although the system we envisage would be effective for the elderly, it would be useful for other people too.

3 3.1

Mobile Navigation System System Overview

Fig. 1 shows a use case. First of all, the user starts the “agent” application on the phone, and keeps it running before going out. When the user comes to the content (barrier or significant spot) registered in a SNBI (Searching NearBy Information) server, the agent barks “bow wow” to alert the user, and displays content consisting of a picture and a brief description (Fig. 2 at right). We set a large character size and high sound volume in view of the impaired vision and hearing experienced by many of the elderly. If the user pushes the orange button when notified (Fig. 3 at left), a map of the area indicating the location of the content is displayed (Fig. 3 at right). For content submission, the user pushes a circular button for useful information or a cross button for a barrier at the location (Fig. 4 at left). Then, the built-in camera is activated (currently, 90% of cellular phones sold in Japan are equipped with a digital camera), and the user can input title, comment, tags, and time such as 9am-5pm after taking a photo (Fig. 4 at right). After completing the inputs, the user pushes the send button to register the information and the current location to the SNBI server. The content becomes available for all subsequent users who approach that location.


581

Fig. 1. Usecase of mobile navigation

Fig. 2. The “agent” and content notification

3.2

Network Architecture

The client system, which has been implemented as a Java application (iAppli) on cellular phones of NTT DoCoMo, uses NTT’s business “mopera” GPS location service. The SNBI server is implemented as a Java servlet. If the client queries the SNBI server to get the nearest content, the GPS module is automatically invoked

582

T. Kawamura, K. Umezu, and A. Ohsuga

Fig. 3. Orange button and map

Fig. 4. Content submission

and the retrieved location information is sent to the SNBI server from a DLP (DoCoMo Location Platform) server (Fig. 5). Then, the SNBI server searches the DB (content table) based on the location, and returns the content, if found. If not, the map including the current location is sent to the client. This is repeated every 2 to 3 minutes. The locations are stored in another DB (location history table) in the SNBI server, and are used to trace and predict the route. The SNBI Server is composed of FreeBSD 6.2, MySQL 4.0 and Tomcat 6.0 on Xeon 2.80GHzCwith 1024MB memory. 3.3

Content Notification Mechanism

Content selection is a key issue concerning the navigation system. If the user is not good at system operation and does not get the desired results, the system is likely to be abandoned. Therefore, unnecessary content notification and related operations must be reduced.


583

Fig. 5. System architecture

Furthermore, it is no wonder that there are many barriers and many ways around them. For example, in the case of a pedestrian bridge, there are two ways: detour or, in other words, find a crossing, or across the bridge. In that case, the notification mechanism should select proper approaches according to the user’s profile, and notify them. Note that if the way around is “across the bridge”, the “bridge” is not a barrier and should not to be notified. In this paper, we used a Bayesian network [3] to select the content. The Bayesian network estimates probabilistic parameters from a given statistical data set and a causal association model, and can be used to predict cases with uncertainty [4]. For the content selection, it’s practically impossible to collect every case for every profile, and then make rules such as in an if-then style. The Bayesian network estimates probability distribution even for missing data sets and reasons unseen cases. Fig. 6 shows our Bayesian network model and parameters for the content selection. It has environmental parameters (weather, temperature) and the user profile (purpose of going out, familiarity of the place, ability to walk a long distance, willingness to walk), and then predicts the importance of each case for the user after filtering out the locations. The location filtering is done by selecting ones located within 100 degrees ahead on the user’s route, which is simply a straight line from previous position to current position. According to this content selection based on the user profile, we would be able to do the following: if the user does not know the place well, it would be important to check the nearest police box. But, if the user goes there frequently, such notification is unnecessary. Furthermore, an escalator is of course important for

584


Fig. 6. Bayesian network for content selection

a disabled user, but stairs should be recommended to the user who tends to walk for exercise.

4 4.1

Experiment on Usability Experiment Overview

Firstly, to check the usability of our system we conducted an experiment at Ueno Park, Tokyo, for 10 days in August 2007. Test users are randomly selected from the elderly people walking in the park (Fig. 7). Since the area includes museums, ponds and many other places of interest, it is popular with the elderly. Before the experiment, we collected the barriers and 20 to 30 significant spots and registered them in advance, such as the places where street people and crows gather, stairs and steps, and benches in the shade.

Fig. 7. Snapshot of experiment


585

Table 2. Profiles of test users ID Sex, Num Age 1 F, 2p 70s 2 F, 2p 70s 3 M 60s 4 M & F 80s 5 F, 2p 80s 6 M & F 70s 7 M & F 60s 8 M, 2p 70s 9 M & F 70s 10 M 80s

Freq. of visit Disability Stairs and Steps Have a Cell Phone? Several times Legs Never No 2 to 3 Legs Never No 2 to 3 Legs Never Yes Several times None Prefer not to Yes Several times None Prefer not to No 2 to 3 None Prefer not to No First time None OK No Several times None OK No Several times None OK Yes Several times None OK No

Fig. 8. Walk routes of 10 users at Ueno Park

We first explained the purpose of this experiment and instructed the subjects in the use of the client, that is, how to get notification and how to submit content. The experiment was conducted from the user’s current position to the user’s goal, and then we orally collected the responses to a questionnaire for 5 min. As a result, we obtained the walk routes (Fig. 8) and the responses to the questionnaire for 10 pairs of elderly people (Table 2). 4.2

Experiment Results

Contents Notification. In terms of notification timing along with the user’s position, the notified content was: – in front of the user’s direction (8 cases, 24%),

586


Fig. 9. Content notification

– at the same location as the user (10 cases, 30%), – behind the user’s direction (8 cases, 24%), – misaligned such as on another street (7 cases, 21%). Also, in their responses to the questionnaire, half of the test users complained that the notifications were not provided at the right time (Fig. 9). We intend to improve this point by estimating the user’s walking speed and direction. Notification of content along with the user’s position is discussed in the next section. However, more than 40% of the users mentioned barrier notifications were useful as a whole. Therefore, although there is room for improvement, utility of a mobile navigation system that automatically notifies the proper content when the user is walking outside has been confirmed to some extent. Content Submission. In this experiment, none of the users submitted a barrier or other useful information. Major reasons are thought to be the difficulty of operation for first-time users and the relatively short distance each user walked in the experiment, resulting in a lack of interesting information to report. So we intend to consider an easier way of content submission. However, more than 70% of the users expressed a desire to submit content. Therefore, we confirmed that the elderly subjects would be willing to cooperate in order to enrich the content, provided it is made easier to do so. User Interface. Some users commented that the maps displayed on the phone were not easy to see. And some users commented that the operations are generally not difficult, as the operation for content retrieval involves pushing just one button (Fig. 10). However, the large images and large characters were appreciated, and moreover, the content was readily understood by the elderly subjects.


587

Fig. 10. User Interface Table 3. Users’ requests Content useful for notification request for notification police box street people and crows benches in the shade automatic vending machine park map toilets resting places restaurants

Users’ Impression. Table 3 shows the content whose notification the elderly subjects considered would be useful. In total, 90% of the users mentioned that this service is fun and they would like to use it again. However, there was a precondition for continuous use, that is, easier operation overall. We observed that the user’s route selection is different depending on whether the user is familiar with the locality or not. For example, users familiar with the locality walked on side roads in the shade, but those unfamiliar with the locality walked on the main street in the sun. Furthermore, we found the barriers can be Table 4. Static and dynamic barriers Static barrier steep stairs pedestrian bridge road without sidewalk no resting place steps, stairs

Dynamic barrier crowd in station bicycles on sidewalk road construction children in public space road under the sun space without people in the night street people, crows hawkers

588


classified into two categories: static barriers and dynamic ones (Table 4). The former are structural barriers such as stairs and steps, whereas the latter change dynamically with the passage of time. For example, the users who walked on the shady side of the street changed their route in the evening, because street people and crows appeared as it became darker. These observations highlighted the importance of selecting notification content according to the user profile and time. In the next section, we evaluate the accuracy of content notification.

5 5.1

Experiment on Accuracy of Content Selection Experiment Overview

We conducted a second experiment to evaluate our content selection mechanism. First, in light of the knowledge gained by means of the first experiment and the responses to the questionnaires, we created 324 pairs of user profiles, environmental parameters, the barrier information, and possible reactions as a data set for learning (Table 5). This data set was input to the Bayesian network, BayoNet. We also created 50 pairs of user profiles, environmental parameters and barrier information as a data set for evaluation. Then, the Bayesian network reasoned the proper reactions for the evaluation data set. In this experiment, any barrier for which the reaction determined was “neglect” was considered to be unnecessary content, and therefore, not notified. The other content was notified together with possible reactions. Secondly, four test users selected one of the proper reactions (as correct data) for the evaluation data set, and then we compared them with the result of the Bayesian network. 5.2

Experiment Result

The experimental result is illustrated in Fig. 11. In 35 cases (70%), the results of our content selection mechanism were the same as more than three of the test users’ answers. On the assumption that we provided six alternatives for the reactions in this experiment, the result shows high accuracy of our content selection mechanism as preliminary consideration. Table 5. Part of learning data set Weather Fine Cloudy Rain Fine Fine Cloudy Cloudy Rain

T 30C+ 5Cother other 5Cother 5C5C-

Locality No Yes No Little Little Little Yes Little

Willingness to walk not walk walk for exercise walk for exercise walk for exercise not walk not walk walk for exercise not walk

Barrier Bicycles on street stairs in station Bicycles on sidewalk stairs in station crowd in station road w/o sidewalk street people stairs in station

Reaction proceed with caution escalator detour neglect change time slot proceed with caution detour elevator


589

Fig. 11. Comparison with Bayesian network and users’ selection

However, the results of the Bayesian network were different from the answers that all of the four users chose in 4 cases (8%). The reason is thought to be that the learning data set was too small. We are planning to conduct another experiment using a larger data set. Furthermore, the results of the Bayesian network were different from the answers that two users chose in 11 cases (22%). For example, 4 cases were about the bicycles on the sidewalk. One of the reasons is different recognition of the degree of danger among the users. Therefore, in addition to building a large data set, we are considering personalization of the content selection in the future, which will reflect the user’s previous content selection and frequent visiting points retrieved by trace logs.

6

Related Works

Sakamura et al. conducted an experiment using Ubiquitous Communicator [5]. It is a dedicated terminal that reads RFID tags embedded at several places in a town in advance, and then shows information on places nearby. RFIDs have o higher sensitivity than GPS, which makes them particularly advantageous for applications within the home. However, for outdoor use the cost of embedding RFID tags throughout a city poses a considerable problem. Yairi et al. created barrier-free maps for a few towns and launched a commercial service [6]. Users use their PCs to browse Web-based online maps and they can input barriers. Further, it can create a route to a designated goal in light of those barriers. However, there is no function to notify the barrier information to users who are outdoors. Kurihara et al. are using high-resolution GPS and trying to create a barrierfree map automatically [7]. In this project, the user’s route is extracted by the high-resolution GPS attached to a wheelchair, whose error is less than a few centimeters. Then, steps and flatness are automatically detected by the trace, speed, etc. We would like to consider the use of that GPS, but in this paper we selected the GPS cellular phone because it is already in widespread use.

590


With regard to a social bookmarking feature in the real world, there are various lines of research such as Space Tag [8], xExplorer [9], Voting With Your Feet [10], Place-Its [11], etc. In those cases, GPS is used not only for information notification, but also for users’ communication and reminder systems. Although we have not described our content submission mechanism at length in this paper, we are currently considering the incorporation of those features to promote the use of the content submission mechanism by the elderly.

7

Conclusion

In this paper, we proposed a mobile navigation system for a GPS cellular phone to help the elderly go out into society. Then, we described two experiments on the system usability and the accuracy of the content notification mechanism by the Bayesian network, respectively. On the basis of the results of those experiments, we confirmed that this “agent” system is useful for the elderly in terms of its notification feature, although the interface needs further improvement. We are currently building a larger learning data set to improve the content notification mechanism and for the personalization, so that we will be able to promote the participation of the elderly in social activities.

References 1. Cabinet Office, Government of Japan: Annual Report on the Aging Society, http://www8.cao.go.jp/kourei/english/annualreport/index-wh.html 2. NAVITIME JAPAN: NAVITIME (in Japanese), http://www.navitime.co.jp/ 3. Motomura, Y.: BAYONET: Bayesian Network on Neural Network, Foundations of Real-World Intelligence, pp. 28–37. CSLI publications, Stanford (2001) 4. Pearl, J.: Fusion, propagation, and structuring in belief networks. Artificial Intelligence, 241–288 (1986) 5. Ubiquitous ID Center: Ubiquitous Communicator, http://www.uidcenter.org/index-en.html 6. Shobunsha: barrier / barrier-free map (in Japanese), http://www.mapple.co.jp/corporate/product/22.html 7. Kurihara, M., Nonaka, H., Yoshikawa, T.: Use of Highly Accurate GPS in NetworkBased Barrier-Free Street Map Creation System. In: International Conference on Systems, Man & Cybernetics, pp. 1169–1173 (2004) 8. Tarumi, H., Morishita, K., Ito, Y., Kambayashi, Y.: Communication through Virtual Active Objects Overlaid onto the Real World. In: Third International Conference on Collaborative Virtual Environments (2000) 9. Munemori, J., Tri, T.M., Itou, J.: Forbidden City Explorer: A Guide System that Gives Priority to Shared Images. In: Third International Conference on Mobile Computing and Ubiquitous Networking, pp. 248–253 (2006) 10. Froehlich, J., Chen, M.Y., Smith, I.E., Potter, F.: Voting With Your Feet: An Investigative Study of the Relationship Between Place Visit Behavior and Preference. In: Eighth International Conference on Ubiquitous Computing (2006) 11. Sohn, T., Li, K.A., Lee, G., Smith, I., Scott, J., Grisword, W.G.: Place-Its A Study of Location-Based Reminders on Mobile Phones. In: Eighth International Conference on Ubiquitous Computing (2005)

Time Stamp Protocol for Smart Environment Services Deok-Gyu Lee1, Jong-Wook Han1, Jong Hyuk Park2, Sang Soo Yeo3, and Young-Sik Jeong4 1

Electronics and Telecommunications Research Institute 161, Gajeong-dong, Yuseong-gu, Daejeon, Korea, 305-700 [email protected] 2 Computer Engineering Department, Kyungnam University 449, Wolyong-dong, Masan, Kyungnam, Korea, 631-701 [email protected] 3 BTWorks Inc., Korea [email protected] 4 Dept. of Computer Engineering, Wonkwang University 344-2 Shinyong-Dong, Iksan, Jeonbuk, 570-749, S. Korea [email protected]

Abstract. By development of wireless mobile communication, many users increased. But, in case of 1st generation or 2nd generation, transfer communication service was not satisfying high speed wireless internet Communication consumer's request such as other multimedia service because serviced based on voice and text basically. Can get through service such as data and transfer multimedia service that is not service of voice putting first in wireless hereafter. Problems by much development of service are happening, because a transmit is exposed, problem point that radio net is much unlawful stealing use and tapping etc. by user that is not right can happen. Need method to keep away purpose that is enemy of third party in contract between both men as well as problem for document or accounting information which the third user that is enemy of third party is shared. By solution about problems, certification of contents for document and visual point confirmation must it. Applied service or certification of contents service that is rapidly point of time that is using in wire to solve problem that refer in front in this treatise in smart environment to develop hereafter. Way to propose proposed efficient way using individual in smart environment just as it is.

1 Introduction Mobile communication has made rapid progress after the first and second generations, with users increasing at the same time. However, the first and second generations mobile communications do not satisfy the needs of users of high speed Internet communication such as multimedia service, as they had been developed with major focus on voice and text service. Recently, mobile communication has been providing a variety of services such as data and multimedia services including voice service. New services are also being developed. Mo-bile communication service is not limited to time and geographical F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 591–601, 2008. © Springer-Verlag Berlin Heidelberg 2008

592

D.-G. Lee et al.

restrictions, providing the convenience of voice and data service. Since it utilizes radio waves as a communication medium, however, it has weakness in security. IMT-2000 is a third generation mobile communication system. It enables most of the ser-vices available for a wired network in the wireless network and ensures quality. However, unauthorized third parties can intercept transmission and use it or malicious third parties may be able to eavesdrop through a shared transmission media since wireless communication transmission is exposed. Malicious third parties can cause this type of problem, or any user can conduct vicious acts in the wireless network. The challenge for the wireless net-work lies in preventing malicious intentions by third parties. One method is to have an authentication authority that will validate communication between users and a time verification service. This type of service should be used for communication between users and for billing in coordination with the billing company. This paper discusses how to implement the time verification service and the content certification service by applying the wired network services to the IMT-2000. When considering the current development phase of the IMT-2000, however, it is impossible to examine the entire service and its system. In the case of CDMA 1X EVDO, there is a view that it is an intermediate form between the IMT-2000 and the second generation. The other view is that it is the IMT-2000 from service provider's perspective. These two different views make CDMA 1X EVDO (Evolution-Data Optimized) difficult to define completely as IMT-2000. Nevertheless, the main purpose of the IMT-2000 is to provide services avail-able to the wireless network. With the focus on the purpose of the IMT-2000, this paper will propose the methods. Part 2 of this paper discusses the IMT-2000 and its security requirements. Part 3 discusses the authentication service. Part 4 introduces the time verification service. Lastly, Part 5 analyzes the proposed methods and gives the conclusion.

2 Related Work The smart environment was done by the development of the preexistence communication service. The smart environment is carried based on the various services, and the various communications. In this section, it inquires about IMT-2000 with the existing research about the smart environment. 2.1 IMT-2000 Overview Compared with second-generation mobile communication systems such as GSM (Global System for Mobile Communication) or IS-95 CDMA (Code Division Multiple Access), the IMT-2000 system features multimedia service and global roaming. This change in mobile communication requires information protection wherein technology should be developed to fit the new environment. Corresponding to these demands, the IMT-2000 leading groups such as 3GPP (3rd Generation Partnership Project) and 3GPP2 (3rd Generation Partnership Project2) are in the process of standardizing their technologies. In particular, the 3GPP standardizes the asynchronous method that consists of ETSI (Europe Telecommunication Standard

Time Stamp Protocol for Smart Environment Services

593

Institute), ARIB (Association of Radio Industries and Business), TTA (Telecommnication Technology Association), T1 (T1 Committee), and TTC (Telecommunication Technology Committee). Currently, various groups under the 3GPP are very active. The TSG SA WG3 (Technical Specification Group Service and system Aspect Working Group 3) are responsible for security architecture, authentication mechanism, and encryption algorithm that are related to information protection. The security features required in the IMT-2000 proposed by the 3GPP are shown below. 1. Network Access Security: It provides safe access to the 3G (3rd Generation) service and prevents attacks from a third party on the radio link. 2. Network Domain Security: It provides the protection for information transmitted and signaling information on the wired portion of the network. 3. User Domain Security: It provides safe access to the MS (Mobile Station). 4. Application Domain Security: It enables the safe transmission of messages between the user and the service provider domain. 2.2 3GPP Authentication Mechanism 2.2.1 Authentication Overview The subscriber authentication of the 3GPP is performed by proving that the USIM (Universal Subscriber Identity Module) and AuC (Authentication Center) have the same secret key. Prior to providing the service, the VLR (Visitor Location Register) uses AV (Authentication Vector) created by HLR (Home Location Register)/AuC to authenticate subscribers. If the authentication is successful, the VLR and MS share CK (Ciphering Key) and IK (Integrity Key) used in the security mode. Based on the value of the KSI (Key Set Identifier), the VLR determines whether the AKA process should be omitted. The 3GPP authentication mechanism can be described using transmitted messages. Initially, the transmitted values (Initial L3) undergo the AKA process and, finally, the Security Mode Command. The detailed process is shown below. MS transmits the Initial L3 message to the VLR during the MM (Mobility Management) connection process. Actually, the RRC (Radio Resource Control) connection configuration transpires between MS and RNC before the AKA process occurs. At this time, the MS transmits encryption algorithm and integrity algorithm to RNC (Radio Network Controller). The VLR determines whether AKA should be performed based on the KSI value because Initial L3 message contains the user ID (Identification), KSI, and LAI. As such, the ID verification process can occur between the old subscriber's network (VLRo) and the new subscriber's network (VLRn). If AKA is performed, the VLR requests the creation of the AV from the HLR/AuC. If the authentication is successful, the security mode negotiation process is initiated. 2.2.2 Authentication and Security-Related Processes This chapter discusses the authentication process of each interface in detail. First, the AKA between the VLR and the HLR/AuC is discussed. The AKA between the MS

594

D.-G. Lee et al.

(Mobile Station) and the VLR, the distribution of authentication data from previously visited location, and IMSI (International Mobile Subscriber Identity), lastly, the security mode setup process are then investigated. 2.2.2.1 AKA between VLR and HLR/AuC. If the VLR (Visitor Location Register) does not have the AV (Authentication Vector) required authenticating users, it requests a new AV from the HLR/AuC. The AV required here is a prerequisite for the authentication process of the MS, since authentication is required prior to the security mode. Thus, if the VLR does not have an AV, it transmits a request message to the HLR/ and receives the AV as a response. The transmitted message includes the identity, node type, and IMSI. After the HLR/AuC creates the authentication vector as a response, it transmits the authentication vector to the VLR. 2.2.2.2 AKA between MS and VLR. The purpose of the AKA is to authenticate the MS and the VLR and set up the new CK (Cipher Key) and the IK (Integrity Key). If the VLR sends the authentication request message including RAND (Random Number), AUTN (Authentication Token), and KSI to MS, the MS compares the MAC (Message Authentication Code) that is one of the elements that make up the AUTN and the XMAC (Expected Message Authentication Code). In case of different results of the comparison, the MS sends the authentication failure message including the reason for failure of the MAC to the VLR. In case of the same results, the MS compares the SQNMS (Sequence Number) created in USIM and SQN as another element of AUTS (Authentication Synchronization Failure Parameter). After determining whether the SQN is within the valid range, the MS sends the authentication response message, i.e., the success response message of the terminal for the network authentication, or it sends the authentication failure message that the SQN is not within the range to the VLR. If the MS authenticates the network, it computes the RES as a response. The RES (Response), along with the authentication response message, is transmitted to the VLR and compared with the XRES (Expected Response) to see if they are identical. This comparison allows the network to authenticate the terminal and to complete the authentication process. Specifically, it allows the authentication to reject the message for unsuccessful authentication to be transmitted to the MS. 2.2.2.3 Process for Distributing IMSI and Authentication Data from Previously Visited Network and Identifying Them. This process occurs when moving the authentication data to the newly visited network, VLRn, from the previously visited network, VLRo, on the identical SN (Serving Network). Right after the VLRn receives the Initial L3 message containing the TMSI and the LAI from the MS; it transmits a message to verify the TMSI from the VLRo. The VLRo retrieves the information related to the TMSI from its own database and transmits the CK and IK to VLRn, i.e., it transmits the TMSI check failure message. If the identification verification process mentioned earlier fails, the VLRn uses the IMSI to verify the MS and the identification. 2.2.2.4 The Security Mode Setup Process. If the AKA is successfully completed or is omitted because of the KSI verification, the security mode negotiation process is initiated. Many negotiation processes between the MS and the RNS such as choosing


595

the encryption algorithm used during the process occur. This paper does not discuss these processes in detail because they have less relevance to the subject of the paper. However, the check for the integrity of the signaling message as a requirement is considered.

3 Notarization Service While normal transactions occur not only in real life business but also in the network, there is no special problem. However, solutions must be established in case a problem occurs with the transactions. For example, ways of checking the eligibility of the opposite party and documenting the business transactions (Contracts, Orders, Memorandum, etc) must be considered. The time verification service contained in this notarization service must meet all notarization requirements. The notarization service is provided in the presence of a notary. The time verification service includes the function of verifying the time and focuses on the time verification rather than the presence of a notary. In real life, a certificate with a seal impression and a register book can be used to validate the opposite party. These documents are official and legally backed by authorities. For existing transactions, the opposite party can also be verified with a telephone call. In the case of e-commerce, resolving the threats from the third party is presumably required. At the same time, trust from the opposite party should be acquired. This paper defines electronic notarization as a concept of obtaining the safety and trust of electronic exchange by proving the occurrence of it for who/when/what in the business transactions of the network. Specifically, electronic notarization maintains the trust of the parties concerned and plays a central role in implementing reliable transactions. It in-creases the provability and provides a useful means of preventing or resolving disputes. Electronic notarization function includes the specification of sender/recipient, delivery confirmation, alteration detection, visual electronic preservation, access log, and process log. The requirements for electronic notarization are shown below. They are identical to those of the time verification service. 1. Authentication: Authentication is carried out to enable the third party to prove who has provided what information to whom. 2. Integrity: The contents delivered between the parties concerned must not be revealed and altered during the delivery. 3. Delivery Confirmation: There must be a way to ensure that the digital information of the sender has been delivered to the recipient. 4. Readability: It means the ability of rendering the contents of data readable depending on the needs. 5. Resolvability: Data must be stored in the format that can be restored within the preservation period. 6. Timing: There must be a way to verify the information or examine the accuracy of date time data. 7. Responsibility: The most important element of responsibility is to seek safety. The responsibility for the operation must be clarified.

596

D.-G. Lee et al.

4 Time Verification Service in IMT-2000 This chapter proposes the PKI-based time verification service structure that can be applied in the IMT-2000 environment. 4.1 Elements The elements used in the proposed method are described below. The proposed method consists of the USIM, ME, SN, and AuC. The feature of each element is described below. • AuC (Authentication Center) / HLR (Home Locater Register): It sends the public key certificate to the user. • NA (Notary Authority): If the communication request from the AuC is completed, the NA receives a key from the AuC and creates a session key based on the received key. • SN (Service Network) / VLR (Visitor Locator Register): When ME0 and ME1 perform encrypted communication; the encrypted contents are stored in the SN. • ME (Mobile Equipment): It requests communication with the other MS and stores a key received from the NA in the USIM. • USIM (Universal Subscriber Identity Module): It stores the key. 4.2 System Values The proposed method uses the following figures: • CKey: Key created in the AuC • NKey: Key created in the NA • EKey: Session key used between ME's. • Res: Response value • ID*: Identity (*: ME: Mobile Equipment, AuC: Authentication Center, NA: Notary Authority) • H(): Hash Function • PKME0, SKME0: Public and private keys of ME0 • PKME1, SKME1: Public and private keys of ME1 • PKAuC, SKAuC: Public and private keys of AuC • PKNA, SKNA: Public and private keys of NA • SigAuC, SigME0, SigME1: Signature of AuC, ME0, ME1 • TSME0, TSSN, TSAuC: timestamp of ME0, SN, AuC 4.3 Proposed Method Figure 4 schematizes the protocol of the proposed method. The entire protocol consists of three stages: requesting communication, transmitting and distributing a key, and providing messages. The stage of requesting communication is the time when communication is started by ME0. If the completion message is delivered for the communication request, the AuC creates a key to deliver to the ME and transmits the message based on this key. If unexpected activities are found in the message afterwards, the time verification service is provided.


597

Based on Figure 3, the detailed description of the protocol is provided as follows: 1) Requesting communication The description of the request and the response needed for first time communication is provided below. This stage shows the initial user's request for communication from the other party for the time verification service and the other party's acceptance of the request. This stage is described as a general request stage. Initially, each user passes the authentication through the AuC, and, without going through the authentication process, receives the service through the authentication of the AuC.

① The ME requests communication with the ME from the SN . ② The SN requests the location of the ME from the AuC. ③ The AuC requests the information of the SN and the communication with the ME from the SN . ④ The SN verifies the request of the ME for communication with the ME . ⑤ The ME responds to the communication verification. ⑥ The SN transmits response results to the AuC. ⑦ The AuC transmits the response for the request to the SN . Lastly, The SN 0

1

0

0

1

0

1

1

1

0

1

1

1

0

0

transmits the response to the ME0.

2) Generating a Key Key creation required for message encryption and time verification in the proposed method is described below. At this stage, the required key receives the time verification service. Using the provided key, support such as the point of time verification and content certification will be provided.

① If ⑥ Stage of requesting communication is completed, the AuC informs that

correct communication with the ME0 and the ME1 has been made and request key was created. The AuC creates the CKey(formula (1)) and transmits the message formula (2) to the NA.

②

CKey = H(IDAuC || IDME0 || IDME1 ||SKAuC ) PKNA (CKey || SigAuC (H(CKey)))

(1) (2)

③ The NA responds to the transmission of the CKey(formula (1)) from the AuC.

The stage of requesting communication and distributing the key is completed.

3) Message transmission with key distribution, transmission, and time verification The proposed method specifies the message transmission stage after the key distribution stage. If a user who wants to transmit a key created from the AuC and the NA using the received key of the NA encrypts the message and appends the timestamp for time verification to the message, the SN of each user in the middle stores the message. If a problem occurs, the time verification or content certification service is provided upon the user's request. Using the Ckey(formula (1)) received from the AuC, the NA creates the Nkey(formula (3)) and Key(formula (4)).

①

NKey = H(IDNA || SKNA ) Key = H(IDME0 ||IDME1 ||CKey||NKey)

(3) (4)

598

D.-G. Lee et al.

② The created key transmits formula (5) and formula (6) based on the Me to the SN0 and the SN1.

PKME0 (IDME0 ||Key || SigAuC (IDME0 ||Key)) PKME1 (IDME1 ||Key || SigAuC (IDME1 ||Key))

③ ④ ⑤

(5) (6)

Again, the SN0 and the SN1 transmit formulas (5) and (6) based on the ME0 and the ME1. The ME0 and ME1 store the received key in the USIM. The ME0 encrypts the message M with Key and transmits the message. EKey (IDME0 || M || TSME0, SigME0 (H(IDME0 || M || TSME0 )))

⑥ After the SN

(7)

0 stores the transmitted formula (7), it transmits the formula to the SN1. When storing, the SN0 attaches the timestamp with its own secret key.

SigSN0 (EKey (IDME0 || M || TSME0, SigME0(H(IDME0 || M || TSME0 )))|| TSSN0 )

⑦ Likewise, after the SN

(8)

1 stores the transmitted formula (7), it transmits the formula to the ME1. The SN1 also attaches the timestamp and stores the formula.

SigSN1 (EKey (IDME0 || M || TSME0, SigME0 (H(IDME0 || M || TSME0 )))|| TSSN1 )

(9)

⑧ After the SN stores the formula, it sends off the verification message to the NA. ⑨ After the ME verifies the received formula (7), it sends the verification 1

1

response to the SN1.

EKey (IDME0 ||RES, SigME1 (IDME0 || RES))

(10)

⑩ The SN sends the response to the SN , which then sends the response message to the ME . ⑪ The ME sends the response verification message and completes the entire 1

0

0

0

process.

5 Analysis of the Proposed Method This chapter analyzes the time verification, certification stage, and the security details of the proposed method. 5.1 Time Verification and Content Certification This chapter discusses the time verification stage of the proposed method. When a message is altered for any reason, a user can request the confirmation for the time verification and content certification from the NA. In this case, confirmation can be requested by a user or all the users involved. The NS is responsible for the final alteration.

① The uncertainty check for the message is performed upon the request of the ME. ② If a request for the uncertainty check is received from a user, the NA requests

the Ckey(formula (1)) from the AuC. Using the Ckey and Nkey(formula (3))


599

transmitted from the AuC, the NA creates a Key(formula (4)) and transmits the Key(formula (4)) to the SN stores the message. AuC -> NA: CKey H(IDAuC ||IDME0 || IDME1 || SKAuC ) NKey H(IDNA || SKNA ) Key H(IDME0 || IDME1 || CKey || NKey )

(1) (3) (4)

③ Each SN performs decryption using the delivered key. The users can use the

time-stamp TS(Time-Stamp) of SN stored in the SN to receive the confirmation for the transmitted time and the content certification.

SigSN0 (EKey (IDME0 || M || TSME0, SigME0 (H(IDME0 || M || TSME0 )))|| TSSN0 ) SigSN1 (EKey (IDME0 || M || TSME0, SigME0 (H(IDME0 || M || TSME0 )))|| TSSN1 )

(8) (9)

5.2 Consideration of Proposed Method The proposed method prevents eavesdropping and alteration on the transmission path using the public key of each object or the session key. Moreover, it prevents each object from maliciously corrupting other objects. In case a user consults with the server, it cannot be created on the server since the CKey is a hash value created from the user ID and the private key of the AuC. Thus, it can prevent illegal consultation between a user and the server. Even if the SN has illegal intentions, it cannot verify the message until a confirmation request is made since it would not be able to obtain the key delivered from the NA. The time verification can use the encrypted message stored in the SN to provide the con-tent certification service to the users or legal authority. Table 1 performs the analysis according to the proposed method. The results are shown below. In examining each part, the authentication for the AuC was carried out and the communication request with the other party is made since the authentication service is initially pro-vided through AuC. Thus, if the authentication with the AuC is correct, the authentication with the party concerned is passed. Integrity is implemented through the signatures of the parties concerned. At this time, threats from the insufficient communication path or threats from the sender/recipients can be properly handled using the keys created in the HE and the NA. Likewise, data for the delivery certification stored in the SN can be decrypted using the key of the NA and the AuC when resolving future disputes between the parties concerned. The delivery certification can be verified in the SN where the parties concerned belong to because it is stored in each SN after the delivery of documents. These documents can be retrieved when a problem occurs in the future. Readability renders the data readable depending on the contents. Thus, the parties concerned should be able to read data. This feature allows the AuC and the SN to use the created key to read data upon the request of the parties concerned. Preservability can determine how long data should be preserved depending on its importance. The party can determine when a message is transmitted through the timestamp. The timing verifies when data was transmitted. It can verify whether the data has been changed or altered by attaching the timestamp whenever the data undergoes each element.

600

D.-G. Lee et al. Table 1. Proposed Scheme Analysis

Proposed Scheme Authenti cation

AuC according to existent certification system each person concerned certification

Integrity

EKey(IDME0 || M || TSME0, SigME0 (H (IDME0 || M || TSME0 )))

delivery receipt

SigSN0(EKey(IDME0 || M || TSME0, SigME_0 (H(IDME0 || M || TSME0 )))|| TSSN0 )

continua nce

EKey(IDME0 || M || TSME0, SigME0 (H(IDME0 || M || TSME0 )))

responsi bility

CKey=H(IDAuC || ||SKAuC )

IDME0

||

IDME1

Analysis This can reduce the authentication process between users by verifying with the party concerned. Encryption using the signatures between the parties concerned according to the delivered messages and keys created by AuC and NA Delivery certification can be verified by storing the following message in the SN and through the verification process as necessary. Each timing is verified by comparing the elements using the TS at each delivery point. The creation and delivery of a key is done in the NA because the key of the AuC is created at the request of the NA. The NA and AuC are responsible for the operation.

Lastly, the AuC and NA are responsible for the operation. Likewise, the SN along with the AuC has the authentication and trust system. Thus, the AuC and NA are responsible for all operations excluding illegal ones between related parties.

6 Conclusion To date, mobile communication has provided many services since the first and second generations. Preparing for future services, the third generation anticipates not only multimedia services but also many other services. However, problems as the contract between users, billing time issues, and content certification between a user and content provider can occur. To resolve these problems, the time verification service with notary authority has been proposed. Since time verification services can provide not only the notarization


601

ser-vice but also the content certification service, the key can be obtained from the notary authority to validate the content when a user raises an issue about the records. This paper introduced the part of IMT-2000 service in the future. The services mentioned here are expected in the near future. At the same time, more services providing convenience and safety are foreseen.

References [1] 3GPP TS 33.102 : 3rd Generation Partnership Project(3GPP); Technical Specification Group Services and System Aspects; 3G Security; security Architecture [2] 3GPP TS 22.022 : 3rd Generation Partnership Project(3GPP); Technical Specification Group Services and System Aspects; Personification of UMTS Mobile Equipment (ME); Mobile functionality specification [3] 3GPP TS 33.103: 3rd Generation Partnership Project(3GPP); Technical Specification Group Services and System Aspects; 3G security; integration Guidelines [4] 3GPP TS 33.105: 3rd Generation Partnership Project(3GPP); Technical Specification Group Services and System Aspects; 3G security; Cryptographic Algorithm Requirements [5] 3GPP TS 33.120: 3rd Generation Partnership Project(3GPP); Technical Specification Group Services and System Aspects; 3G security; Security Principles and Objectives [6] 3G TR 33.901: 3rd Generation Partnership Project(3GPP); Technical Specification Group Services and System Aspects; 3G security; Criteria for cryptographic algorithm design process [7] 3G TR 33.902: 3rd Generation Partnership Project(3GPP); Technical Specification Group Services and System Aspects; 3G security; Formal Analysis of the 3G Authentication Protocol [8] 3G TR 33.908: 3rd Generation Partnership Project(3GPP); Technical Specification Group Services and System Aspects; 3G security; General Report on the Design, Specification and Evaluation of 3GPP Standard Confidentiality and Integrity Algorithms [9] ETSI SAGE: Security Algorithm Group of Experts(SAGE); General Report Design, Specification and Evaluation of the MILENAGE Algorithm Set: An Example Algorithm Set for the 3GPP Authentication and Key Generation Functions [10] ESTI SAGE: Specification of the MILENAGE Algorithm Set: an Example Algorithm Set for the 3GPP Authentication and Key Generation Functions f1,f1^* ,f2,f3,f4,f5 and f5^*. Document 1: Algorithm Specification [11] ESTI SAGE: Specification of the MILENAGE Algorithm Set: an Example Algorithm Set for the 3GPP Authentication and Key Generation Functions f1,f1^* ,f2,f3,f4,f5 and f5^*. Document 2: Implementers’ Test Data [12] ESTI SAGE: Specification of the MILENAGE Algorithm Set: an Example Algorithm Set for the 3GPP Authentication and Key Generation Functions f1,f1^* ,f2,f3,f4,f5 and f5^*. Document 3: Design Conformance Test Data [13] ITU: ITU-R Security Principles For International Mobile: Telecommunications-2000 (IMT-2000) Recommendation ITU-R M.1078 [14] ITU: Evaluation Of Security Mechanisms For IMT-2000: Recommendation ITU-R M.1223

An Analysis of the Manufacturing Messaging Specification Protocol Jan Tore Sørensen1 and Martin Gilje Jaatun2 1

2

mnemonic as, NO-0167 Oslo, Norway SINTEF ICT, NO-7465 Trondheim, Norway [email protected]

Abstract. The Manufacturing Messaging Specification (MMS) protocol is widely used in industrial process control applications, but it is poorly documented. In this paper we present an analysis of the MMS protocol in order to improve understanding of MMS in the context of information security. Our findings show that MMS has insufficient security mechanisms, and the meagre security mechanisms that are available are not implemented in commercially available industrial devices. Keywords: Process control; Industrial Networks; Protocols; Security.

1

Introduction

We will in this paper present an analysis of the Manufacturing Messaging Specification (MMS1 ) protocol and propose improvements based on our findings. MMS is defined in the ISO 9506 standard [1], and there is also some information available in white-papers from SISCO2 [2,3,4]. We have tested the MMS protocol as implemented in a major brand of controller used in the process control industry. We will in the following look closer into the construction of MMS packages and how they may be altered and forged. MMS is an application-layer protocol which specifies services for exchange of real-time data and supervisory control information between networked devices and/or computer applications. It is designed to provide a generic messaging system for communication between heterogeneous industrial devices, and the specification only describes the network-visible aspects of communication. By choosing this strategy, the MMS does not specify the internal workings of an entity, only the communication between a client and a server, allowing vendors full flexibility in their implementation. In order to provide this independence, the MMS defines a complete communication mechanism between entities, composed of [3]: 1. Objects: A set of standard objects which must exist in every conformant device, on which operations can be executed (examples: read and write local variables, signal events). 1 2

In this paper, MMS does not stand for Multimedia Messaging Service, as is often the case elsewhere. System Integration Specialists Company - not to be confused with Cisco.


An Analysis of the Manufacturing Messaging Specification Protocol

603

2. Messages: A set of standard messages exchanged between a client and a server station for the purpose of controlling these objects 3. Encoding Rules: A set of encoding rules for these messages (how values and parameters are mapped to bits and bytes when transmitted) 4. Protocol: A set of protocols (rules for exchanging messages between devices). MMS composes a model from the definition of objects, services and behavior named the Virtual Manufacturing Device (VMD) Model. The VMD uses an object-oriented approach to represent different physical industrial (real) devices in a generic manner. Some of these objects are variables, variable type definitions, programs, events, historical logs (called journals) and semaphores. Along with the definition of these objects, MMS defines a set of communications services that an application can use to manipulate these objects. We observe that in the literature the terms services, service primitives and messages are all used to describe the functions that manipulate objects or their attributes. We will therefore in this paper use the term service primitive as this is used in the ISO 9506 standard [1], unless we are citing directly from a written source, in which case the quote will be evident in the text. The standard also refers to physical industrial devices as “real devices” and we will continue to use this terminology to avoid confusion. As MMS is based on an object oriented approach, we will give a brief introduction to the addressing and object hierarchy of MMS, before focusing on the network communication. 1.1

Architecture and Addressing

The MMS architecture is based on a common client server model. Real devices used in industrial networks often contain an MMS server allowing the device to be monitored and managed from an MMS client. An MMS client is typically part of an control builder application or an MMS-to-OPC gateway (MMS/OPC GW). A control builder is an application used to program and monitor industrial controllers. Both the control builder and the MMS/OPC GW use service primitives provided in the MMS to manage devices containing MMS servers. This is depicted in Fig. 1 [2]. As MMS does not specify how to address clients and servers, an entity containing an MMS client or server must rely on the addressing scheme of underlying protocols in the process of establishing an application association to support the MMS environment[1]. In practice, clients and servers are addressed by their IP address and the MMS server uses port number 102. The addressing allows for an MMS context to be negotiated between two peer applications. To address an MMS object variable, MMS provides several different address modes. MMS allows an address to have different syntax, based on the implementer’s choice of what is most appropriate for that device. The specification separates between named and unnamed variables.

604

J.T. Sørensen and M.G. Jaatun

Fig. 1. The VMD model depicting communication between an MMS client and an MMS server

1.2

MMS Objects, Services Primitives and Access Control

Associated with each object is a set of variables that describe values in a given instance of the object. For each object there are corresponding MMS service primitives that allow client applications to access and manipulate those objects. The top level object in the MMS is the VMD which has at least one networkvisible address. Each real device is represented by a real object with vendor-specific features associated with them. The VMD model maps the real object and devices onto virtual objects and devices, described in a generic manner which is in conformance to the VMD model. In other words, a real variable is an element of typed data that is contained within a VMD object. An MMS variable is part of a virtual object that represents a mechanism for the MMS client to access the real variable. The MMS server containing the virtual MMS object can be understood as a communication driver which hides the specifics of a real device from the client. From the client’s point of view the virtual MMS variable represents a pointer or an access method to the real variable and it is only the MMS server with its objects and its behavior that is visible to the client. The MMS client can never interact with real device variables directly. All MMS objects contain an access method variable. This attribute contains the information which a device needs to identify the real variable as described above. It contains values which are necessary to find the memory location of the real variable with the contents that lie outside MMS. A special method, the method PUBLIC, is standardized for accessing the real variables.


605

Table 1. The basic methods inherited from the VMD object MMS General Description methods Get This method is used to obtain the value of a specified object. Set This method is used to write/put value or contents into a specified object. Query Attributes This method is used to obtain structure or capability information of a specified object. Create This method allows objects of particular classes to be instantiated. Rename This method allows instantiated objects to be renamed. Delete This method allows instantiated objects to be destroyed.

For each object there are corresponding MMS service primitives that allow client applications to access and manipulate those objects. The MMS defines the service primitives of both clients and servers, but the VMD focuses only on specifying the network-visible behavior of MMS servers. And thus, each vendor of an MMS server device is responsible for hiding vendor specific details of the real objects and devices by providing an executive function which maps the real entities up to the virtual level, which shall comply with the VMD model definitions. To ensure vendor implementation compliance with the VMD model, it specifies how MMS devices containing an MMS server shall provide a consistent and well-defined view of the object contained in the VMD. And thus, MMS provides a common interface for communication with different devices through the generic virtual objects. All MMS objects except the Operator Station object inherit six abstract services from the VMD object. These are depicted and described in table 1. E.g. service primitives read and RequestDomainUpload for the objects Named Variable List and Domain respectively inherit from the abstract service primitive get. MMS uses access control lists to provide explicit control of the ability to access or alter MMS objects. Protection requirements for an MMS variable are inherited from the underlying real variable in the real device. These requirements are established by the access method in the MMS object. ISO 9506 [1] states that each object within an MMS implementation must contain a reference to an Access Control List object that specifies the conditions under which services directed at the named object may succeed. For the purposes of specifying the control conditions, services are grouped into six classes as described in table 1. Access control is enforced through special mechanisms provided by MMS. These mechanisms include possession of a semaphore, identity of user (Application Reference), and the submission of a password (which may be arbitrarily complex). 1.3

Network Services

As we have stated earlier, MMS is not by itself a communication protocol, as it only defines messages that have to be transported by an unspecified network.

606


MMS was originally developed as a part of the MAP specification [5], and is therefore specified on all seven OSI layers as depicted in figure 3. MAP was originally created by General Motors as an internal standard for communications in industrial automation networks. It is now a public, multivendor communications standard for industrial automation equipment. MMS supports the use of both confirmed and unconfirmed services, but we will in this paper focus on the confirmed services. The MMS defines the following Protocol Data Unit (PDUs) for a confirmed service exchange: – – – – – – –

Confirmed-RequestPDU Confirmed-ResponsePDU Confirmed-ErrorPDU Cancel-RequestPDU Cancel-ResponsePDU Cancel-ErrorPDU RejectPDU

These messages will be used in the communication between a MMSclient and server. When a client wishes to invoke a service primitive on the server side application (the VMD), the transitions depicted in Fig. 2 may occur, depending on the responce from the VMD. Before a service primitive is called through a Confirmed-RequestPDU, the server is in a Responder Idle state. Upon receipt of a Confirmed-RequestPDU (1) for any of the confirmed services, the MMS-provider issues an indication primitive to the application, specifying the particular service being requested and an invoke ID that specifies the service instance and enters the state Service Pending Responder. Upon receipt of a response service primitive, from the overlaying application, containing a result parameter specifying the service previously indicated and an invoke ID that identifies the service instance, the MMS-provider sends a Confirmed-ResponsePDU (2) which specifies the service type and the invoke ID from the response primitive along with the requested data. Then a state transition into the Responder Idle state occurs. If the application can not provide the requested data, the MMS-layer will receive a response service primitive containing a Result parameter specifying an errorstate along with the invoke ID used to identify the the requesting MMS service instance. The MMS-layer then sends a Confirmed-ErrorPDU (3) containing the service type and the invoke ID from the response primitive. When sending the Confirmed-ErrorPDU a state transition back into the Responder Idle state occurs. Upon receipt of a Cancel-RequestPDU from the client containing the invoke ID of the matching service instance request, the MMS-layer on the server issues a cancel indication service primitive to the overlaying application. This indication specifies the invoke ID of the service request to be canceled; this information is obtained from the Cancel-RequestPDU parameters. The state Canceling Service Responder (4) is then entered. Now two things may occur on the server side: 1) If the application has provided the requested data, the server sends a


607

Fig. 2. The MMS Confirmed Service Request as seen by the Service Responder (server)

Cancel-ResponsePDU containing the data and Confirmed-ErrorPDU entering the Responder Idle state through transition 6 (T6). 2) If the application has not provided the requested data the server sends a Cancel-ResponsePDU without any data and enters the Service Pending Responder state through T5. The server then sends a Confirmed-ErrorPDU through T3 and returns to the Responder Idle state. According to [1] the MMS runs on the network stack depicted in Fig. 3. Like all ISO standards this network stack relates to the Open System Interconnection stack describing the abstract service layers such as session and presentation layer. We will now give a short description of some of the protocols/layers based on their relevance to this paper theme.

2

ASN.1

MMS uses ASN.1 to encode data at the OSI presentation layer before transmission. The ASN.1 representation of data is independent of machine-oriented structures and encodings and also of the physical representation of the data (referred to as transfer syntax in communication terminology). MMS uses BER to encode ASN.1 data before transmission. As we will be decoding BER code, we will explain BER encoding in the next section.

3

BER

The Basic Encoding Rules are one of the original encoding rules specified by the ASN.1 standard for encoding abstract information into a concrete data stream. The rules, collectively referred to as a transfer syntax in ASN.1 parlance, specify

608


Fig. 3. The MMS communication stack specified as OSI layers

the exact octet sequences which are used to encode any given data item before it is transmitted over a network. The BER syntax is defined by the ITU-T’s X.690 standards document, which is part of the ASN.1 document series. BER is a self-identifying and self-delimiting encoding scheme, which means that each data value can be identified, extracted and decoded individually [6]. Each data element is encoded using a triplet consisting of a type identifier (tag), a length description and the actual data element. The use of such a triplet for encoding is commonly referred to as a tag-length-value (TLV) encoding. A generic triplet is depicted below. [identifier (tag)] [length (of the contents)] [contents] The use of TLV encoding allows any receiver to decode the ASN.1 information from an incomplete information stream, without any pre-knowledge of the size, content or semantic meaning of the data. Assuming that the communicating parties share the same context specific module definitions. BER uses a unique code as an identifier for an ASN.1 data type. This identifier is encoded as one or more bytes of every data type and creates the tag. The identifier is well-structured to allow the representation of three levels of Table 2. Description of the BER identifier Bit number 7 0 0 1 1

6 5 4 3 2 1 0 Implication 0 UNIVERSAL 1 APPLICATION SPECIFIC 0 CONTEXT SPECIFIC 1 PRIVATE 0 primitive data type 1 constructed data type X X X X X numeric identifier


609

information within one such code, as illustrated in table 2. On the highest level, represented by the highest-order two bits of the tag octet(s), the class of the data type is encoded [6]. The third highest bit of the identifier indicates whether the represented data type is a primitive or constructed one. A constructed data type can be seen as a complex or compound data type hierarchically based on one or more primitive data types. The remainder of the identifier is a numeric tag associated with a data type within a class. Tags ranging from 0 to 30 can be associated with the remaining 5 bits of the octet. For larger tags, these 5 bits are set to 111111, and one or more subsequent octets are used to encode the tag.

4

Analysis of MMS Communication

ISO 8823 states that the OSI transport protocol exchanges information between peers in discrete units of information called Transport Protocol Data Units (TPDUs) [7]. This is a fundamental difference between the TCP and the network service expected by Transport Protocol Class 0 (TP 0). The difference is that TCP manages a continuous stream of octets, with no explicit boundaries, while TP0 expects information to be sent and delivered in discrete objects termed network service data units. Therefore RFC 1006 [8] describes that all TPDUs shall be encapsulated in discrete units called TPKTs. The TPKT layer, depicted in figure 4, is used to provide these discrete packets to the OSI Connection-Oriented Transport Protocol (COTP) on top of TCP. We have intercepted some packages using Wireshark. As Wireshark did not support the MMS protocol at the time of testing, we were forced to manually decode the MMS PDUs. When looking at MMS communication in Wireshark we found the underlying MMS protocol stack depicted in 4. As there was no publicly available documentation on how the vendor had implemented the MMS protocol we had to analyse the protocol stack. As we see in Fig. 4, there are two protocols running on top of TCP. Above TCP we find TPKT, which is a packet format used to emulate the ISO transport services COTP on top of TCP. RFC 1006 [8] describes how to implement ISO’s transport protocol class 0 on top of TCP. ISO 9506 [1] stipulates the use of OSI transport class 4 in conjunction with MMS. Nevertheless RFC 1006 [8] describes the use of OSI transport class 0 to emulate an ISO Transport Service on top of the TCP.

Fig. 4. The MMS communication stack as Wireshark detects it

610


Fig. 5. Format of TPKT header

Fig. 6. Format of COTP PDU

The reason for using ISO’s OSI transport class 0 on top of TCP/IP instead of transport class 4 is that transport class 0 achieves identical functionality as transport class 4 when running on top of TCP. The TCP layer provides reliable transport service through error detection and retransmission. It also handles segmentation and reassembly of PDUs. As TCP provides all these properties as part of its service to the next layer, there is no reason to implement them again. A TPKT consists of two parts: a packet-header and a TPDU. The format of the header is constant regardless of the type of packet, as illustrated in Fig. 5. The field labeled vrsn is the version number which according to RFC 1006 [8] always is three. The next field, reserved, is reserved for further use. The last field is the packet length. This field contains the length of entire packet in octets, including packet-header. The maximum TPDU size is 65531 octets, with a payload of maximum 65524 octets. According to Wireshark we find COTP above the TPKT layer. RFC 0905 [9] describes the ISO 8073 specification. The COTP PDU is described in Fig. 6. The header length in octets is indicated by a binary number in the length indicator (LI) field. This field has a maximum value of 254 (1111 1110)3 . The next field is divided into two parts, first the PDU type specification (T), which describes the structure of the rest of the PDU, e.g., Data Transfer (1111) as described in Figure 6. The PDU type is encoded as a four bit word. The full list of codes for data types can be found in [9]. The second part, is the credit part (CDT) which is used to indicate a reliable transport service, but this is always set to 0000 as TP 0 does not offer reliable transport. The third field contains the TPDU number and an end of transfer indication flag. In all data transfer packets 3

The value 255 (1111 1111) is reserved for possible extensions.


611

the EOT flag is set and the TPDU number is zero; this might be because the service relies on TCP sequence numbering on the transport layer, but we have not found any written documentation to support this theory. We wish to note that there is no reference to ACSE in our packet dump. We verified through Wireshark’s documentation that ACSE is a supported protocol [10]. That leaves us with two possible conclusions: 1. The MMS protocol uses an implementation of ACSE which is not in conformance with the standard, which leaves Wireshark unable to decode the packet layer. 2. The implementors of the MMS protocol have omitted the ACSE layer when implementing the protocol. We are fully capable of decoding the whole payload of the COTP PDU to MMS structured ASN.1 text. We therefore conclude that the current implementation of the MMS has omitted the ACSE layer, making the ACSE authentication facilities forfeit. This means that there are no authentication or access control facilities at the lower layers of the MMS stack.

5

Decoding MMS Communication

Now knowing the underlying protocols which MMS is running on, we will study the MMS message communication between the MMS client and the MMS server and try to determine if there are any signs of security mechanisms. We used the client software to create a small program which we downloaded to the controller over MMS. The program was a very simple counting application as decribed in C code below: int i=0; while(TRUE) i=i+1; The controller will now report the value of i back to the client at regular intervals using MMS. Once the program was downloaded to the controller and running, we used Wireshark to capture MMS communication on the network. The first thing we noticed when we examined the packet dump in Wireshark, was that there is a pattern in packet communication repeating itself in a period of eight. This pattern was first identified by packet sizes repeating themselves at a period of eight. We wish to note that we chose an arbitrary packet in our packet dump as our starting point and decoded packages sequentially from that point. We chose this strategy to simulate an attacker tapping into a network at an arbitrary point in time.

612


5.1

Decoding the First PDU

We will now look closer at the first PDU and attempt to decode it. We have extracted the payload of the COTP package at our randomly chosen starting point. We know from the manufacturer’s web page that the equipment employs MMS. a0 80 11 4f 35

41 08 24 52 32

02 24 48 4d 36

01 4d 57 30 35

7b 53 53 11 38

a4 47 34 a0 39

3c 24 35 0f 36

a1 31 38 80

3a 24 35 0d

a0 24 34 24

38 30 33 4d

30 15 32 53

0c a0 30 47

a0 13 3a 24

0a 80 4e 35

The package above is encoded in BERs TLV format. We know this from [2] and [3] which are whitepapers publicly available on the internet. We must therefor use the decoding rules described in section 3 to decode each TLV pair. When decoding this first PDU, with the help of the MMS syntax module[11], we found that this is a confirmed-Request PDU. This confirmed-Request PDU contains an integer id named invokeID with value 123 and confirmedServiceRequest for an read operation. The read-request specifies a listOfVariables with three items. Each item is a vmd-specific object name containing the identifier. We decoded these identifiers to: – $MSG$1$$ – $HWS45854320:NORM – $MSG$55265896 Space does not permit going through the entire decoding process, but for illustrtive purposes, the beginning of the first PDU is decoded below. Using table 2 we decode the first PDU to the following textual ASN.1 structure: 1

3

5

a0 41

CONTENT SPECIFIC c o n s t r u c t e d nr 0 LENGTH=65 02 01 7b

7

9

UNIVERSAL p r i m i t i v e nr 2 (INTEGER) LENGTH=1 123

a4 3c

CONTENT SPECIFIC c o n s t r u c t e d nr 4 LENGTH=60

11

13

15

a1 3a

CONTENT SPECIFIC c o n s t r u c t e d nr 1 LENGTH=58 a0 38


17

30

UNIVERSAL c o n s t r u c t e d nr 16 (SEQUENCE)


0c

19

LENGTH=12

a0 0a

21

613


23

80 08

25

24 4d 53 47 24 31 24 24

27

29

31

33

35

CONTENT SPECIFIC p r i m i t i v e nr 0 LENGTH=8 $ M S G $ 1 $ $

. . . etc .

We observe that MMS utilizes many CONTENT SPECIFIC tags to identify MMS specific data types. As stated earlier we can use the MMS module definition to decode these tags. This module is publicly available for anyone at [11] or TM through Google .

6

Security in MMS

After analysing MMS protocol communications we will in this section look into the security mechanisms defined by the MMS standard. The standard does specify means for access control through accessControlList objects. We quote from the ISO standard [1]: T he &accessMethod field for an Named Variable object shall specify the mode of access. If the Address is declarable (and obtainable) using MMS services, the &accessMethod field shall have the value public, and the Address attribute shall be defined and available to MMS clients requesting the attributes of the Named Variable object. Otherwise, the value of this field is a local issue. The public access method shall not be available unless vadr is supported. From the quote above we see that each Named Object Variable has an accessMethod field, which specify the mode of access. According to the standard, if the address is declarable and obtainable using MMS service primitives the accessMethod shall have the value public. Access to all objects can be controlled by a special object, the Access Control List, that tells which client can read, delete or modify the object. On a general level MMS specifies that if the accessMethod is public the following field shall appear and if the &accessMethod is anything but public, the following field shall not appear. But there are some exceptions.

614


An MMS server may declare an MMS variable that exists only at the instant of access. Such a variable does not have an address per se, but is still accessible at that instant. We see that the standard provides mechanisms for access control, but to our knowledge there are no other security mechanisms included in the MMS standard. We quote from the Security Considerations section in the ISO standard [1]: When implementing MMS in secure or safety critical applications, features of the OSI security architecture may need to be implemented. This International Standard provides simple facilities for authentication (passwords) and access control. Systems requiring a higher degree of security will have to consider features beyond the scope of this International Standard. This International Standard does not provide facilities for non-repudiation. As stated above, MMS itself is not designed with information security in mind. This indicates that security should be enforced at some lower layer, but as we have seen through our analysis of the MMS, there is no security enforced at any layer. The ACSE layer could have offered some security features, but as the ACSE layer is omitted from our implementation those features are forfeit. According to the ISO standard the MMS protocol should have implemented some simple facilities for password authentication and access control. We wanted to study these mechanisms to see what security they really offer, but as they are at best optional and at worst not implemented they provide no security what so ever. We have through our analysis seen that they do not exsist. When analysing the MMS we have found no protection against replay attacks. This is a major concern, as anyone with access to the network may sniff up a packet and then replay it on the network at a an inappropriate moment. We do not regard the invokeID field as such a mechanism as it is easily changed.

7

Conclusion

The Manufacturing Messaging Protocol (MMS) is a complex protocol that is rendered even more complex by the implementation of an OSI transportation protocol on top of TCP. MMS offers very limited security mechanisms, and equipment we have studied does not appear to have implemented even these mechanisms. It is clear that if MMS is to be used in process control networks that have to fulfil information security requirements, major modifications have to be made to the protocol.

Acknowledgements The research for this paper was conducted while Mr. Sørensen was a student at the Norwegian University of Science and Technology (NTNU).


615

References 1. Industrial automation systems, Manufacturing Message Specification. Part 1, ISO ISO Standard ISO 9506-1:2003(E) (2003) 2. Overview and introduction to the Manufacturing Message Specification (MMS), System Integration Specialists Company (SISCO), 6605 19,5 Mile Road, Sterling Heights, MI 48314-1408, USA, Tech. Rep. (1995), http://www.sisconet.com/downloads/mmsovrlg.pdf 3. Falk, H., Robbins, J.: An Explanation of the Architecture of the MMS standard. System Integration Specialists Company (SISCO), 6605 19, 5 Mile Road, Sterling Heights, MI 48314-1408, USA, Tech. Rep. (1995) 4. Falk, H., Burns, D.M.: MMS and ASN.1 Encoding. System Integration Specialists Company (SISCO), 6605 19, 5 Mile Road, Sterling Heights, MI 48314-1408, USA, Tech. Rep. (2001) 5. Floyd, L., Ronald, D.: Manufacturing automation protocol. In: Conference Record - International Conference on Communications, pp. 620–624 (1985) 6. Basic encoding rules, Vijay Mukhi’s Computer Institute, India (February 2007), http://www.vijaymukhi.com/vmis/ber.htm 7. Information technology – Open Systems Interconnection – Connection-oriented presentation protocol: Protocol specification. ISO ISO Standard ISO/IEC 88231:1994 (1994) 8. Rose, M.T., Cass, D.E.: RFC 1006: ISO transport services on top of the TCP: Version 3 (May 1987), obsoletes RFC0983. Updated by RFC 2126. Status: STANDARD, ftp://ftp.internic.net/rfc/rfc1006.txt 9. McKenzie, A.M.: RFC 905: ISO transport protocol specification ISO DP 8073 (April 1984), ftp://ftp.internic.net/rfc/rfc905.txt 10. Wireshark wiki on ACSE, Published on Wireshark’s wiki page (June 2006), http://wiki.wireshark.org/ACSE 11. SISCO’s MMS syntax, Published on Systems Integration Specialists Company (SISCO), Inc. Homepage (February 2007), http://www.sisconet.com/downloads/mms abstract syntax.txt

A Long-Distance Time Domain Sound Localization Jhing-Fa Wang, Jia-chang Wang, Bo-Wei Chen, and Zheng-Wei Sun Department of Electrical Engineering, National Cheng-Kung University No.1, Dasyue Rd., East District, Tainan City 701, Taiwan, R.O.C. [email protected]

Abstract. This paper presents a computer simulation of a long distance time domain sound source localization algorithm that localizes a single sound source by using two separated channel microphones. This paper uses the TimeDifference-Of-Arrival (TDOA) method for estimating the time delay difference between the incoming sound signals to microphones using the AverageMagnitude-Difference-Function (AMDF) algorithm. This algorithm has the main advantage of requiring no multiplication operations. The key design objectives are to reduce hardware circuit complexity and implement the algorithm on FPGA board. The computer simulation results have average error within 8° of accuracy in 1 to 5 meters.

1 Introduction Numerous localization system of sound source using microphone arrays has been explored in the past [1][2][3]. The precise and robust localization system is often necessary for a variety of applications, such as speech recognition system, robot [4] and toy products. In recent years, portable consuming electronic products toward to short, small, light and thin, multiple microphone are not suitable to use due to the limitation area. Now, the most widely to use localization of sound source algorithm is Generalized Cross-Correlation (GCC) [5]. This method is to get the TDOA through look for the correlation peak values. The Generalized Cross-Correlation algorithm has been already implemented on FPGA board [6] and chip [7]. In this paper, we expect using another algorithm called Average-MagnitudeDifference-Function (AMDF). This method keeps high accuracy comparing with other method [8][9]. It only uses subtraction operations to get TDOA and expected to reduce design complexity. This paper is organized as follows. Section 3 introduces the principle of AverageMagnitude-Difference-Function. Section 4 demonstrates the sound source localization algorithm simulation flow and expected hardware architecture. Section 5 shows the experimental results of all kinds of angle and distance.

2 Motivation In recent years, market trends are toward the entertainments and Human-Computer Interaction (HCI). We can see many novel toys product in the market such as F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 616–625, 2008. © Springer-Verlag Berlin Heidelberg 2008

A Long-Distance Time Domain Sound Localization

617

Ugobe-Pleo [10] and Hasbro-furby [11], etc. For this reason, we want to make localization system applicable to the general toys. This localization system is able to keep the localization error within 10° of accuracy under a long distance position. This performance is acceptable for toys applications.

3 TDOA Estimation Method This paper applies the principle of hyperbolic characteristic to sound source localization. Hyperbolic location involves estimation of the TDOA between microphone receivers through the time-difference estimation algorithm. The estimated TDOA results are then converted to angle-difference measurement between microphones and hence locate the sound source position. 3.1 Hyperbolic Sound Localization Principle Fig.1 illustrates the hyperbolic sound location. To explain the hyperbolic property, we assume the following assumptions first. Focus-to-focus distance is 2c, namely MIC1 to MIC2 distance. The difference between any points on the curve and focus is 2a, namely the distance difference between sound and both microphone. Therefore, we can locate the sound source θ as

x2 y2 − =1 a2 c2 − a2

(1)

c2 − a 2 x a

(2)

y=±

sound Y B MIC1

O a

c ? a

A

MIC2 X

Fig. 1. Hyperbolic sound location curve

2c = MIC1 − MIC 2

(3)

618

J.-F. Wang et al.

2a = TDOA * v ⎛a⎞ ⎝c⎠

⎛

TDOA * v ⎞⎟ ⎟ ⎝ MIC1 − MIC 2 ⎠

θ = arccos⎜ ⎟ = arccos⎜⎜

(4) (5)

where v is the velocity of sound 3.2 Incoming Signal Model for Time Delay Estimation

Assume a sound signal s(t) is transmitted from a remote source with noise n(t), two separated microphones can be modeled as x1 (t ) = s (t − τ 1 ) + n1 (t ) x2 (t ) = s (t − τ 2 ) + n2 (t )

(6)

where τ 1 and τ 2 are the signal delay times. This model assumes s(t), n1(t) and n2(t) are real, jointly stationary random-process and that s(t) is independent with noise n1(t) and n2(t). Assume τ 1 < τ 2 , we can rewrite Equation (6) as x1 (t ) = s (t ) + n1 (t ) x2 (t ) = s (t − τ ) + n2 (t )

(7)

where τ = τ 1 - τ 2 is the time delay. We need to estimate τ, the TDOA of sound signal s(t) between the two microphones. 3.3 Average-Magnitude-Difference-Function Algorithm

Various time delay estimation is proposed and compared. For accurate estimation of τ, we use Average-Magnitude-Difference-Function (AMDF) algorithm. From Equation (8), we can find that no multiplication operation is required. It is able to reduce the design complexity and implement hardware practical.

AMDF (τ ) =

1 ∑ x1 (t ) − x2 (t + τ ) N

(8)

4 Simulation Flow and Hardware Architecture The flow chart of Average-Magnitude-Difference-Function algorithm is shown in Fig.2. This algorithm is simulated and verified through MATLAB. The sound signal is received from a microphone pair and then amplified through the microphone preamplifier. Fig.3 shows the microphone pre-amplifier and then passes output signal to PC Line-in port after. The parameter setting for sound source localization simulation is shown in Table 1. Fig.4 shows the sound source localization hardware scheme. The


619

input signals are sampled and then stored in two input buffers (one of each channel). With the voice activity detect (VAD), we are able to take the voice components and stored them into the buffer. Finally, the TDOA values will output from the buffer contents of each channel are processed by adder and substrator.

MIC1 MIC2

Frame sound signal s1(t),s2(t) AMDF algorithm Microphone preamplifer TODA to Angle converter ADC conversion

Fig. 2. Average-Magnitude-Difference-Function algorithm design flow

Fig. 3. Two channel microphone pre-amplifier

620

J.-F. Wang et al.

VAD

MIC1

Ch1. input buffer

MIC2

Ch2. input buffer

M U X

buffer1

M U X

adder

substrator M U X

buffer2

M U X

TDOA

adder

VAD

Fig. 4. Hardware architecture of sound source localization Table 1. Parameter setting for sound source localization simulation

Input bit Sampling frequency Inter-microphone separation Microphone type

16bit 64KHz 14cm Omni-directional

5 Experimental Results and Discussion Fig.5 gives the experimental environment diagram. Fig.6 shows the real experimental environment with 1 meter distance. We tested all kinds of angle and distance between the speaker and receiver from degree -75° to 75° with every 15° increase and 1 meter to 5 meters, respectively. Every angle is tested twenty-five times at different distance. Totally have 1375 samples test. Table 2 shows the experimental results of all kinds of angle accuracy on distance 1 meter to 5 meters that experimental error lower than 10°. From Table 2, we can find out that angles accuracy almost between 88% and 100% at angles 0° to 60° regardless of how length is. When the degree reaches 75° and distance between 4 meters and 5 meters, the accuracy are poor. This is caused by echo effect. The echo effect is also called reverberant effect. Miscellaneous of shape environment room will cause a variety of reverberant effects [12]. Here, we are trying to reduce this effect now. Fig.7 to Fig.11 shows the average error measurement results from distance 1 meter to 5 meters. Experimental result shows the average errors within 6.5° of accuracy for distance 1 meter to 3 meters. For distance 4 meters and 5 meters, the average errors are within 8° of accuracy due to echo effect. Compared with other paper [13] in our experimental test of distance, the accuracy of this paper has long distance measurements and the values are almost greater than 88%. Table 3 shows the compared results on distance 1 meter.


621

Table 2. All kind of angles accuracy of sound source localization experimental results Degree 0°

15°

30°

45°

60°

75°

88% 100% 100% 100% 100% 100% 100% 100% 100% 100%

100% 100% 100% 96% 100% 100% 100% 96% 100% 96%

96% 100% 100% 96% 100% 100% 100% 100% 96% 100%

96% 96% 100% 84% 92% 100% 88% 88% 68% 92%

88% 100% 96% 68% 88% 84% 68% 72% 60% 56%

Length 1 meter 2 meter 3 meter 4 meter 5 meter

Left Right Left Right Left Right Left Right Left Right

100% 92% 100% 100% 100%

12.5m French windows

2.6m width wall 8m

mic1

14cm (Mic height 1.35m) mic2 wall 1m 75o 2m 60o 3m o 45 4m o o 30 5m o 15 0

wood wall Fig. 5. Experimental environment diagram

Door

622

J.-F. Wang et al.

Fig. 6. Real experimental environment

Fig. 7. Measurement results for distance of 1 meter


Fig. 8. Measurement results for distance of 2 meters


623

624

J.-F. Wang et al.




625

Table 3. Compared with the accuracy of error within 10° in distance 1 meter

Error corpse.iris > corpse.DN A 4.4

Define Extended Key

If there is no dominant key for the instances, but there is a combination of concepts or properties applicable as extended key which can calculate the similarity of instances, the user can define the rules of combination to be applied. For example, if there are no records about fingerprints, DNA and iris in the missing people ontology, the combination of person’s age, height, hair, dressing can be a good reference to identify the corpse, of course the death time should nearby or after the date a person disappeared. For example, the corpse age is 30, height is 180cm, hair is brown, dressing is T-shirt, estimate death time is 2007-11-01. One missing people in the ontology information: birthday is 1975-10-01, height is 178cm, hair is gold, dressing is T-shirt, missing date is 2007-10-28. Assuming the user defines that every 2 years age difference leads to 10 percent dissimilarity, the age similarity between corpse and missing people would be: V

−V

1 − (corpse.i1 ).(body character).age (missing 2people).((born year)(birthday))−(current year) ∗ 0.1 For all concepts or properties in the extended key, the user can define special rules for the correspondence similarity. Based on the user’s rules, assume that we get the following similarity results: 4.5

Polygon-Based Aggregation Extended Key

In this section, we extend our polygon-based aggregation approach introduced in [9]. We aggregate extended keys by using polygons. To compare similarity

User-Based Constraint Strategy in Ontology Matching

693

Table 1. Extended Key Similarity Between Corpse and Missing People Corpse Missing People age (37) birthday (1975-10-01) height (180cm) height (178cm) estimate death time (2007-11-01) missing date (2007-10-28) cloth (T-shirt) dressing (T-shirt) hair (brown) hair color (gold)

Similarity 0.9 0.9 0.8 1 0.7

SubArea (Missing people) = 1.7404 BigArea (Corpse) = 2.3776 1 0.8 0.6 0.4 0.2 0 −0.2 −0.4 −0.6 −0.8 −1

−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

Fig. 4. Comparing Two Instances in Polygons

between instances, we compare the area of polygons corresponding to each instance, i.e. if polygons have exactly the same area, the similarity between the two instances represented by the polygons is maximal. We are using the following rules to map extended key values to polygons: – Choose the standard ontology. All the values of the standard instances are marked as 1 unit when creating the polygon. – Circles radiuses are used to present the similarities instances values. The circle is divided the number of similaritys elements. The points are added clockwise. The polygon is constructed by connecting the points created. When similarity is 0, this node is not taken into account when connecting the points. For example, the corpse instance is chosen as the standard instances, there are five elements and every 72◦ has one radius. Every radius is 1 unit. The polygon of corpse instance is constructed as Figure 4 (the big one). Using the same way, the instance of missing people is drawn as Figure 4 (the small polygon with the filled color) which radius value is based on the similarity results (see table 1). The similarity of two instances depends on their polygons areas result then would be: 1.7404 = 0.7320 (1) 2.3776

694

5

F. Lin and K. Sandkuhl

User-Based Constraint in Multi Ontology Matching

After processing automatic schema-level and user-based constraint instance matching, the source ontology instances are transformed into the target ontology instances according to the used relations. This leads to an enrichment of the knowledge captured in the source ontology. The enriched knowledge of the ontology is useful in multi ontology matching, as comparisons will be possible that could not be performed without the enrichment. There are two ways to represent the enriched onotology. One way is mergering source and target ontologies into a new ontology and combining the instances based on the new ontology structure. The other way is keeping the ontologies independent, but to include the new knowledge based on rules. We focus on the second way in this paper. For example, to answer the question about “who is the suspect? ” in the corpse example, the ontologies for corpse, missing people and suspect need to be considered together. If the corpse can be identified or a correspondence or similarity in the missing people instances is discovered, the corpse instance knowledge is expanded to include missing people instance knowledge, e.g., name, social relationship, professional, etc. To include correspondences of missing people instances into corpse instances, the user can define the rules as: (corpse(ii ) =missing people(i2 )) −→ (Vcorpse(ii ).name = Vmissing

people(i2 ).name ),

(Vcorpse(ii ).prof essional = Vmissing people(i2 ).prof essional ), (Vcorpse(ii ).social relationship=Vmissing people(i2 ).social relationship ), . . . (2) Based on the matching of corpse and suspect ontologies, the criminal character information is compared, e.g., fingerprint, footprint, blood, availability of crime time. However, based on the corpse and missing people ontology matching results, more information can be examined. For example, the social relationship between the corpse and suspect can be helpful to identify the motive of the crime, this can be checked by either looking for the suspect name in the missing people’s social relationship or if the missing people’s name can be found in society relationship. The rules can be formulated as: corpse(ii ) =missing people(i2 )∧ (Vmissing people(i2 ).name ⊆ Vsuspect(i3 ).social −→ (suspect(i3 ) corpse(ii ))

6

relationship )

(3)

Case Study

The ontologies corpse, missing people and suspect are our test examples. Let us assume there is an instance in the ontology corpse, which shall be identified in the missing people ontology that has 1000 instances. The user defined excluding

User-Based Constraint Strategy in Ontology Matching

695

rules are applied first, because comparing the dominate key (like fingerprint, iris and DNA) is more complicated than applying excluding rules, such as excluding all male instances. If 550 persons are male and the others are female, and if the corpse is female, the computation rate is decreased from 1000 instances to 450. Based on user defined priority for the dominant key, other attributes like age, hair, etc. are not compared in this process. If the dominant key values are missing and the corpse can’t be identified this way, the extended key will be applied, i.e. the dominant key will be skipped. User defined semantic constraints can provide more information and improve accuracy of results. Without user defined rules, all the attributes inside the instances would have to be compared, which obviously is less efficient. Furthermore, if results of such a comparison are integrated, this could lead to incorrect results. User defined constraints including rules are helpful in multi ontology matching. These rules make comparisons possible, which are not feasible without such constraints or rules. For example, as shown in section 5, does the suspect have a social relationship with the corpse? This question can’t be answered through comparing corpse and suspect directly. If the corpse can be found in the missing people and if the user defines instance including rules, this question could be answered. The above discussion about the case study shows that our approach can contribute to performance improvement, decreasing computation rate. Furthermore, the approach will lead to imoproved efficiency and exactness. However, the approach requires that the user or domain expert is involved in the ontology matching process. But in many application cases, the user or domain expert anyhow plays a very important role (as shown in the section 2) and should be involved.

7

Conclusions and Future Work

In this paper, we discussed and motivated the need for user-based constraints in ontology matching. User or domain expert need to provide semantic constraints during the ontology matching process. The proposed approach for user-based constraints includes excluding rules, dominant key and extended key definition and extended key aggregation. We illustrated how to integrate user-based constraint in the ontology matching process. We also proposed to use a polygonbased approach to aggregate the extend key’s values when determining similarity. Furthermore, the application of user-based constraints in multi ontology matching was discussed. One of the problems of the polygon-based approach is, as discussed in [9], how to add weights to the polygons reflecting the importance of extended keys. For example, in the case of corpse identification using an extended key (see section 4.5), the user currently cannot set special weights for the key values, such as ”the age value has higher importance as compared to the other keys”. This will be solved in future work. The main limitation of our work currently is that is has not been implemented and tested in real-world settings. Future work will include to integrate user-based constraints into an ontology matching system.

696

F. Lin and K. Sandkuhl

Based on this integration, we will have the possibility to perform experiments comparing performance and results of ontology matching with and without userbased constraints.

Acknowledgements Part of this work was financed by the Hamrin Foundation (Hamrin Stiftelsen), project Media Information Logistics.

References 1. Euzenat, J., Castro, R.G., Ehrig, M.: D2.2.2: Specification of a benchmarking methodology for alignment techniques. Technical report, NoE Knowledge Web project delivable (2004) 2. Klein, M.: Combining and relating ontologies: an analysis of problems and solutions (2001) 3. Rahm, E., Bernstein, P.: A survey of approaches to automatic schema matching. The VLDB Journal 10(4), 334–350 (2001) 4. Shvaiko, P., Euzenat, J.: A survey of schema-based matching approaches. Data Semantics (2005) 5. Dzemydiene, D., Kazemikaitiene, E.: Ontology-based decision support system for crime investigation processes. Information Systems Development, 427–438 (2005) 6. Euzenat, J., Shvaiko, P.: Ontology matching. Springer, Heidelberg (2007) 7. Lim, E.-P., Srivastava, J.P.S.R.J.: Entity identification in database integration. In: Proceedings. Ninth International Conference on Data Engineering, April 19-23, 1993, pp. 294–301 (1993) 8. Ehrig, M., Sure, Y.: Ontology mapping – an integrated approach. In: Bussler, C.J., Davies, J., Fensel, D., Studer, R. (eds.) ESWS 2004. LNCS, vol. 3053, pp. 76–91. Springer, Heidelberg (2004) 9. Lin, F., Sandkuhl, K.: Polygon-based similarity aggregation for ontology matching. In: Frontiers of High Performance Computing and Networking ISPA 2007 Workshops. LNCS, pp. 255–264. Springer, Berlin (2007)

RFID-Based Interactive Learning in Science Museums Yo-Ping Huang1, Yueh-Tsun Chang2, and Frode Eika Sandnes3 1

Dept. of Electrical Engineering, National Taipei University of Technology Taipei, Taiwan 10608 [email protected] 2 Dept. of Computer Science and Information Engineering, Tatung University Taipei, Taiwan 10451 [email protected] 3 Faculty of Engineering, Oslo University College Oslo, Norway [email protected]

Abstract. Science and engineering education play a very important role in the industrial development of a country. Although most people are aware of the importance of science, students often find science boring and uninteresting as traditional textbooks only provide static information. Students who achieve better usually have practical experiences related to the subject or have been exposed to interactive learning tools. In order to promote science education development, a learning system based on RFID is proposed to provide personalized learning service. This paper introduces the architecture of the proposed service and the technologies used including RFID and collaborative filtering. The contributions of the proposed system and how the profiles are used to provide personalization are also discussed. Evaluations of the system reveal that the system inspires and nurtures their interest in science. Keywords: Digital learning, RFID, collaborative filtering, learning assistant service.

1 Introduction Museums and galleries are often provided to raise the awareness in the population for a particular subject, provide inspiration and provide general education. Many such facilities do not sufficiently manage to captivate the audience and visitors consequently do not return. One problem is that there may be a lack of trained guides present. Guides are sometimes only offered at certain times and only for groups of visitors. A good guide can made all the difference between a mediocre and a memorable museum experience as a talented guide is able to captivate the audience with their stories. To combat the lack of human guide visitors can often borrow devices including headsets, handhold controllers and audio playback devices. Pre-recorded records are then used to present exhibition related introductions to the visitors. However, static audio is not interactive and may be too uninteresting for very young visitors with a short concentration span. Another problem is that some visiting routes may be blocked by visitors grouped around other popular exhibitions. F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 697–709, 2008. © Springer-Verlag Berlin Heidelberg 2008

698

Y.-P. Huang, Y.-T. Chang, and F.E. Sandnes

In order to make the museum guide experience more dynamic multimedia content and tests (learning objects) designed by educators who are expert in the specific domains is combined with an interactive learning assistant. The learning assistant is based on a visitor’s personal profile [14-17], RFID technology where each visitor carries a card with an embedded RFID tag with a unique identification number, collaborative filtering and a history of learning records are combined to provide a personalized learning service. The service is called The Hands Free Interaction Guide System. Users obtain learning materials and questions using their RFID-equipped membership card to touch the sensor installed at the viewing kiosks [1] to obtain the personalized learning service. The system provides an optimal route recommendation derived from user’s browsing history [2-4]. This system has been in active use at the National Science Education Center (NTSEC) in Taipei, Taiwan since October, 2007. This paper is structured as follows. Section 2 introduces the motivation, the system architecture and selected components of interest from the system. The strategy for generating personalized interactive contents by collaborate filtering is illustrated in the third section together with a brief introduction to RFID technology. Next section 4 presents an evaluation of the system based on visitors of different age groups. Section 5 concludes the paper.

2 Motivation and Technologies The goal of the system is to make science education more attractive. Visitors are not always able to be given a guided tour when they come to the education center. However, the learning outcome is related to the skills of the guides that explain the exhibitions. In order to provide a self-learning environment, interactive kiosk were designed to provide relevant content according to age when the visitors are browsing unsupervised. The objective was to replace the professional guide. Guides use various pedagogical strategies to make the contents interesting and attractive. Different descriptions are used depending on the age of students and the students are asked suitable questions to stimulate the learning process. In order to provide personalized learning, RFID technology is used to associate each user with a unique identification. 2.1 Radio Frequency Identification Radio Frequency Identification (RFID) technology comprises a reader and a tag. The reader receives the identity of an object from the embedded tag wirelessly using radio waves. RFID has been successfully used in logistics, healthcare, small value payment and personal ID. Wal-Mart is one well-known case. Researchers have also experimented with embedding a small capsule containing RFID tag in their arms to assess its effectiveness and practicability as a form of personal ID for physical access control and healthcare. In our system, the embedded RFID membership card is used to uniquely identify visitors. A trial card is also provided to occasional visitors, but this only provides limited service. The trial card cannot be used to gather personalized information.

RFID-Based Interactive Learning in Science Museums

699

When the visitors touch the kiosk sensor with their cards collaborative filtering is used to make recommendations [5]. Collaborative filtering is different to rule-based mining and classification [11, 12]. These are used for providing better learning recommendations. The science education center has a range of interactive displays, such as simple electric circuits, vacuum discharge, and solar batteries. Some of the displays are permanent and others are only on display temporary for a particular theme or exhibition. 2.2 Collaborative Filtering Collaborative filtering [6, 7] utilizes the profile of the visitors as the building blocks for making recommendations. Such information is gained by the browsing and learning activities of visitors who have similar or related interests and learning behaviors. Item-based filtering is used as the recommendation mining method. The purpose of the adopted methodology is to connect different science subjects based on the visitor selections and educators’ pre-defined rules based on their teaching experiences. The selection records of visitors can be used to establish associations between subjects. For example, the interactivity kiosk collects visitors’ responses to a learning test, i.e., wrong or right answer [8, 10, 13, 18]. After connecting each subject covered by the visitors, their learning habits are estimated. Thus, the methodology can retrieve a relationship to other subjects when the visitor is learning a specific subject. The following equation is used for calculating the similarity of learning efficiency between subjects.

∑

N

sim(a, b) = corrab =

j =1

( paj − pa )( pbj − pb )

∑j=1( paj − pa )2 N

∑j=1( pbj − pb )2 N

.

(1)

Then, the most suitable subject for recommendation is determined by:

∑ ( p − p ) × sim(c, i) Score(c, j ) = , ∑ sim(c, i) ij

i∈H

i

(2)

i∈H

where paj represents subject a’s rating by visitor j and pa is the average rating from all visitors. As illustrated in Table 1, subjects viewed by the visitor are assigned 1 while the subjects are assigned 0 if they were not viewed. In this example, there is a high similarity of learning between subjects B and E. Therefore, any visitor who had viewed subject B but has not viewed subject E should be recommended subject E. After the recommendations are calculated, the system suggests that visitor E views subject E after viewing subject B. The system can also employ pre-fixed rules based on the recommendation of educators and exhibition managers. This may be useful for suggesting browsing routes for new exhibitions that lack a browsing history. The browsing records of visitors and their positions have to be continually monitored for the suggestions to be valuable [9].

700

Y.-P. Huang, Y.-T. Chang, and F.E. Sandnes Table 1. Example of collaborative filtering Visitor A Subject A Subject B Subject C Subject D Subject E

0 1 0 1 1

Visitor B Visitor C 1 1 0 0 1

0 0 1 0 0

Visitor D

Visitor E

0 1 0 0 1

1 1 0 0 0

3 System Components and Architecture To ensure adaptability, expansibility and stability, the system has been divided into two components, namely the interactivity kiosks and the central server. Our implementation covers 20 different science subjects distributed across three floors. Each interactivity kiosk services a single subject. These 20 kiosks are connected to a single central server that collects all the browsing records and calculates personal learning profiles for the visitors. The personal learning history can later be accessed from the website using a membership account. Physical constraint forced the system to use wireless network connections for most interactivity kiosks. Consequently, to ensure stability and high quality of service a three-layer control design was adopted to prevent the learning service from crashing. As illustrated in Fig. 1, the network reconnection checking module is located at the first layer to ensure all connections are running correctly. The module attempts to reconnect the central server cyclically whenever a network connection is broken. The reason to use a wireless network is due to the physical constraints of installing a wired network in the exhibition area. The limitation also causes some connection problems in the system design. A reconnection module is deployed in the network checking layer to resolve wireless network interruptions, prevent resource exhaustion and prevent wireless devices from overheating. Network failures usually occur unpredictably. Still, the service kiosk must maintain users’ learning records. The reconnection module therefore notifies the learning service layer which uploads the learning records once the connection is available. This procedure guarantees that visitors can retrieve their records despite network failures. The network checking layer ensures a stable and robust connection. In the second layer the learning service provides teaching materials and tests according to the visitor’s profile. Every kiosk is equipped with an embedded RFID reader near its screen. The reader senses the RFID tag which is embedded in the visitor’s member card for verifying the visitor’s profile that is received wirelessly from the central server. The central server recommendation engine determines the suitable learning materials. For each subject there are three levels of learning materials. Each level comprises unique multimedia contents and tests. Visitors view the contents via an interactive interface using a trackball or a touch panel. In total the system contains 60 independent digital learning objects (20 kiosks have three levels each). The system automatically selects a suitable level according to the visitor’s profile, but the visitor can also choose a


701

different level manually. Fig. 2 shows that the visitors name is shown after the visitor login the system and the user may select the level manually. After the visitor has watched the media, the system asks the visitor three questions to assess the learning behavior. These personal records are stored on the central server and used for subsequent recommendations. Although steps are taken to ensure a stable network link, the system may run out of CPU resources. To overcome this problem a resource controller is implemented in the third layer. This controller cyclically monitors the system resource and restarts the learning service when the resource utilization exceeds a given threshold. Through experimentation 80% of CPU usage sustained for 20 seconds was found to be a suitable threshold. This three-layer design is shown in Fig. 1. Each layer is subdivided into several sub-objects. The first layer comprises the reconnection and network status monitor. The RFID, multimedia and recommendation modules are implemented on the interactive kiosks and the server. The third layer contains the system reloader that restarts the service and resource surveillance that invokes the reloader.

　Resource Controller Layer

　Learning Service Layer

System Reloader Resource Surveillance RFID

Multimedia

Recommendation

　Network Checking Layer

Reconnection Network Status

Fig. 1. System architecture

The main service is located in the learning service layer. In this layer, the personalized learning procedure is designed by educators and generated automatically by the server. The learning system architecture is shown in Fig. 2. It is divided into three main parts, namely login, learning and response. The components enclosed by rectangles and ellipses are located on kiosk and server, respectively. Initially, visitors log into the

702


system using their member card. When the visitor is logged in, a personalized welcome message is sent from the server. It is generated according to the unique id containing the visitor’s name and customized message, for example, birthday, Christmas greeting, etc. The server also identifies which materials and tests that are most suitable for the visitor. The learning record is sent to the server and stored. The interactive kiosk will then switch to the welcome screen to await the next visitor. The process is illustrated in the Fig. 2. Visitor

Login Part

Personalized Welcome

Topic Displayer

Learning Part

Suitable Learning Materials

Suitable Tests

Response Part

Personalized Response and Recommendations

Learning Record

Fig. 2. The system flowchart

The hardware configuration of the system is shown in the Fig. 3. The interactive kiosk includes an RFID reader, display and wireless device. Each kiosk is pre-stored all the learning materials to avoid any unpredictable crash. Each kiosk stores sixty videos and up to 600 questions. To prevent malicious attacks and damage, each service kiosk is made of resistible materials and the display panel is protected by a transparent plastic plate. The server stores the visitor’s learning history into the learning material database and responds with the recommendations based on the visitors’ learning histories. A single server is capable of serving more than 100 interactive kiosks based on the available network bandwidth and the processing power of the machine. In this specific case, the implemented server only serves 20 kiosks.


703

Centre Server Learning Material Database

ΞΞ Interactive Station Visitor

Interactive Station Visitor

Fig. 3. The hardware setup

In collaboration with a team of educators a series of multimedia learning objects were designed. An example of the personalized welcome message is shown in Fig. 4(a) and Fig. 4(b), respectively. The system also connects to the server for retrieving the profile of the current visitor to provide more services as shown in Fig. 4(b). The kiosk has set up a variety of celebration events including national birthday celebration, Chinese New Year, father’s day, mother’s day, Christmas, etc. The interface will also remind the membership expiration date. The system selects one level from three available learning materials for the visitor, depending on the visitor’s historical record. Each level contains three different questions that are forwarded to the user after an instructional video is played. The level selection and a teaching example are illustrated in Fig. 4(c) and 4(d). These questions are devised by educators who have a strong track record in science education and supervision. In most cases, the visitors prefer to watch the flash video with rich anime rather than textual descriptions. Thus, most learning materials are expressed by anime to attract visitors’ attention. However, not every subject can be easily transformed to simple anime and sometimes a few textual materials are still used for learning at some specific exhibitions. Fig. 4(e) shows the question interface. After answering the system stores the answer in server database. When the learning session is finished, the kiosk will retrieve the visitor’s email account from the database and ask whether to send the learning records to the visitor. The user can choose whether an e-mail is to be forwarded with information about the visit. A website is provided that allows the visitors to later browse and download visitor-specific items related to their museum visit. These learning histories can later be browsed at home through a web-based interface. Visitors

704


can also print their personal learning record for reference. Generated recommendations presented on the interactive kiosks are shown in Fig. 4(f). If visitors leave their e-mail address, the system can also send the visitor today’s learning history via e-mail. The interface is shown in Fig. 4(g). Fig. 4(h) shows a questionnaire for gathering comments from visitors with the purpose to improve the quality of the learning materials and tests.

Fig. 4(a). Welcome

Fig. 4(b). Celebration

Fig. 4(c). Level selection

Fig. 4(d). Video

Fig. 4(e). Learning test

Fig. 4(f). Recommendation


Fig. 4(g). Learning record

705

Fig. 4(h). Visitor response

4 System Evaluation The system has been deployed for several months starting October 2007. During the period, the system has been used for almost 8000 sessions. To assess the performance of the system, statistics were collected from Oct. to Dec. 2007. These are presented in Fig. 5 and Fig 6, respectively, and present usage statistics for each interactive kiosks and the learning performance associated with different age groups. 4.1 The Usage Frequency for the Interactivity Kiosks Fig. 5(a) and 5(b) show the usage frequencies for the system. Fig 5(a) shows that most members visit the fourth kiosk the most. The reason is that this interactive kiosk is near the entrance. Some kiosks are less used and this may be due to their textual interface that is less attractive. The result shown in Fig. 5(a) implies that most visitors are likely to get a superficial understanding through quick and casual observation rather than to learn a specific subject during their visit to the museum. Another interesting observation is that the tenth to the fourteenth kiosks provide similar statistics that may arise from their closeness of installation and the related subjects to each other. A visitor

Fig. 5(a). Usage statistics for members

706


Fig. 5(b). Usage statistics for non-members

browsing one kiosk may continue to find related subjects of interest at the neighboring kiosks. Therefore, it is suggested subjects should be presented on kiosks in close vicinity with related subjects. Moreover, the relationship between subjects should be taken as an important factor when establishing recommendations and routing rules. Compared to Fig. 5(b) which shows the usage statistics for non-member, most non-members preferred the sixteenth and nineteenth kiosk which cover Vacuum Discharge and Conservation of Angular Momentum. Vacuum discharge shows a vacuum ball that contains electricity light and Conservation of Angular Momentum shows a rotating plate that visitor can stand on to experience the angular momentum. Obviously, those two exhibitions have more interactive and interesting effects for visitors. These statistics may verify that most visitors are not seriously in learning new subjects but are only interested in playing newfangled stuff. Therefore, we should design more special and learnable exhibitions to attract non-member visitors. The statistics reveal that non-members are more interested in dazzling exhibitions than members. One reason for this may be that members want a deeper understanding of a specific subject. The results tell us that more dazzling and interacting exhibitions are needed if we want to get the general public interested in science. The total statistics are shown in Fig. 5(c). The most used kiosk is the Vacuum Discharge and the least popular kiosk is the Dangerous of Our Environment because it is conspicuously located and the materials used are more conventional than the other items on display.

Fig. 5(c). Total usage statistics


707

4.2 The Learning Performance in Different Age Groups Fig. 6(a) and 6(b) present the ratio of correct answers. Fig. 6(a) shows that the rate of correct answers from students of ages from fifteen to twenty is zero. This suggests that the system does not appeal to students in this age segment. However, another explanation is that the educational strategy of NTSEC focuses on elementary and junior high school students; therefore, not many high school students showed activities in the database. Moreover, for the age group of 20 and up, the questions are easy enough for them to answer and correct answer rate is very high. The results for every other age segment display a high correct learning rate of more than 80%. This suggests that the materials are suitable for both children and adults. According to Fig. 6(b) students younger than fifteen still makes some mistakes after having viewed the learning materials. Based on this result the material was reviewed. It

Fig. 6(a). The superior learning performance

Fig. 6(b). The inferior learning performance

708


was found that the vocabulary is too complex and does not match the vocabulary level of young children. This material is currently being revised to such that it is more suitable for younger visitors.

5 Conclusion and Future Work In this paper, we presented a personalized learning assisting system based on RFID technology and recommendation mining method. The proposed system has been tested for several months in a museum. Educators have endorsed the system. Future work includes combining RFID with a camera for personalized capture, remote teaching, in-door location for social contact and routing recommendations. A touch panel will be adapted for a richer interactive experience. By providing the learning interface and agent into visitor’s cell phone for better personal learning is being considered for the future design.

Acknowledgments This work is supported by National Science Council, Taiwan under Grants NSC 95-2745-E-036-001-URD and NSC 95-2745-E-036-002-URD.

References 1. Bellotti, F., Berta, R., Margarone, M.: User testing a hypermedia tour guide. IEEE Pervasive Computing 1(2), 33–41 (2002) 2. Davies, N., Cheverst, K., Mitchell, K., Efrat, A.: Using and determining location in a context-sensitive tour guide. IEEE Computer 34(8), 35–41 (2001) 3. Derntl, M., Hummel, K.A.: Modeling context-aware e-learning scenarios. In: Proc. of the 3rd IEEE International Conference on Pervasive Computing and Communications Workshops, Kauai Island, HI, USA, March 2005, pp. 337–342 (2005) 4. Facer, K., Joiner, R., Stanton, D., Reid, J., Hull, R., Kirk, D.: Savannah: mobile 75 game and learning. Journal of Computer Assisted Learning 20(4), 399–409 (2004) 5. Billsus, D., Pazzani, M.J.: Learning collaborative information filters. In: Proc. of International Conference on Machine Learning, Madison, WI, USA, pp. 46–54 (July 1998) 6. Claypool, M., Gokhale, A., Miranda, T., Murnikov, P., Netes, D., Sartin, M.: Combining Content-Based and Collaborative Filters in an Online Newspaper. In: Proc. of ACM SIGIR Workshop on Recommender Systems, Berkeley, CA (1999) 7. Good, N., Schafer, J., Konstan, J., Borchers, A., Sarwar, B., Herlocker, J., Riedl, J.: Combining collaborative filtering with personal agents for better recommendations. In: Proc. of the Conference of the American Association of Artificial Intelligence, Orlando, FL, USA, July 1999, pp. 439–446 (1999) 8. Haverkamp, D.S., Gauch, S.: Intelligent information agents: review and challenges for distributed information sources. Journal of the American Society of Information Science 49(9), 304–311 (1998) 9. Bahl, P., Padmanabhan, V.N.: RADAR: An in-building RF-based user location and tracking system. In: Proc. of IEEE Computer & Communications Societies, Tel-Aviv, Israel, vol. 2, pp. 775–784 (March 2000)


709

10. Ponnusamy, R., Gopal, T.V.: A user adaptive self-proclamative multi-agent based recommendation system design for e-learning digital libraries. In: IEEE Conference on Cybernetics and Intelligent Systems, pp. 1–7 (June 2006) 11. Wen, Q., He, J.: Personalized recommendation services based on service-oriented architecture. In: IEEE Asia-Pacific Conference on Services Computing, pp. 356–361 (December 2006) 12. Xi-Zheng, Z.: Building personalized recommendation system in e-commerce using association rule-based mining and classification. In: International Conference on Machine Learning and Cybernetics, vol. 7, pp. 4113–4118 (August 2007) 13. Zaiane, O.R.: Building a recommender agent for e-learning systems. In: International Conference on Computers in Education, vol. 1, pp. 55–59 (December 2002) 14. Chen, T., Han, W.-L., Wang, H.-D., Zhou, Y.-X., Xu, B., Zang, B.-Y.: Content recommendation system based on private dynamic user profile. In: International Conference on Machine Learning and Cybernetics, vol. 4, pp. 2112–2118 (August 2007) 15. Puntheeranurak, S., Tsuji, H.: Mining Web logs for a personalized recommender system. In: 3rd International Conference on Information Technology: Research and Education, pp. 445–448 (June 2005) 16. Youji, O., Wakita, R., Yano, Y.: Web based self-directed learning environment using learner’s annotation. In: International Conference on Computers in Education, vol. 2, pp. 1207–1211 (December 2002) 17. Tan, X., Yao, M., Xu, M.: An effective technique for personalization recommendation based on access sequential patterns. In: IEEE Asia-Pacific Conference on Services Computing, pp. 42–46 (December 2006) 18. Furukawa, M., Watanabe, M., Kinoshita, M., Kakazu, Y.: A mathematical model for learning agents on a multi-agent system. In: IEEE International Symposium on Computational Intelligence in Robotics and Automation, vol. 3, pp. 1369–1374 (July 2003)

Real-Time Detection of Passing Objects Using Virtual Gate and Motion Vector Analysis Daw-Tung Lin and Li-Wei Liu Department of Computer Science and Information Engineering National Taipei University 151, University Rd., San-Shia, Taipei, 237 Taiwan

Abstract. Real-time human tracking and pedestrian counting in very complex situations with different directions of motion has been important for video surveillance and our daily life applications. This work presents a virtual gate method for the pedestrian detection without the need to construct a background model a priori. The proposed method utilizes motion estimation with three step search and a novel motion vector analysis algorithm which detects moving objects passing through the gate along any desired direction. This method is particularly applicable to complex situations. The experimental results demonstrate that the proposed strategy is reliable. Keywords: People counting, video surveillance, motion estimation, three step search, motion vector analysis.

1

Introduction

To accurately track and count pedestrians in very complex situations with different directions of motion has been important for video surveillance. Examples include the entrance control of buildings, estimation of customer flow in market places, detection of overcrowding hazards in public areas, and the management of effective departure intervals for mass transportation systems. Vision-based surveillance is a very active research area in the computer vision community. Sacchi et. al. developed a people counting application for tourist site-monitoring [1]. Sidla et. al. presented a vision based pedestrian detection and tracking system which is able to count people in very crowded situations [2]. Snidaro et. al. and Yang et. al. utilized a multi camera approach to conquer the mutual occlusion problems [3,4]. Liu et. al. focused on the problem of segmentation of group of people into individuals with a model based segmentation method and then applied it to a people counting application [5]. Zhao and Nevatia proposed a head detector algorithm utilizing the information of camera calibration and ground plane parameters to estimate the individuals [6]. Several classification

This work was supported in part by the National Science Council, Taiwan, R.O.C. grants NSC96-2221-E-305-008-MY2, NSC 95-2221-E-305 -006 and Ministry of Economics 94EC17A02S1-032, 95EC17A02S1-032.


Real-Time Detection of Passing Objects

711

methods such as radial basis function network and support vector machine have been adopted to solve the pedestrian identification problem based on the extracted features [7,8]. One popular video-based people counter solution is based on top-view camera configuration [9,10,11,12,3]. However, we consider motion object detection and people counting as an added value to security and safety applications for those systems and cameras have already been mounted for facing the entrance of a particular location. In this paper, we propose a vision-based real-time people counting system for complex situations that is not dependent on an existing building background model. Furthermore, we hope that users can specify any moving direction of people passing through a virtual gate. If a person passes through the gate in a specific direction that users appointed in advance, this system will capture image immediately and analyze it for other applications. Figure 1 shows a scenario of the passing objects through a virtual gate which defined by a user illustrated with red lines. Notably, there are many people passing through this virtual gate. Other objects including vehicles and pedestrian may move in any directions in this scene as well. The objective of the proposed system is to capture the image when people passing through the virtual gate and fit the direction specified by the users. Virtual Gate

Fig. 1. Scenario of the virtual gate (illustrated with red line) and specified direction (denoted with green arrow)

The rest of this paper is organized as follows. Section 2 illustrates the overall architecture and user interface of the proposed system. Section 3 elucidates the technique of motion detection. The principle and algorithm of motion estimation and motion vector analysis are then described. Section 4 then describes the experimental setup. Several simulations of the proposed approach are presented, with experimental results demonstrated the effectiveness of the motion object detection method. Conclusions are provided in Section 5, along with recommendations for future research.

712

2

D.-T. Lin and L.-W. Liu

System Architecture

To deal with complex environmental situations and changes, we do not adopt traditional background modeling for foreground subtraction which could lead to large amount of false motion alarms. Thus, one of our main goals is to accurately extract the region of interest (ROI) for moving objects without using a background model. We also developed the graphic user interface (GUI) with which the user can define the virtual gate, detecting area, capture area, motion vector display button and the other detail parameters of the system. Figure 2 presents the major functions which are indicated by arrows. User Defined Direction

Virtual Gate

Detection Result

Capture Area

Fig. 2. Graphic user interface (GUI) of the proposed system

The proposed method has been decomposed into the following two steps: I. Motion Estimation: Due to complicated environment, we apply a motion estimation approach to determine whether any potential moving objects have entered the scene; II. Motion Vector Analysis: After motion estimation, we record the ROI of motion objects until ROI reaches the detecting area. When the detecting area has been reached, the moving direction of tracked objects will be analyzed to decide whether this direction matches the one appointed by user. The overview block diagram of the proposed algorithm is shown in Fig. 3. First, user needs to define the virtual gate, detecting and capture area by utilizing convenient click and drag function in our GUI. For many applications, there are only one or two locations which need to be concerned in a video sequence. For instance, if we deploy a camera a few meters above the ground facing the entrance at a convenience store, all we need to known is how many people get into this convenience store. By utilizing the proposed system, user can define the detecting and capture area to monitor the customers without extra equipments. Motion Vector is computed by motion estimation method using the three step


Previous Frame Set Virtual Gate

Set Moving Direction

Set Capture Area

713

Current Frame

Motion Estimation

No

Capture Image

Match

Objects Score Judgement

Motion Vector Analysis

Yes

Objects in Detecting Area?

Fig. 3. Block diagram of the proposed motion detection algorithm

search (TSS) algorithm [13] which has been widely applied in video compression. Section 3.1 will elucidate this algorithm in detail. To estimate the motion vector of moving objects, we propose a motion vector analysis approach where the final judgment is accomplished. Then the principle and algorithm of motion vector analysis are described in detail in Section 3.2.

3 3.1

The Proposed Motion Detection Algorithm Motion Estimation

Motion estimation plays an important role in video coding. The block matching (BM) algorithm is a well known method of motion estimation. In the proposed algorithm, we first divide the current image frame into fixed-size blocks. Next, the motion vector of each block is estimated by finding the most similar block in the previous frame according to the defined matching criterion. That is, for each block in the current image frame, we need to determine the best matched block within a search window in the previous image frame. Let the size of search window be (2P + 1)×(2P + 1) and the size of each block be N ×N with N = 2m . The matching criterion used in our system is the mean square error (MSE) which is expressed as: N N 1 M SE(k,l) (x, y) = |Ft (k + i, l + j) − Ft−1 (k + x + i, l + y + j)|2 (1) N ×N i=1 j=1

for −P ≤x, y≤P , where Ft (k + i, l + j) denotes the gray value of the pixel at position (i, j) in the current block (k, l), and Ft−1 (k + i + x, l + j + y) denotes the gray value of the pixel at position (x + i, y + j) in the block (k + x, l + y) of the previous frame. Therefore, the best matched block within the search window in the previous image frame is to find the minimum match error, (i.e., minimal M SE(k,l) (x, y)).

714


One of the simplest block matching algorithms is the full search (FS) algorithm which provides an optimal solution by searching all candidates within the search window. But its high computational complexity made it difficult to be implemented in real-time. Thus, we apply the three step search (TSS) algorithm [13] in our system due to its simplicity and robust and near optimal performance as well. Searching with a large pattern in the first step, TSS more efficiently find the globe minimum match error. The algorithm is described as follows. Step 1: Choosing an initial center location. Eight blocks at distance of step size from the center (around the center block) are selected for comparison. Step 2: Moving the center to the point with the minimum match error. The step size is reduced to half.

1

1

1

1

1

1

2

2

2

Motion Vector

1

1

2

2 3

2

3

2 3

3

3

3

3

3

3

2

Fig. 4. Example path for convergence of Three Step Search method

Step 1 and 2 are repeated until the step size becomes smaller than 1. Fig. 4 shows a simple example path for the convergence of this algorithm. Distance of four is applied in the first stage search. The red dots with label 1’s denote the blocks chosen in the first stage in which the best match occurs at the lower-right dot. Thus the new center is moved to this particular location for the second stage search (denoted in blue dotted arrow in Fig. 4). By reducing the distance to two, new neighbors are chosen. The orange dots with label 2’s are the blocks chosen for the second stage search. Assume the best match happens at the lower-middle dot, the new center location will be moved to it. Similarly, the green squares with label 3’s represent the blocks chosen for the third stage comparison. The matching calculation will be performed and stop at the location with minimum value because the distance has been reduced to one at this point. 3.2

Motion Vector Analysis

After the motion estimation, the motion vector M V(k,l) of the block (k, l) is given by: M V(k,l) = arg min M SE(k,l) (x, y) (2) (x,y)


715

where the vectors (k + x, l + y) are valid block coordinates. The proposed system computes and records the motion vector of each block for further analysis. If there is no moving object in the desired detecting area (i.e., blocks in the detecting area have no significant motion variations), then the system will wait until there are sufficient motion changes occur in the detection area. Once there appear sufficient motion changes, the system will analyze the motion vector of each block in the detecting area based on our criterion. In order to determine whether the moving objects in the detecting area are passing through the virtual gate along the desired direction, the moving direction and moving distance of each block are compared and analyzed between the current image frame and the previous image frame. Also, another benefit of the information is that the detecting results can be stabilized and enhanced by setting a vigilant threshold. By only adopting the direction of moving objects information, the detecting results might not be reliable enough. Therefore, we can evaluate the direction M B(k,l) of the block (k, l) according to the value of X and Y axis components which are derived form the motion vector M V(k,l) and is expressed as the following equation.

M B(k,l) =

arctan (β/α)×180 π

(3)

where β denotes the value of Y-axis component of motion vector M V(k,l) , α denotes the value of X-axis component of motion vector M V(k,l) , and the result of arctan is in radians. To continue the next analysis, the difference of the direction between the motion vector M V(k,l) of block (k, l) and the desired direction defined by user M Vd can computed by the following equation: M BD(k,l) = T1 − ( M B(k,l) − M Bd )

(4)

where the motion vector angle M Bd of the desired direction can be computed by Equation (3), and T1 is the threshold of the M BD(k,l) . If a higher value of T1 is chosen, the constraint of detecting will become looser. 64 is chosen for T1 through extensive experiments in this study. After the direction information has been estimated, another information of the distance value M B(k,l) of block (k, l) can be deduced as: M B(k,l) = M V(k,l) =

(α)2 + (β)2

(5)

Although the value of direction and distance of motion vector have been computed, we still need to make integration for the final justification. The benefit of integration is that the system can consider this integration value as a judgement weighting factor, the higher value indicates there is higher probability that the block (k, l) is one of the ROI object consistent with the desired moving direction. Another benefit is that we can track the block for a while, so that the system can ensure that the block is belong to the moving object but not the noise. The integration value M BH(k,l,t) of the block (k, l) is defined as:

716


⎧ M BI(k,l,t−1) + M B(k,l) ×M BD(k,l) if M BD(k,l) ≥T2 ⎪ ⎪ ⎪ ⎨ and M B(k,l) ≥T2 , M BI(k,l,t) = ⎪ ⎪ ⎪ ⎩ M BI(k,l,t−1) /2 otherwise. (6) where T2 is a threshold of the final step parameters. The larger the T2 is, the more the number of blocks of the moving object will be included. We set T2 to be 16 in this study. From Equation (6), if there is a noising block resides in the detecting area, the integration value of the block will become smaller than the value of the same block in the previous image frame. The noising block will not be considered as a part of moving object. To obtain the complete object information without broken fragments, morphological operation and labeling method [14] are applied for the integration value M BH(k,l,t) in our algorithm. In this paper, we use closing operation (dilation followed by erosion) which is one of the popular techniques to eliminate the effect of noise and to get the smoothed integration values. Then, we utilize connected components algorithm to make label for these smoothed values. Because this algorithm is effective in determining the whole moving object, moving region can be characterized not only by their position, but also their size, and other shape information. After morphological operation and labeling are performed, each object is determined whether it is a main object or a part of the object. The part of the object is merged into the closest main object. Thus, those blocks with the same labels are considered a moving object passing through the virtual gate along the direction, then the system will capture the image from a defined region in video sequence.

4

Simulation Results

The proposed pedestrian detection system was implemented to operate at about 30 frames per second (30 fps) on Intel Core2 Duo 2.4GHz either for life analog CCD camera and video sequence files. The experiments were performed to demonstrate the detection performance of the proposed system. To test the performance, we have recorded five video clips containing several people entering and exiting a grocery store. The length of each video clip is around twelve minutes. Figure 5 demonstrates the results of motion estimation, in which Fig’s 5(a), (b) and (c) show three consecutive image frames of the testing video clips. Figure 5(d) shows the result of motion vectors of moving object corresponds to the motion estimation of the difference between the previous frame (Fig.’s 5(a)) and the current frame (Fig.’s 5(b)). The red arrows in Fig 5(d) stand for the motion vector of each block. Similarly, Fig. 5(e) presents the result of motion vectors (plotted in red arrows) of moving object corresponds to the motion estimation of the difference between its previous frame (Fig.’s 5(b)) and the current frame (Fig.’s 5(c)). In this study, the block size and the search window size were 8 and 15, respectively.


(a)

(b)

(d)

717

(c)

(e)

Fig. 5. Experimental results of motion estimation

(a)

(e)

(b)

(f)

(c) No Match

(d)

Detection Result

(h)

(g)

Fig. 6. Experimental results of motion object detection for entering direction

Figures 6(a)(b)(c) and (e)(f)(g) are two cases of consecutive video sequences of object passing through the gate along two contrary directions with exiting and entering, respectively. Assume we desire to monitor the people entering the grocery store utilizing the proposed system. Figure 6(d) shows that there is no match result in the right-hand side window because the moving object in Fig. 6(a)(b)(c) is exiting the virtual gate which doesn’t match the desired direction. Figure 6(h) demonstrates the correct outcome for the incoming case. When the moving object in Fig. 6(e)(f)(g) is passing the gate along the desired

718

D.-T. Lin and L.-W. Liu Table 1. Evaluation for the proposed system with five real world video clips Test Sequence Direction No. of people Count Accuracy(%) 1

2

3

4

5

Entering

27

25

93

Exiting

29

26

90

Entering

18

17

94

Exiting Entering

17 14

15 13

88 93

Exiting

13

11

85

Entering

14

14

100

Exiting Entering

11 32

10 30

91 94

Exiting

29

26

90

Overall

92

direction, then the system will capture the snapshot and shows the matching motion blocks. To test the performance of the proposed algorithm, we evaluate on the above mentioned five video clips. Table 1 presents the statistical results of the experiments. The overall accurate detection performance is 92%.

5

Conclusion

This work presents a virtual gate method for the pedestrian detection without the need for constructing a background model a priori. The proposed method utilizes motion estimation with three step search and a novel motion vector analysis algorithm which detects moving objects passing through the gate along any desired direction. This method is particularly useful for complex situation. The experimental results demonstrate that our system produces reliable performance. Because of the low computational complexity, our system even can be used in other applications for the pre-processing to reduce false alarm of motion object detection. Future work will be to combining more than two features for detection, and to finding another method of effectively combining multiple features for performance enhancement.

References 1. Sacchi, C., Gera, G., Marcenaro, L., Ragazzoni, C.S.: Advanced image-processing tools for counting people in tourist site-monitoring applications. Signal Processing 81, 1017–1040 (2001) 2. Sidla, O., Lypetskyy, Y., Brandle, N., Seer, S.: Pedestrian detection and tracking for counting applications in crowded situations. In: Proc. IEEE Conference on Advanced Video and Signal Based Surveillance, p. 70 (2006)


719

3. Snidaro, L., Micheloni, C., Chiavedale, C.: Video security for ambient intelligence. IEEE Trans. Systems, Man and Cybernetics A 35(1), 133–144 (2005) 4. Yang, D.B., Gonzalez-Banos, H.H., Guibas, L.J.: Counting people in crowds with a real-time network of simple image sensors. In: Proc. IEEE Int. Conf. Computer Vision, vol. 1, pp. 122–129 (2003) 5. Liu, X., Rittscher, J., Perera, A., Krahnstoever, N.: Detecting and counting people in surveillance applications. In: Proc. IEEE Conference on Advanced Video and Signal Based Surveillance, pp. 306–311 (2005) 6. Zhao, T., Nevatia, R.: Tracking multiple humans in complex situations. IEEE Trans. Pattern Analysis and Machine Intelligence 26(9), 1208–1221 (2004) 7. Lin, S.-F., Chen, J.-Y., Chao, H.-X.: Estimation of number of people in crowded scenes using perspective transformation. IEEE Trans. Systems, Man and Cybernetics A 31(6), 645–654 (2001) 8. Huang, D., Chow, T.W.S.: A people-counting system using a hybrid RBF neural network. Neural Processing Letters 18, 97–113 (2003) 9. Albiol, A., Mora, I., Naranjo, V.: Real-time high density people counter using morphological tools. IEEE Trans. Intelligent Transportation Systems 2(4), 204– 218 (2001) 10. Chen, T.-H., Chen, T.-Y., Chen, Z.-X.: An intelligent people-flow counting method for passing through a gate. In: Proc. IEEE Conference on Control, Automation, Robotics and Vision, pp. 1–6 (2006) 11. Kim, J.W., Choi, K.S., Choi, B.D., Ko, S.J.: Real-time vision-based people counting system for security door. In: International Technical Conference on Circuits/Systems Computers and Communications, pp. 1416–1419 (2002) 12. Septian, H., Tao, J., Tan, Y.-P.: People counting by video segmentation and tracking. In: Proc. IEEE Conference on Control, Automation, Robotics and Vision, pp. 1–4 (2006) 13. Koga, T., Linuma, K., Hirano, A., Lijima, Y., Ishiguro, T.: Motion compensated interframe coding for video conferencing. In: Nat. Telecommun. Conf., New Orleans, La, USA, pp. G5.3.1–G5.3.5 (December 1981) 14. Gonzalez, R.C., Woods, R.E.: Digital Image Processing 2/e. Prentice Hall, Englewood Cliffs (2002)

A Ubiquitous Interactive Museum Guide Yo-Ping Huang1, Tsun-Wei Chang2, and Frode Eika Sandnes3 1

Dept. of Electrical Engineering, National Taipei University of Technology Taipei, Taiwan 10608 [email protected] 2 Dept. of Computer Science and Information Engineering, De Lin Institute of Technology Taipei, Taiwan 236 [email protected] 3 Faculty of Engineering, Oslo University College Oslo, Norway [email protected]

Abstract. This paper proposes a PDA-based guide and recommendation system. The intelligent guide provides an alternative to the current audio-based guide tools. Techniques from both data mining and extension theory and RFID technology are deployed to provide better interaction between exhibitions and visitors. Keywords: Guide system, RFID, extension theory, data mining.

1 Introduction Tourism plays a substantial role in the economies of some countries. Tourism can secure employment, foreign exchange earnings, investment and regional development. To attract more tourists and local visitors, many units such as natural parks, museums, art galleries, hotels and restaurants tailor various personalized routes for the individual needs. Recently, various recommender-related systems have been proposed in the literature [1-3]. Most museums or exhibition centers offer assistance to tourists who explore the exhibits in the form of a personal guide or information booths. However, such manual services can be problematic. Lack of staff is the key obstacle. Often, a prior appointment with museum guide is required, especially for the weekends and public holidays. Without the personal guide, the visitors often have to rely on a pre-recorded audio-tape, tape player, and a headset for information. Besides, museums or exhibition centers often set a series of routes through the exhibits [4]. Consequently, popular exhibits may become congested with too many visitors. To tackle these limitations, a guide system running on handheld devices is proposed. The proposed guide system recommends interesting exhibitions to users by the means of data mining. Furthermore, extension theory is used to recommend items to the users that are related to the items the users have already browsed. Moreover, some visitors prefer to browse the exhibitions alone and to explore the diversity of the exhibitions. F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 720–731, 2008. © Springer-Verlag Berlin Heidelberg 2008

A Ubiquitous Interactive Museum Guide

721

This paper is organized as follows. Section 2 presents an overview of the proposed system, detailed design and related techniques. Section 3 presents the recommendation method based on association rules. Experimental results and discussions are presented in section 4. Conclusions and future research are given in section 5.

2 System Implementation The two main components of the proposed guide and recommendation system are the client-side PDA and the server-side database. The PDA client comprises the guide user interface and the server-side database maintains visitors’ log files stored. The system architecture is depicted in Fig. 1. A traditional audio-tape based guide system can only deliver static exhibitions contents. To enhance the quality of interaction, the proposed guide and recommendation system is implemented on a handheld device. The RFID reader, wireless network and EPC class 1 electronic tag allow the PDA-based platform to support a dynamic and interactive learning model. ISO 15693 passive electronic tags are attached to the information plaque of each exhibition. When a user holds the PDA equipped with a RFID reader close to the tag, the detailed contents of the exhibition is downloaded via the wireless network including historical background, the motivation behind the creations, etc. In addition, personal customized guidance is recommended to individuals according to their association rules-based preferences. The following sections describe the components of our system.

Fig. 1. The architecture of the proposed system

2.1 The Client-Side PDA-Based Guide System The PDA shown in Fig. 2 comprises eight modules. The corresponding functions include: (1) RFID Reader Module: This module emits radio waves for reading the passive electronic tag. Both the tag ID and the electronic tag memory can be read.

722

Y.-P. Huang, T.-W. Chang, and F.E. Sandnes

(2) RFID Identification Module: This module sends control command to the RFID reader module. When the RFID reader receives the response messages and identifies the corresponding ID, the relevant contents can be accessed directly from the PDA or downloaded from the mobile communication module. (3) RFID Middleware – Mapping Module: This module identifies the ID of the exhibition obtained from the RFID Identification Module. (4) Wireless/Mobile Communication Module: This module transmits data from the server to the PDA through WLAN, Bluetooth, or 3G. (5) Guide Content Storage Module: This module saves the guide information or content in advance such that a fast fetch for the detailed explanation of contents is possible. (6) Guide Content Display Module: This module presents the contents to the users such as textual introductions and multimedia. (7) Guide Activity Monitor Module: This module records the visitors’ routes and the browsed exhibitions. A suggested route of additional items to browse is recommended by the proposed recommendation model. (8) Guide Interaction Module: This is the user interface of the proposed mobile guide system. All the exhibitions and related information are presented in this module.

Fig. 2. The modules of the PDA guide system

2.2 Server Functionalities of the Guide System There are five modules in the backend as shown in Fig. 3, namely: (1) Network Communication Module: Information and multimedia contents can be received or transmitted between the Network Communication Module and the Mobile Communication Module on the PDA.


723

(2) Guide Content Repository Module: This module stores the exhibition contents, related multimedia data and special exhibition programs. (3) Data Mining Module: This module finds association rules based on visitors’ browsing records. The recommendations are made according to these association rules. (4) Video Streaming Module: The multimedia data are encoded as video streams and then transmitted to the PDA. (5) Guide Activity Log Repository Module: This module maintains and manages the overall system.

Fig. 3. The backend modules of the guide system

2.3 RFID-Enabled Interactive Guide System The Radio Frequency Identification (RFID) technique emerged in the 1980s [5-6]. Since then, RFID has been applied in a diverse set of domains. RFID-systems comprise tags and readers. The tag contains object identification which is read by the reader and then compared to the corresponding identification stored in the database. When a match is found, detailed information is retrieved. Domains such as education, business or science can successfully utilize RFID [7-8]. Due to its wide applicability, convenience, wireless information transmission, robustness to the environment and the possibility of performing multiple reads and writes, RFID is adopted in the proposed system. 2.4 Collaborative Filtering Usually, galleries position fine arts according to different categories such as photographic collections, Chinese ink-water paintings, oil paintings or ancient architectures in different zones. Alternatively, artifacts from different categories are mixed and presented together for some particular theme. The main goal of the proposed system is to customize an optimal route for visitors. While browsing exhibitions in one zone, a visitor may miss some particular artists’ works belonging to different themes but with similar characteristics. An optimal guiding route is therefore recommended based on item-based collaborative filtering (CF) method [10-11].

724


The exhibit center first gathers all the information about what the visitors have viewed. Then the relationships between the browsed items can be identified. More related exhibitions can therefore be recommended to the visitors when they browse particular items. The CF method first identifies visitors with similar interests in exhibitions as the target user. The recommendation is then issued assuming that the groups of visitors have similar interests in exhibitions. The CF method continues identifying the correlations between items within visitors’ groups. Once the correlations have been found, potential exhibitions are recommended to the visitors. The preference similarity of exhibitions between the visitors is calculated as follows.

sim( a, b) =

Score( c, j ) =

∑ Nj=1 ( paj − pa )( pbj − pb ) ∑ Nj=1 ( paj − pa ) 2

∑( p i∈H

ij

(1)

∑ Nj=1 ( pbj − pb )2

− pi ) × sim( c, i ) (2)

∑ sim(c,i ) i∈H

where Paj : user a’s rating of exhibition j, and Pa : user a’s average rating of exhibitions.

Table 1 illustrates the CF method. Exhibitions browsed by a user are marked with 1, otherwise it is marked with 0. According to this table, visitor A has high preference similarity to visitor D. Hence, any exhibition browsed by visitor A not yet browsed by visitor D should be recommended to visitor D. Table 1. Collaborative filtering example

visitor A visitor B visitor C visitor D visitor E

People 1 0 0 1 1

Scenery 1 1 0 1 0

Bird 0 0 1 0 0

Flower 0 1 0 0 1

Mountain 1 1 0 0 0

3 Employing Extension Theory to Refine Recommendation The CF recommendation method recommends items for groups of visitors who have similar taste. Hence, the automatic guide system has the ability to “guess” what exhibitions that should be recommended to different visitors. Further relationships between exhibitions are found by mining association rules from visitor profiles. 3.1 The Construction of FP-Tree

In this section a simple database example, shown in Table 2, is used to introduce the FP-tree. The minimum support value is predefined as 2. Just as in the Apriori


725

algorithm, the database is first scanned for frequent 1-itemsets with their support counts. A sorted list of frequent 1-itemsets L according to the support count in descendant order is as follows. L = [4:4, 2:3, 3:3, 6:3, 1:2]. A corresponding FP-tree is shown in Fig. 4. While building the FP-tree, every transaction has to be the same sequence as the sorted list L. If an item cannot find a node containing itself along any branch, it will be inserted into the tree and initialized with a support count of 1; otherwise, the support count is increased by 1. Table 2. The database used to build the FP-tree TID 100 200 300 400

Transaction (items) 2, 3, 4, 6 2, 4, 5, 6, 7 1, 2, 3, 4 1, 3, 4, 6

Fig. 4. A FP-tree from the database in Table 2

When the FP-tree is built, the mining procedures are preceded with the FP-tree as follows. (1) Construct the corresponding conditional pattern base from the beginning of frequent pattern whose length is 1. (2) Build its conditional FP-tree. (3) Mine the conditional FP-tree recursively. During steps 1 and 2, a table containing the conditional pattern base of items in the list L is generated before mining the FP-tree (see Table 3). The items in the Table 3 are in the inverse order compared to the list L. In other words, when item 1 is checked, all frequent itemsets with initial 1 are found. For example, if we want to mine item 6 with its conditional FP-tree, it is not necessary to consider item 1. That is because the entire frequent itemsets with initial 1 will be generated when the conditional FP-tree of item 1 is mined.

726


Considering item 1, it is evident that there are two branches in the FP-tree, and . These two branches are the conditional pattern base of item 1. Based on this conditional pattern base, it is easy to generate the relative conditional FPtree. Having obtained the conditional pattern base, then we can also find that the is the conditional FP-tree of item 1. Since the support count of is smaller than the minimum support, it thus cannot be the element of the conditional FP-tree of item 1. Figure 5 demonstrates how to build and mine the conditional FP-tree. Table 3. The conditional pattern base and relative conditional FP-tree Item 1 6 3 2

Conditional pattern base {(4 2 3:1), (4 3 6:1)} {(4 2 3:1), (4 2:1), (4 3:1)} {(4 2:2), (4:1)} {(4:3)}

Conditional FP-tree ,

Fig. 5. The conditional FP-trees for item 1

The frequent pattern can be found using the above-mentioned steps. Meanwhile, the candidate itemsets are generated such that the computing cost can be reduced. Nevertheless, FP-tree has a high memory requirement. Instead of loading the entire database into memory, the database can be partitioned into several projected databases. Each projected database is used to build the FP-tree and is mined separately [12]. 3.2 Extension Set Theory

Classical set theory utilizes 0 and 1 to represent the certainty of an event; while extension set theory adopts real number ranging from (-∞, +∞) to indicate the degree of certainty of an event [13-14]. Positive and negative numbers indicate the degree of an event with or without the possibility, respectively. Definition: Suppose U is the universe of discourse. For any element u , u ∈ U , there exists a real number such that K (u ) ∈ ( −∞,+∞) . We call the relative set


727

~ = {(u, y ) | u ∈ U , y = K (u ) ∈ ( −∞,+∞ )} is a extension set of the domain U. And the A ~ , and K (u ) is the degree of u related to A ~. y = K (u ) is the incidence function of A ~ . A = {u | u ∈ U , K (u ) ≤ 0} And A = {u | u ∈ U , K (u ) ≥ 0} is the positive domain of A ~ . J = {u | u ∈ U , K (u ) = 0} is zero boundary of A ~. is the negative domain of A 0 Obviously, if u ∈ J 0 , then u ∈ A and u ∈ A occurs at the same time. An example is used to illustrate the concept of extension sets. Suppose U is a set of processed machine spare parts and the standard diameter is 50±0.1mm. When the diameter of a spare is between 49.9mm and 50.1mm, it will belong to the qualified product. On the other hand, a spare part belongs to a set of unqualified products when it is outside the range from 49.9mm to 50.1mm. Normally, there are only two groups of machine spares under classic set theory, qualified and unqualified. Actually, if a spare’s diameter is larger than 50.1mm, then it is a returnable product. On the contrary, if the diameter is smaller than 49.9mm, then it will be a rejected product. Figure 6 shows the difference between a classical set and an extension set.

(a) Classic set.

(b) Extension set. Fig. 6. Classic versus extension sets

3.3 Data Mining Based on FP-Tree and Extension Sets

Traditional FP-tree mining uses a single threshold to derive the frequent itemsets. It means that the mining results highly depend on the pre-assigned threshold. A new

728


strategy is proposed for mining FP-trees. We are not only interested in extracting frequent patterns based on the pre-assigned threshold (lower bound), but also infrequent patterns. These infrequent patterns may become frequent under some conditions. For example, a product is not a frequent item if it is not popular. It will be frequent while associated with other products. Our goal is to discover the original frequent patterns and extension frequent patterns for recommending more valuable exhibitions to visitors. To mine the extension frequent patterns, an extension threshold has to be set in advance. Currently, a default value is set as (threshold − 0.005) × num _ of _ transaction where num_of_transaction is total number of transactions. A domain expert can also determine the extension threshold.

Fig. 7. The selected range is moved to the head of the array

(a) num_original only considers the original part

(b) num_exten only considers the extension part. Fig. 8. Header tables and their corresponding FP-trees


729

The traditional FP-tree employs all frequent large 1-itemset as nodes. We are not interested on some items that are always frequent. Thus, we have to prune some large 1-itemset and specify a range of large 1-itemset. First, all the large 1-itemsets are put in an array in descending order. If the upper bound is not 1, then the range is moved to the head of the array. Figure 7 illustrates this process. Therefore, the interesting range of frequent patterns can be mined. Since the new FP-tree is built based on both original large 1-itemsets and extension large 1-itemsets, the header table must be constructed separately. Two parts are in the header table and controlled by two parameters. After extracting large 1-itemsets, two parameters, num_original and num_exten, are counted to represent the numbers from the original large 1-itemsets and extension large 1-itemsets, respectively. These parameters control the reading number of header table nodes. Figure 8 shows the two header tables and their corresponding FP-tree. While a FP-tree is mined, the number of paths is counted because the number of paths affects the decision making procedure that is executed recursively.

4 Experimental Results and Discussions Figure 9 demonstrates the entire functionalities in the system described herein.

Fig. 9. The guide system architecture

730


The guide system is developed in Embedded C 4.0 and runs in a PDA Windows Mobile environment. The main screen of the system is shown in Figure 10(a). The proposed system provides four languages including: 1) Chinese, 2) Taiwanese, 3) English and 4) Japanese. After selecting the language, users enter the guide system.

(a) The main screen of the guide system.

(b) The selection screen, for users to click.

(c) The screen for viewing browsing history.

(d) The related descriptions.

(e) The video introduction.

(f) Search by painted year.

(g) The keyword-based painting search.

(h) The texture descriptions for painting.

Fig. 10. The operations of the proposed guide system deployed in The Li Mei-shu Memorial Gallery


731

5 Conclusions An interactive guide system is proposed based on RFID-technology, information retrieval, association rules, and personalized recommendation, to assist visitors browsing exhibitions. Visitors view exhibition-related multimedia content on PDAs equipped with RFID-readers. The exhibition centers gather the visitors' information based on the interactions between the PDAs and the server and the results are used for making recommendations to future visitors. Acknowledgments. This work is supported by National Science Council, Taiwan under Grants NSC95-2745-E-036-001-URD and NSC95-2745-E-036-002-URD.

References 1. Kim, C.-Y., Lee, J.-K., Cho, Y.-H., Kim, D.-H.: VISCORS: A Visual Content Recommender For The Mobile Web. IEEE Intelligent Systems 19(6), 32–39 (2004) 2. Huang, Y.-P., Tsai, T.-W.: A Fuzzy Semantic Approach To Retrieving Bird Information Using Handheld Devices. IEEE Intelligent Systems 20(1), 16–23 (2005) 3. Brown, P.J., Bovey, J.D., Chen, X.: Context-Aware Applications: From The Laboratory To The Market Place. IEEE Personal Communications 4(5), 58–64 (1997) 4. Derntl, M., Hummel, K.A.: Modeling Context-Aware E-Learning Scenarios. In: The 3rd IEEE International Conference on Pervasive Computing and Communications Workshops, USA, pp. 337–342 (2005) 5. Haverkamp, D.S., Gauch, S.: Intelligent Information Agents: Review And Challenges For Distributed Information Sources. Journal of the American Society of Information Science 49(9), 304–311 (1998) 6. Landt, J.: The History of RFID. IEEE Potentials 24(4), 8–11 (2005) 7. Weinstein, R.: RFID: A Technical Overview And Its Application To The Enterprise. IT Professional 7(3), 27–33 (2005) 8. Roussos, G.: Enabling RFID in Retail. IEEE Computer 39(3), 25–30 (2006) 9. Billsus, D., Pazzani, M.J.: Learning Collaborative Information Filters. In: International Conference on Machine Learning, USA, pp. 46–54 (1998) 10. Gokhale, M., Miranda, A., Murnikov, T., Netes, P.D., Sartin, M.: Combining ContentBased And Collaborative Filters In An Online Newspaper. In: ACM SIGIR Workshop on Recommender Systems: Algorithms and Evaluation, USA (1999) 11. Good, N., Schafer, J., Konstan, J., Borchers, A., Sarwar, B., Herlocker, J., Riedl, J.: Combining Collaborative Filtering With Personal Agents For Better Recommendations. In: International Conference of the American Association of Artificial Intelligence, USA, pp. 439–446 (1999) 12. Han, J., Kamber, M.: Data Mining: Concepts and Techniques. Morgan Kaufmann Publishers, San Francisco (2001) 13. Cai, W.: The Extension Set And Incompatible Problem. Journal of Scientific Exploration 1, 81–93 (1983) 14. Huang, Y.-P., Chen, H.-J.: Using Extension Theory To Design A Fast Data Processing Model. In: IEEE-SMC International Conference, USA, pp. 3410–3415 (2000)

Dynamic Probabilistic Packet Marking with Partial Non-Preemption Wei Yen and Jeng-Shian Sung Computer Science and Engineering Department, Tatung University, Taipei, Taiwan, R.O.C. [email protected]

Abstract. This paper studies the technique of probabilistic packet marking. The main purpose is for the receiver to trace the true sources of the marked packets and rebuild their paths rapidly. In many existing schemes, marked packets are overwritten by other marking routers because of the nature of probabilistic packet marking. Although some IP traceback schemes can improve the overwriting problem, they usually make unrealistic assumption[11] or do not consider the load balancing issue of the routers. In probabilistic packet marking, how to address the re-mark problem and how to balance routers’ overhead are two important design considerations. A scheme is proposed in this paper to address these problems in probabilistic packet marking. Using computer simulation, we observe that our scheme can greatly reduce re-marking. When compared with the current IP traceback schemes, our scheme can reduce 65% marked packets than the other scheme. Moreover, the overhead of marking packet in each router is even out in the proposed scheme than the other. Keywords: DDoS, IP traceback, probabilistic packet marking, re-marking, convergence packet number.

1 Introduction In this age, the Internet has become an important way to get information. This technology enables revolutions in ways of doing business and increasing productivity. In the downside, many hackers or even criminals are attracted to using the Internet as their playground. They exploit network security pit holes and raise all kinds of security issues. Because new network attack techniques have been coming out and old ones have been morphing constantly, network security issues pose enormous threats and challenges in recent years. One attack technique is the so called Denial of Service (DoS) attack [1, 2, 3]. The Denial of Service (DoS) attack is an attack that prevents normal users from accessing their desired services. Typically, the attacker assaults the targeted victim system with malicious packets. The loss of network connections and services can be achieved by depleting the computing resources or consuming the bandwidth of the victim systems. In the worst case, if the load is heavy enough, the targeted victim cannot function well and will be forced to shut down. The goal of IP traceback is to discover the routing path or, at least, the maximum subset of it. This function can verify the sender of IP packets destined to a receiver. F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 732–745, 2008. © Springer-Verlag Berlin Heidelberg 2008

Dynamic Probabilistic Packet Marking with Partial Non-Preemption

733

One of the most important applications of IP traceback is to deal with certain DoS /or DDoS attacks, where the source IP addresses may be spoofed by the attackers. Once we find out the real perpetrator, we can hold him accountable for what he does. Also, it helps in mitigating DoS/or DDoS attacks by isolating the identified attack source. Packet marking is one technique of IP traceback schemes. Generally speaking, the router in a packet marking scheme will write the router’s information into the packet header when it decides to mark a packet. After the victim receives a number of the marked packets from the routers and collects sufficient information from the headers, the attack path and the sender of the packets can be computed. If the victim is attacked by malicious attackers, it can use these marked packets to trace the upstream router and find out the attack source by using received marked packets. In short, packet marking is a viable traceback technique that can help the victim to find the malicious attackers. Many DoS /or DDoS attackers use spoofed IP addresses to hide the traces of the attacks. To solve the spoofing problem, the packet marking technique can be employed which asks the routers in their way to the victim to leave information in the packets. This technique can be used to reveal the concealed identity of the sender by collecting the information marked in the attack packets. Although the packet marking scheme can be effective against the spoofing problem, the victim usually needs a large number of the marked packets to compute the attack path. The main reason for this inefficiency is contributed to the repeated marking by the routers in large. This re-marking will waste the effort of the upstream marking router and results in unbalanced number of packets marked by the routers. In this paper, we propose a novel scheme that requires less marked packets for the path calculation. Our approach can help the victim rebuild the path rapidly and reduce the marking overhead in the routers. In general, we use non-preemptive compensation to avoid repeated marking. Moreover, the marking probability for each router is no longer a constant. It is adjusted to so that the numbers of the packets marked by the routers are statistically equal. In summary, we are motivated to improve the performance the probabilistic packet marking by cutting the required number of the marked packets and lowering the router overhead. The rest of the paper is organized as follow. In Section 2, we will describe DDoS attack types and packet marking schemes including deterministic packet marking (DPM), probabilistic packet marking (PPM) scheme, and other IP tracebacks. In Section 3, we will present our proposed Dynamic Probabilistic Packet Marking with Partial Non-Preemption (DPPM-PNP) marking scheme. The supporting algorithms in the routers and the victim are also included in this chapter. In Section 4, we will analyze the performance of the DPPM-PNP and compare it with that of other schemes. Finally, we conclude the paper in Section 5 which highlights the main results of contributions of this research.

2 Hierarchical and Self Coupling Schemes In the following, we give coverage on two aspects. First, we would like to introduce DDoS attacks. The purpose is to understand whether the packet marking can be used

734

W. Yen and J.-S. Sung

Fig. 1. DDoS attack model

as a countermeasure to these attacking methods or not. Then, the existing packet marking schemes are surveyed. In Figure 1, it is shown that the packets from the attackers can take different attack paths. In the following, we outline several challenges when dealing with the DDoS attack. First, when the attack packet is sent to the victim, it may not be immediately identified as ill intended because it may very well appear normal to the victim. The attack packet will be treated as other normal packets and receive services. However, since its request is phony, the victim will waste its resources on serving the attack packet. This situation leads to degradation in system performance. Because the packet may have a legitimate source ID and request for common services, it is difficult for the victim to detect the attack packet. Many web sites employ some schemes such as typing in somewhat distorted letters to confirm human interaction before granting services. A trackback mechanism can be triggered to locate the source of the attack packet if a user fails repeatedly to pass the confirmation test. Secondly, when the victim does detect that it is under the DDoS attack, it may use the implemented defense mechanisms to counter the attack. However, it is possible that the zombies spoof their IP addresses to avoid being pinpointed by the victim [4]. IP spoofing is for the source to replace its IP address in the packet header with another IP address. IP spoofing can conceal the true source and cover up the attack path. Hence, even if the attack is detected by the victim, the true identities and locations of the zombies will still be elusive because of IP spoofing. To observe these attack network techniques, if these attacks are not solved or improved completely, the victim will be attacked by these malicious attackers. Next, we will introduce the packet marking technique for addressing these problems.


735

2.1 Probabilistic Packet Marking (PPM) With the PPM schemes, each router marks packets randomly and independently [5, 6, 8]. A random number is generated when a router receives a packet. It is compared with a pre-defined threshold, known as marking probability. The router will mark the packet if the random number is smaller than the marking probability. Otherwise, the packet is left intact. Therefore, a packet can be marked multiple times by the routers on the path. The previous marking will be overwritten by the latter one. This phenomenon is referred to as the re-marking problem. As we will explain later, remarking will slow down the traceback. When the receiver tries to rebuild the path, it depends on the router information carried in the packets. Since each marked packet contains only a fraction of the required information, the receiver has to wait for all available information to arrive. It should be noted that when the marking function is partially supported in the network, to traceback the entire path may be impossible. Hence, we are interested on finding the so called maximum subset of the route which is an ordered sequence containing the routers with marking capability in the path. In addition, the number of marked packets necessary for the receiver to compute this maximum subset is called the convergence packet number.

p (1 − p) d −1

(1)

There are several variations of PPM including node append, node sampling, and edge sampling [7]. Node append requires unavailable space in the packet header to accommodate marking information. It is largely considered unrealistic approach. The node and edge sampling are both subject to the re-marking problem. Eq. (1) characterizes the probability of a marked packet reaching its receiver. The p value is the marking probability of each router and d denotes the number of routers between the current router and the receiver. If the marking probability is too large, the receiver may have difficulty to get marked packets from the routers close to the source of the packet. Eq. (1) becomes a constraint as far as the convergence packet number is concerned. 2.2 Probabilistic Packet Marking with Non-Preemptive Compensation (PPM-NPC) The PPM-NPC scheme improves the edge sampling scheme in the PPM scheme [9]. In this scheme, a compensation counter is adopted in each router to allow nonpreemptive actions, i.e., no overwriting by the downstream routers. The idea is detailed in the following. Assume a marked packet goes through a router which is supposed to mark the packet based on its random number and the marking probability. In the PPM-NPC scheme, the packet is not marked by the router. Instead, a compensation counter in the router will be increased by one. This modification removes re-marking situations. The compensation counter, hence, records the number of packets that should be marked by the router in the future. When (i) the router receives a packet that has never been marked previously and

736


(ii) the compensation counter is not zero, the router will mark the packet and decrease the counter by one. In summary, the re-mark problem in the PPM scheme is avoided in the PPM-NPC scheme. In turn, the numbers of the required marked packets are reduced. According to [19], 52% reduction is achieved by the PPM-NPC scheme in some cases when compared with the edge sampling scheme. 2.3 Efficient Dynamic Probabilistic Packet Marking (EDPPM) The EDPPM adjusts the marking probability of each router depending on its location in the route. Marking probabilities of the routers far away from the receiver are higher than those of the closer routers. The adjustment in marking probabilities is to ensure that the numbers of the marked packets by each router are statistically the same. Basically, this scheme uses the TTL value of the marked packet to change the marking probability of the router. Each router can get its marking probability to mark the packet. Using this designed, it will reduce re-mark problem for the receiver because of the marking probability. For the remote routers from the receiver, their marking probabilities are much higher than other routers close the receiver. pd = 1 / d ;

d = 1, 2,3,...

(2)

Eq. (2) is the marking probability designed with the EDPPM scheme. The parameter, d, indicates the hop distance from the source to the router and pd is marking probability of the router d. The marking probability of the router nearest the source is the highest. When the routers are getting closer to the receiver, their marking probability will become smaller. Then, the marked packet of the remote router will get easier delivery to the receiver because of the mitigated re-mark problem. According to the calculation in [10], the probability a marked packet can reach the destination is independent of the router where it is marked. The probability is fixed at 1/D where D is the hop distance of the route. Although the re-mark problem seems solved in the PPM-NPC scheme, a sufficient condition must be met first. When the marking probability of each router is relatively

Fig. 2. Drawback of the PPM-NPC scheme


737

smaller, then this scheme will work as designed. However, if the marking probability with each router is high enough, some routers will never have a chance to mark their router information. Specifically, if the multiplication of the number of the router in the path and the marking probability exceeds one, then some of the compensation counters will not converge. For example, if the number of the router are 15 and the marking probability is 0.1 as shown in Figure 2, then the compensation counters in the last five routers near to the receiver will go to infinity because there will be no mark-free packets for the counters to deflate. Therefore, in some cases, when the marking probability is too high, the receiver will never be able to find the complete path. To tackle this critical problem, we can set a small marking probability that can accommodate most routes in the practical networking environment. Unfortunately, this solution will become inefficient for shorter routers and defies the goal of this scheme. This issue is addressed in the proposed DPPM-PNP scheme. The routers cannot leave their marks on packets in the stochastic sense are referred to as starving routers.

3 Dynamic Probabilistic Packet Marking with Partial Non-Preemption (DPPM-PNP) The DPPM-PNP scheme utilizes the un-rewrite property of the PPM-NPC scheme to mark the packets. About non-convergence problem of the PPM-NPC scheme, the DPPM-PNP scheme uses dynamic marking probability for the starving router to remark the marked packets. The dynamic marking probability is used by the starving router to adjust their marking probability. If the starving routers are nearer the receiver, their marking probabilities are smaller than the farther starving routers. This approach can reduce re-marks for the receiver. Our scheme can achieve convergence with a small number marked packets to the receiver. In the DPPM-PNP scheme, we utilize the start, end, and distance fields in marked packets from the edge sampling scheme. In addition, our scheme also marks two pieces of router information. They are Time-To-Live (TTL) and the marking probability of the router. They are marked into the ID field of the marked packet by the marking routers. The marked TTL can help to determine distances between the last router marking the packet and the current router. The marking probability of the router is recorded in the packet for the downstream starving routers to compute their marking probability. The marking probabilities of the starving routers need to be kept low because the DPPM-PNP scheme uses these probabilities for the re-marking function. We know a general operating system will set the TTL to one of the values of 32, 64, 128, and 255 [12]. These four values are considered as default values. And, there are very few packets will travel though more than 25 routers on the practical network, [14, 15]. In fact, a packet usually passes through 17 routers on average [13]. In DPPM-PNP, each router retrieves the marked_TTL, the marked_P, the packet_TTL, and distance value from a marked packet header. In the first step, if the router is a starving router, it uses the information in this database to compare with the retrieved

738


information. When a router receives a packet, the distance value of the packet and the distance value in the routing table are compared. When the former is smaller, this router will update its database with this new value. Otherwise, this router will keep its original distance value. The objective is to figure out the the minimum distance between the router and the source. The second step is about changing marking probability for the starving routers. If the marking probability is high, some routers suffer from starvation as in the PPMNPC scheme. So, we utilize the compensation counter in each router. When the counter reaches a threshold value, the marking probability needs to be changed. When a router receives a marked packet and leave it along, the compensation counter will be increased by one. However, if the router finds a mark-free packet while its compensation counter is not equal to zero, then it will mark the packet and reduce the counter by one. If the counter reaches C = 2/p, then the router concludes that it is in a starving state where p is the marking probability of the router. In Figure 3(a), it is shown how to change and compute marking probability for the starving routers. For example, if there 30 routers labeled from 0 to 29. Their marking probability is 0.04. In such a case, the five routers near the receiver will rarely see any mark-free packets. Hence, their compensation counters will be increased if they decide to mark the marked packet. We assume the 26th router’s compensation counter gets to the threshold value C. Thus, the 26th router computes a new marking probability with Eq. (3).

Pi =

1 1 + ( Marked_TTL − Packet_TTL ) Marked_P

(3)

where Pi is the new marking probability of the starving router and P is the original marking probability. Using this formula, the new marking probability of 26th router is 1/26 in Figure 3(a). After changing its marking probability, the 26th router will use this new marking probability to re-mark the marked packets. Therefore, the receiver will receive the marked packets from the 26th router marked. In our example, after the 26th router changes its marking probability and starts to re-mark the marked packets, the other starving routers may start to adjust their marking probability due to threshold violation. As shown in Figure 3(b), the compensation counter of the 27th router also exceeds the threshold value. It will then adopt Eq. (3) to compute the new marking probability and enters the rewrite mode. Using the provided information, the 27th router will obtain 1/27 as its new marking probability. It should be noted that the starving router in the downstream of another starving router may get into the rewrite state prior to its upstream peer. This is because the compensation counter in the router operates independently. In fact, the starving routers closer to the destination are usually prone to earlier entrance to the rewrite state. In Figure 3(c), such a case is illustrated. Using Eq. (3) and the required information, the new marking probability is calculated as demonstrated in Figure 3(c). It is obvious that the new marking probabilities are identical regardless of the situation under which the starving router gets into the rewrite state.


739

(a)

(b)

(c) Fig. 3. An example of the DPPM-PNP

4 Performance Evaluation In this section, we compare our scheme with other existing probabilistic packet marking mechanisms. First, the convergence packet numbers of the DPPM-PNP scheme, the Edge sampling scheme, and the PPM-NPC scheme are measured and compared. The convergence packet number indicates the number of packets received by the destination before it sees at least one marked packet from each of the routers in the connection path. The convergence packet number is expected to increase as the length of the path extends. In Figure 4, the marking probability is fixed at 0.01. In this experiment, it is shown that the edge sampling scheme is inferior to the PPM-NPC scheme and the DPPMPNP scheme. On the other hand, the PPM-NPC scheme and the DPPM-PNP scheme

740

W. Yen and J.-S. Sung DPPM-PNP P=0.01 Edge Sampling P=0.01 PPM-NPC P=0.01

1000

re 750 b m uN te 500 kc aP 250

0 5

10

15 Path length

20

25

Fig. 4. Convergence packet number (P = 0.01) DPPM-PNP P=0.1 Edge Sampling P=0.1

500

PPM-NPC P=0.1

400

re b 300 m u N te kc 200 aP 100

0 5

10

15

20

25

Path length

Fig. 5. Convergence packet number (P = 0.1)

share similar performance. In this simulation, the marking probability is relatively small given the path length. The re-mark problem is mild. Actually, most of these convergence packets are marked-free packets. In other words, more mark free packets arrive at the receiver than marked packets. As we explained previously, a small marking probability will be inefficient in terms of path convergence time. The PPM-NPC scheme and the DPPM-PNP scheme are better than the edge sampling scheme because they completely remove the re-marking problem. To see how the marking probability affects the performance of these schemes, it is changed to 0.1 and the experiment is repeated. In Figure 5, the performance is plotted. When the marking probability is 0.1, the convergence packet numbers demonstrate salient drop in all three marking schemes. It is contributed to the elimination of

Dynamic Probabilistic Packet Marking with Partial Non-Preemption Table 1. Convergence packet number in the PPM-NPC

Router number P

5

10

15

20

25

0.01 226 292 338 365 385 0.02 118 146 170 185 195 0.03

78

99 113 122 129

0.04

59

75

83

94

124

0.05

39

51

61

91

∞

0.06

39

51

62

∞

∞

0.07

30

40

∞

∞

∞

0.08

29

39

∞

∞

∞

0.09

24

36

∞

∞

∞

0.1

22

44

∞

∞

∞

0.2

14

∞

∞

∞

∞

0.3

∞

∞

∞

∞

∞

DPPM-PNP P=0.2 Edge Sampling P=0.2

4000

PPM-NPC P=0.2

3000

re b m u N te 2000 kc aP 1000

0 5

10

15

20

Path length


25

741

742

W. Yen and J.-S. Sung DPPM-PNP P=0.05 EDPPM Edge Sampling P=0.05 PPM-NPC P=0.05

300

250

re 200 b m u N 150 etk ca P100 50

0 5

10

15 Path length

20

25


marked-free packets. Nevertheless, the re-marking problem starts to become troublesome. In addition, the non-convergence problem of the PPM-NPC scheme takes place when the path length is greater than 10. This situation prohibits the routers closer to the destination from getting any mark free packet. These routers become starving and the destination will not see any packet marked by these starving routers. The PPM-NPC scheme shines in this situation because it allows the starving routers to switch to the rewrite mode. Table 1 emphasizes the non-convergence problem of the PPM-NPC scheme. It occurs when the product of the path length and the marking probability exceeds one. The DPPM-PNP scheme does not have this problem. Moreover, it outperforms the other two schemes in Figure 5. When the marking probability is set to 0.2, the marked-free packet numbers are further reduced in Figure 6. However, the re-marking problem becomes so serious that the receiver needs 3140 packets to converge when the edge sampling scheme is used and there are 25 routers in the path. The PPM-NPC scheme is basically useless in this situation. Next, we will consider the overhead of the packet marking schemes at each router. We compare the four schemes including the DPPM-PNP scheme, the edge sampling scheme, the PPM-NPC scheme, and the EDPPM scheme. At first, we show the convergence packet numbers for these schemes. In Figure 7, we can find the EDPPM scheme is better than the other schemes. This reason is that the EDPPM scheme adjusts the marking probability of each router based on its order in the path. This approach can reduce the re-mark problem and the receiver will see similar numbers of marked packets from the routers. It is noted that the DPPM-PNP and EDPPM share similar performance as the length of the path increases.


743

DPPM-PNP P=0.05 EDPPM Edge Sampling P=0.05 PPM-NPC P=0.05

7000 6000

re 5000 bm uN te 4000 cka Pg 3000 inkr aM2000 1000 0 1

2

3

4

5 6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Path length

Fig. 8. Number of marked packets by each router observed by the receiver

100%

DPPM-PNP P=0.05 EDPPM

90%

Edge Sampling P =0.05

80%

PP M-NPC P=0.05

)% (r 70% eb m uN 60% te 50% kc aP gni 40% kra 30% M 20% 10% 0% 1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Path length

Fig. 9. Marking overhead for each router

In Figure 8, 100000 packets are sent from the source to the receiver with the marking probability set to 0.05 except for the EDPPM. The numbers of marked packets by each router are counted and summarized in this figure. We see for both EDPPM and DPPM-PNP the numbers of packets marked by the routers are close to one another. However, this observation hides an important issue for the EDPPM. That is, the routers close to the source suffer from heaving workload imposed by packet

744


marking. On the other hand, the DPPM-PNP scheme uses the compensation counter to avoid this problem. In Figure 9, the marking overhead is illustrated. If the traffic is heavy in the network, the routers with heavy marking workload may experience unacceptable system slowdown. In the worst case, this situation can lead to intensive packet loss at these routers. The DPPM-PNP scheme does not have this issue because each router marks approximately the same number of packets as Figure 9 suggests.

5 Conclusion We review packet marking schemes and introduce a novel solution in this paper. The early works focus on the issue of the convergence packet number and the re-mark problem are rule based without the overhead and the marking probability consideration. Our scheme exhibit several good characteristics. First, the convergence is guaranteed. Secondly, the marking probability and the router number can be randomly selected. Out scheme will adapt to the operating environment. In the experiments, we compare our scheme with the edge sampling scheme, the PPM-NPC scheme, and the EDPPM scheme. In our scheme, the convergence packet number is reduced 65% than the edge sampling scheme if the marking probability is 0.2 and the router number is 25. And, our scheme solves the non-convergence problem in the PPM-NPC scheme. In addition, we analyze the marking packet number of each router with the various marking probability for the receiver. We discover the marking packet numbers of the routers are average and better than the EDPPM scheme. So, the reducing re-mark problem, the shortening convergence packet number, and balancing the packet number of each router are finished for the receiver by our scheme. Acknowledgments. The author would like to express their appreciation to the support of National Science Council and Tatung University for supporting this research under grant numbered NSC 95-2745-E-036-004-URD and B95-I02-035, respectively.

References 1. CERT Coordination Center, CERT CA-1997-28 IP Denial-of-Service Attacks (December 1997), http://www.cert.org/advisories/CA-1997-28.html 2. CERT Coordination Center, Denial of Service Attacks (February 1999), http://www.cert.org/tech_tips/denial_of_service.html 3. CERT Coordination Center, Internet Denial of Service Attacks and the Federal Response (February 2000), http://www.cert.org/congressional_testimony/ Fithen_testimony_Feb29.html 4. Ferguson, P.: Network Ingress Filtering: Defending Denial of Service Attacks Which Employ IP Source Address Spoofing, RFC 2267 (January 1998) 5. Belenky, A., Ansari, N.: On IP traceback. IEEE Communications Magazine 41(7), 142– 153 (2003) 6. Savage, S., Wetherall, D., Karlin, A., Anderson, T.: Network support for IP traceback. IEEE/ACM Transactions on Networking 9(3), 226–237 (2001)


745

7. Savage, S., Wetherall, D., Karlin, A., Anderson, T.: Practical network support for IP traceback. In: Proc. of ACM SIGCOMM 2000, vol. 30, pp. 295–306 (August 2000) 8. Park, K., Lee, H.: On the effectiveness of probabilistic packet marking for IP traceback under denial-of-service attack. In: Proc. of IEEE INFOCOM 2001, vol. 1, pp. 338–347 (April 2001) 9. Tseng, Y., Chen, H., Hsieh, W.: Probabilistic Packet Marking with Non-Preemptive Compensation. IEEE Communication Letter 8(6), 359–361 (2004) 10. Liu, J., Lee, Z.-J., Chung, Y.-C.: Efficient dynamic probabilistic packet marking for IP traceback. In: The 11th IEEE International Conference on Networks, pp. 475–480 (October 2003) 11. Al-Duwairi, B., Govindarasu, M.: Novel Hybrid Schemes Employing Packet Marking and Logging for IP Traceback. IEEE Transactions on Parallel and Distributed Systems 17(5) (May 2006) 12. SWITCH. Default TTL Values in TCP/IP (1999), http://www.switch.ch/docs/ttl_default.html 13. Theilmann, W., Rothermel, K.: Dynamic Distance Maps of the Internet. In: Proc. of the 2000 IEEE INFOCOM Conference, vol. 1 (March 2000) 14. Carter, R., Crovella, M.: Server Selection Using Dynamic Path Characterization in WideArea Networks. In: Proc. of the IEEE INFOCOM Conference, vol. 3 (April 1997) 15. Cooperative Association for Internet Data Analysis, Skitter analysis (2000), http://www.caida.org/tools/measurement/skitter

Fractal Model Based Face Recognition for Ubiquitous

Environments Shuenn-Shyang Wang, Su-Wei Lin, and Cheng-Ming Cho Department of Electrical Engineering Tatung University 40 Chungshan N. Road, 3rd Sec. Taipei, Taiwan, R.O.C. [email protected]

Abstract. In Ubiquitous environments, it is required to identify the user in order to utilize the knowledge of the user’s behavior and situation. In this paper, we propose a novel method of face recognition using dominant facial region extraction and fractal model. In order to improve the performance of the face recognition system, we propose an algorithm to extract the dominant facial region from the face images that includes the most discriminated part of the face, and for each dominant facial region it was presented by its fractal model and stored in database. The fractal model of the dominant facial region is then utilized as fractal facial features for face recognition. To further improve the performance of the face recognition system , we also propose the techniques of weighting mask and DC_Free MSE. Finally, some experimental results are presented and demonstrate the excellent performance of our face recognition approach.

1 Introduction A smart environment has various sensors and digital appliances networked together. To enable a user to obtain high-level services from networked appliances , it is required to identify the user in order to utilize the knowledge of the user’s behavior and situation without any explicit manipulation of computers. Therefore, it is desirable to have a realizable face recognition system that can identify the user with high speed and high accuracy. In the literature, many algorithms have been proposed for face recognition[1-5]. The fractal model has been applied in many applications[6-14] and recently it has been devised as a new approach to face recognition [6-10]. Up to present, the face recognition approaches based on fractal model are implemented using the whole face with hair. It is expected that the fractal based face recognition system can be further improved. In this paper we propose some new techniques to improve the performance. Firstly, we propose an algorithm to extract the dominant facial region from the face image that includes the most discriminated part of the face, so that the influence of different hair styles can be removed. Then dominant facial region is presented by its fractal model and stored in database. The fractal model of the dominant facial region is thus utilized as fractal facial features to realize the face recognition. Note that the dominant facial region is a square region of the face including eyes, nose F.E. Sandnes et al. (Eds.): UIC 2008, LNCS 5061, pp. 746–760, 2008. © Springer-Verlag Berlin Heidelberg 2008

Fractal Model Based Face Recognition for Ubiquitous Environments

747

and mouth, so that the influence of the different hair styles can be removed. Secondly, the weighting mask and DC_Free MSE (mean square error) are proposed in recognition stage. For the same person it may be with some different facial expressions, such as a smile, an angry or a sad expression. The weighting mask is proposed to enhance the performance of the system against some variation in expressions. Moreover, in different lighting condition the recognition error for the same person may be variant greatly. The DC_Free MSE is presented to eliminate the effect of lighting variation. The rest of this paper is organized as follows. Section 2 proposes an algorithm to extract the dominant facial region from the face image. Section 3 shows how to obtain the fractal facial features from the dominant facial region and presents the new techniques of weighting mask and DC_Free MSE to construct an improved face recognition system. Experimental results and conclusion are given in section 4 and 5 respectively.

2 The Dominant Facial Region Extraction This section presents a method to extract the dominant facial region from the input face image. We deal with the face image by Sobel Operator and binarization operator[14] to get the binary image with enhanced contour of the face. Then the binary image will be normalized to a preset size and is dealt with histogram equalization to reduce the influences of the illumination. There are seven steps needed to locate the dominant facial region of a face image. Step 1: The face image, as shown in Fig. 1, is first treated by Sobel operator and binarization operator to get the binary image as shown in Fig. 2. By scanning from left side to right side and summarizing the number of the mark pixels in the resultant binary image column by column, when one column contains more than 8 mark pixels with

Fig. 1. The face image

748

S.-S. Wang, S.-W. Lin, and C.-M. Cho

Fig. 2. The binary image

value “1” and appears first, we consider this column as the left boundary of the head. By the same way, we can find the right boundary of the head. Note that the reason for choosing 8 mark pixels is to reduce the effect of noise. Step 2: Detect the top of the head which is the upper boundary of the head. Step 3: With row by row scanning we locate the vertical position of the neck by finding the smallest distance between right boundary and left boundary of the face in binary image. Step 4: We intend to locate the vertical position of the eyes and the mouth. In view of Fig. 3, we denote the distance between head and eyes as D_he, distance between head and mouth as D_hm and distance between head and neck as D_hn. By the fact that the vertical position of the eyes is about (0.43~0.50)*D_hn from the top of the head, we find the vertical projection of the region and its maximum is the vertical position of the

Fig. 3. Some distance defined in the binary image


749

eyes. By the same way, we can locate the mouth in the face by the knowledge that the vertical position of the mouth is at (0.73~0.85)*d_hn from the top of the head. Step 5: We intend to locate the vertical centerline of a face by the symmetric property of the face. We define a search window that enclosed the face. Since the face is symmetric with respect to its vertical centerline, based on this property we define a symmetric measurement to search the vertical center line. For each pixel with coordinate ( i , j ) in the search window, a symmetric measurement is set as R

s (i, j ) = ∑ f (i, j + k ) − f (i, j − k )

(2.1)

i =1

where R is a constant indicating the range for computing the symmetric measurement and f (i, j) is the gray value on the position (i, j). The vertical centerline is found to be the vertical line with smallest s(i,j). Step 6: We intend to locate the location the horizontal position of eyes. Denoting the width of head as D_hw, we can locate the horizontal position of right eye at the region about (center - 0.2*D_hw ~ center-0.15*D_hw). By the same way, the horizontal position of left eye is at the region about (center + 0.15*D_hw ~ center + 0.2*D_hw). Then we just find the maximum projection of the region and which is the place where the eyes lie. Step 7: We will locate the dominant facial region of the face image, as shown in Fig. 4. First, we define a vertical margin of face as margin_y, which is equal to D_hn/13. Then the upper boundary of the face image we want to locate is at eyes-2*D_hn/13 and the lower boundary of face is at mouth-D_hn/13. Now we can get the width of face is face_size = | (mouth-D_hn/13) (eye-2*D_hn/13) |. Based on the center line of face what we get at step 5, we can set the left and right boundary of the dominant facial region of the face image as (center-face_size/2) and (center+face_size/2), respectively.

－

Fig. 4. The square region marked by square on the face is the dominant facial region

After we obtain the dominant facial region using above steps, we normalize the dominant facial region into the preset dimension. And the histogram equalization[14] is applied to the normalized dominant facial region. The lighting effect in dominant facial region can thus be reduced. Fig. 5 shows the effect of histogram equalization.

750


(a)

(b)

Fig. 5. (a) before histogram equalization (b) after histogram equalization

3 Face Recognition Using Fractal Model After we extract the dominant facial region followed by applying normalization and histogram equalization, we will apply the fractal image coding [9,11] to obtain the corresponding fractal facial features. Fig 6 illustrates the concept of face recognition by fractal model. There are two stages for face recognition: training stage and recognition stage. The first step in training stage is to extract fractal facial feature from the dominant facial region. In order to obtain the fractal facial features of a dominant facial region, the dominant facial region is partitioned into non-overlapping smaller blocks (range blocks) and overlapping larger blocks (domain blocks). For each range block, a full search is done to find a domain block whose contractive transformation best approximates the range block. A distance metric such as root mean square (RMS) can be used to find the approximation error. The parameters of the related transformations for each range block are then stored. The union of these parameters represents the fractal facial features of the face image. In the recognition stage the face recognition can be achieved by using the fractal facial features obtained in training stage. The dominant facial region of the input face image is iterated once by the fractal facial feature of each object (person) in face database. Then, the distances from the dominant facial region of the input face image to that of each iterated face image are calculated using mean square error (MSE). which is defined as

MSE =

1 N2

N

N

i =1

j =1

∑∑

( A ij − B ij ) 2 where 1 ≤ i , j ≤ N

where N is the size of the block,

(3.1)

Aij and Bij are the gray value of pixels in block A

and block B. Finally, we normalize all the MSEs between 0~1 by maximum MSE among them to define as recognition error (RE):

RE i =

MSE i max ( MSE i ) 1≤ i ≤ N

(3.2)


751

MSEi is the MSE between the input face image and the iterated face image using fractal facial features of object i . The minimum RE is the smallest REi ; that is, RE= min RE i . If the minimum RE is less than a threshold, the input face image is where

i

recognized as the corresponding object, else the input face image is considered not in

Fig. 6. Face recognition by fractal model

After first iteration Fig. 7(a). The face image of the person No.6 is iterated once by itself fractal facial feature

After first iteration Fig. 7(b). The face image of the person No.1 is iterated once by the fractal facial feature of person No.6

752


Fig. 7(c). The recognition errors for input face image of person No. 6. Note that minimum

RE6

is the

RE

the face database. Fig 7 shows an example that a person No.6 is recognized by fractal-based face recognition approach. To further improve the performance of the face recognition system , we propose two techniques: Weighting mask and DC_Free MSE. (A) Weighting Mask: Here, a weighting mask for calculating MSE is proposed to remove the influence of facial expressions. Fig 8 shows the T-shape weighting mask in the face region. The square region is the dominant facial region and the weighting of the pixels in T-shape region is 1.0 and the weighting of the pixels outside the T-shape region is 0.1. The weighting is set by the observation that in the T-shape region which included eyes and nose there is rare change for different face expressions. Thus the minimum recognition

L

2L/3

L/4

L/3

(a) the T- shape weighting mask in the facial

(b) the region of the weighting mask

Fig. 8. The T-shape weighting mask and the corresponding region


753

error of the same person in database is almost invariant even for different expressions. The experimental results will be shown in section 4. (B) DC_Free MSE: Fig 9 shows the facial images under different lighting condition. In brighter or darker lighting condition the MSE between the dominant facial region of input face image and that of iterated face image may vary greatly for the same person. So we propose the DC_Free MSE to avoid the effect of the variant lighting condition. The idea is to delete the average component of the dominant facial region from the MSE in the recognition stage. The DC_Free MSE is defined as:

DC_Free MSE =

1 N N ( (( Aij − avgA ) − ( Bij − avgB )) 2 ) 2 ∑∑ N i =1 j =1

where NxN is the block size.

Aij and Bij

(3.3)

are the gray value of pixels in block A and

block B. avgA and avgB are the average gray value of block A and block B, respectively. The recognition error based on DC_Free MSE is given by

RE i =

DC_Free MSE i max ( DC_Free MSE i )

(3.4)

1≤i ≤ N

where

DC - Free MSE i is the DC_Free MSE of the input face image and iterated

face image for person No. i , and N is the number of objects in face database.

(a)

(b)

(c)

Fig. 9. (a) dominant facial region in normal lighting condition. (b) dominant facial region in lower lighting condition. (c) dominant facial region in brighter lighting condition.

Finally, an improved face recognition system based on the weighting mask and DC_Free MSE can be constructed. The procedure is summarized as following: Training Stage: Step1: Extract the dominant facial region from an input face image. Step2: Obtain the fractal facial features by fractal coding of dominant facial region. Step3: Store the fractal facial features of the dominant facial region in face database. Recognition Stage: Step1: Extract the dominant facial region from an input face image.

754


Step2: Iterate the dominant facial region of input face image by fractal facial features of each person in face database. Step3: Calculate all the recognition errors with DC_Free MSE and weighting mask Step4: Find the object (person) with minimum recognition error.

4 Experimental Results In order to assess the performance of the proposed algorithm, several experiments are carried out on a database consisting of 30 front view face images of size 320x240 with 256 gray levels. The whole experiment is conducted in two stages. In first stage, the dominant facial region of the face image will be located, extracted and normalized. Fig. 10 shows the dominant facial region of total 30 candidates. The dimension of the facial feature after normalization is 128x128, and all the fractal facial features of the face image are obtained and stored. Here, the domain block size and the range block size are

No.1

No.2

No.3

No.5

No.6

No.7

No.9

No.10

No.11

No.4

No.8

No.12

Fig. 10. Extracted dominant facial region of 30 persons in face database


No.13

No.14

No.15

No.16

No.17

No.18

No.19

No.20

No.21

No.22

No.23

No.24

No.25

No.26

No.29

No.30 Fig. 10. (continued)

No.27

No.28

755

756


set to 4x4 and 8x8 pixels, respectively. In the second stage, the face recognition procedure is performed. 4.1 The Experimental Results of Face Recognition by Fractal Model In the first experiment the lighting condition in the surrounding is well controlled. Fig 11 shows the partial results when person No. 9 and person No. 22 are input for recognition. As expected, it reveals that the recognition error corresponding to itself is the minimum one among all. And, when all persons are input for recognition the overall experimental results demonstrate that 100% recognition rate is achieved.

(1)The RE for input face of person No. 9

(2) The RE for input face of person No. 22

Fig. 11. Shows the partial results for two input face images

4.2 The Experimental Results with Weighting Mask In second experiment a weighting mask is applied for the case that the person No.13 is input for recognition. The face images of person No.13 with several facial expressions, as shown in Fig 12, are input for recognition and the recognition errors of them are

(a) A normal face

(b) a smile face

(c) a sad face

Fig. 12. The person No.13 with different expressions


757

(a) The recognition error of the person No.13 with no

(b) The recognition error of the person No.13 with a

expression (minimum recognition error is 0.385)

smile (minimum recognition error is 0.452)

(c) The recognition error of the person No.13 with sadness(minimum recognition error is 0.468)

Fig. 13. The variation of minimum Recognition Error for the same person with different expressions

(a) The recognition error of the person No.13 with

(b) The recognition error of the person No.13 with smile

normal expression (minimum recognition error is

expression(minimum recognition error is 0.291)

0.287)

Fig. 14. The minimum recognition error remains close for the same person with different facial expressions

758


(c) The recognition error of the person No.13 with sad expression(minimum recognition error is 0.298)

Fig. 14. (continued)

calculated. Fig 13 shows the variation of recognition errors without weighting mask for the same person. Fig 14 shows the recognition errors with weighting mask. It is seen that by using weighting mask the minimum recognition errors for the same person with different expressions are close. This demonstrates that the influence of different expression can be overcome by using weighting mask. 4.3 The Experiment Results with DC_Free MSE In third experiment the lighting condition in surrounding is not well controlled in recognition stage. The luminance of input face could be influenced by lighting condition. Fig 15 shows the faces that are extracted under normal, darker, brighter lighting condition. Table 1 shows the recognition errors in accordance with MSE as well as DC_Free MSE.

(a)

(b)

(c)

Fig. 15. (a) the face in normal lighting condition (b) the face in darker lighting condition and (c) the face in brighter lighting condition


759

Table 1. The recognition errors of MSE and DC_Free MSE

Normal Face

Darker Face

Brighter Face

MSE

177.63

439.81

587.21

DC_Free MSE

170.59

164.01

169.21

It is found that the MSEs for normal face, darker face and brighter face are quite different, but their DC_Free MSEs are close. This demonstrates that using DC_Free MSE the influence of the lighting condition can be removed. 4.4 The Experimental Results with Weighting Mask and DC_Free MSE In this experiment the all faces in face database are recognized by the improved face recognition system which corporates the proposed techniques of weighting mask and DC_Free MSE.

(1)The RE for input face of person No. 9

(2) The RE for input face of person No. 22

Fig. 16. The partial experiment results recognized by the improved face recognition system

Fig. 16 shows the partial results when person No. 9 and person No. 22 are input for recognition. By comparing Fig. 16 with Fig.11, it is found that better performance is achieved for the improved face recognition system. Extensive experiments demonstrate the high recognition rate for the input face image with different expressions and different lighting conditions.

5 Conclusion In this paper, we have proposed an improved face recognition system using dominant facial region extraction and fractal model. In order to improve the performance of the face recognition system, we propose an algorithm to extract the dominant facial region

760


from the face images that includes the most discriminated part of the face, and then for each dominant facial region it was presented by its fractal model and stored in database. The fractal model of the dominant facial region is thus utilized as fractal facial features for face recognition. To further improve the performance of the face recognition system, we also propose the techniques of weighting mask and DC_Free MSE. Finally, the experiment demonstrates that high recognition rate can be achieved. Our approach needs less computational time in recognition and when a new face is added to the database the whole database needs not to be reconstructed.

References 1. Lawrencs, S., Giles, C.L., Tsoi, A.C., Back, A.D.: Face Recognition: A convolutional Neural Network Approach. IEEE Trans. on PAMI 20, 673–686 (1998) 2. Sakaue, F., Shakunaga, T.: Combination of projectional and locational decompositions for robust face recognition. In: Proc. IEEE Int. Workshop on Analysis and Modeling of Faces and Gestures, pp. 407–421 (2005) 3. Sakaue, F., Kobayashi, M., Migita, T., Shakunaga, T.: A real-life test face recognition system for dialogue interface robot in ubiquitous environments. In: The 18th Int. Conference on Pattern Recognition, pp. 1155–1160 (2006) 4. Lam, K.M., Yan, H.: An Analytic-to-Holistic Approach for Face Recognition Based on a Single Frontal View. IEEE Trans. on PAMI 20, 670–688 (1998) 5. Turk, M., Pentland, A.: Eigenfaces for recognition. Journal of Cognitive Neuroscience 3, 71–86 (1991) 6. Kouzani, A.Z., He, F., Sammut, K.: Towards invariant face recognition. Infornation Sciences 123, 75–101 (2000) 7. Tan, T., Yan, H.: Face Recognition by Fractal Transformation. In: IEEE International Conference on ASSP, pp. 3537–3540 (1999) 8. Jacquin, A.E.: Fractal Image Coding: A Review. Proc. of the IEEE 81(10), 1451–1465 (1993) 9. Lu, N.: Fractal Imaging. Academic Press, London (1997) 10. Wu, H.-Y.: A System of facial feature extraction and recognition, master thesis, Department of Electrical Engineering, Tatung University (1999) 11. Li, C.-H.: Two Level Fractal Image Coding Algorithm, master thesis, Department of Electrical Engineering, Tatung University (1996) 12. Jacobs, A.E., Fisher, Y., Boss, R.D.: Image coding based on a fractal theory of iterated contractive image transformations. IEEE Trans. Image Processing 1(1), 18–30 (1992) 13. Jacquin, A.E.: Fractal image coding: A Review. Processing of the IEEE 81(10), 257–261 (1993) 14. Jain, A.K.: Fundamentals of Digital Image Processing. Prentice Hall, New Jersey (1986)

Author Index

Apduhan, Bernady O.

61, 397

Govindaraju, Venu 75 Gu, Gaung-Hui 297

Bagula, Antoine B. 453 Becker, Benjamin 216 Beigl, Michael 647 Berbers, Yolande 46 Berchtold, Martin 647 Berger, Michael 143 Blumenthal, Jan 20 Bosse, Tibor 229 Braun, Iris 143 Bunoza, Ana 258 Catarinucci, Luca 119 Chamizo, Javier 105 Chang, Jae-Woo 677 Chang, Tsun-Wei 720 Chang, Yao-Jen 284 Chang, Yueh-Tsun 697 Cheah, Zi Bin 271 Chen, Bo-Wei 616 Chen, Zhaoji 3 Cho, Cheng-Ming 746 Cho, Sung-Bae 158, 535 Choi, Hyohyun 363 Choi, Keunho 386 Chowdhury, Mohammad M.R. Colella, Riccardo 119 Cui, Min-Woo 323 Dargie, Waltenegus 143 Decker, Christian 647 Deegan, Mark 490 DiBari, Angelo 119 Du, Kejun 187 Esposito, Alessandra

119

Fang, Hui 468 Faruk, A.B.M. Omar Filho, Julio Oliveira Fu, Ping-Fang 352

271 258

Garnes, H˚ avard Husev˚ ag 271 G´ omez, Juan Miguel 105

Haaland Thorsen, Kari Anne 626 Han, Guangjie 439 Han, Jong-Wook 548, 591 Harada, Fumiko 411 Hauswirth, Manfred 439 Herrmann, Klaus 20 Hong, Dowon 338, 548 Hong, Seung-Tae 677 Hoogendoorn, Mark 229 Hsu, Wen-Jing 468 Huang, Benxiong 426 Huang, Chen 426 Huang, Chung-Ming 636 Huang, Kuo-Feng 352 Huang, Runhe 61, 397 Huang, Yo-Ping 697, 720 Huber, Manuel 216 Hussain, Chauhdary Sajjad 323 Hwang, Ren-Hung 201 Indulska, Jadwiga

105

1

Jaatun, Martin Gilje 271, 602 Jayaraman, Bharat 75 Jeong, Ik Rae 338 Jeong, Young-Sik 591 Jho, Nam-Su 338 Jin, Qun 61 Kang, Joenil 548 Kashif, Ali 311 Kawamura, Takahiro 578 Kawashima, Tomomi 61 Kim, Dong-Sub 311 Kim, Jung-Hwan 311, 323 Kim, Kee-Bum 323 Kim, Kyoung-Yun 386 Kim, Tai-hoon 35 Klein, Michel C.A. 229 Klinker, Gudrun 216 Kuan, Ta-Wen 297 Kulyukin, Vladimir 244

762

Author Index

Kutiyanawala, Aliasgar 244 Kwon, Ohbyung 90, 386 Lee, Deok-Gyu 591 Lee, KyungHee 548 Lee, Yonnim 90 Lee, Young-Seol 535 Liang, Yan 482 Liao, Zhensong 35 Lin, Daw-Tung 710 Lin, Feiyu 687 Lin, Su-Wei 746 Line, Maria B. 271 Liu, Li-Wei 710 Locatelli, Marco P. 505 Loregian, Marco 505 Louta, Malamati 520 Ma, Jianhua 61, 397 MacCormac, Daniel 490 Maeng, YoungJae 548 Mazandu, Kuzamunu G. 453 Meersman, Robert 169 Menon, Vivek 75 Ming, Xue 311 Mohaisen, Abedelaziz 338, 548 Mtenzi, Fred 490 Nakamura, Yuichi 131 Nakanishi, Kei 397 Noll, Josef 105 Nyang, DaeHun 338, 548 O’Driscoll, Ciaran 490 O’Shea, Brendan 490 Ohsuga, Akihiko 578 Øyan, Petter 2 Pan, Liuqing 426 Park, Han-Saem 158 Park, Jong Hyuk 35, 591 Park, Myong-Soon 311, 323 Pelechano, Vicente 662 Preuveneers, Davy 46 Pyykk¨ onen, Mikko 563 Qin, Weijun

187

Reynolds, Vinny 439 Riedel, Till 647

Riekki, Jukka 563 Rong, Chunming 482, 626 Rosenstiel, Wolfgang 258 Rothermel, Kurt 20 Rudolph, Larry 468 Sanchez, Ivan 563 Sandkuhl, Kurt 687 Sandnes, Frode Eika 697, 720 Schuhmann, Stephan 20 Schøyen, Anne Liseth 2 Serral, Estefan´ıa 662 Shi, Yuanchun 187 Shimakawa, Hiromitsu 411 Shon, Taeshik 363 Shu, Lei 439 Sommer, J¨ urgen 258 Song, In-Jee 158 Springer, Thomas 143 Sun, Jun-Zhao 373 Sun, Zheng-Wei 616 Sung, Jeng-Shian 732 Sørensen, Jan Tore 602 Takada, Hideyuki 411 Tang, Yan 169 Tarricone, Luciano 119 Timmermann, Dirk 20 Treur, Jan 229 Tsai, Chang-Zhou 636 Tsai, Shan-Yi 201 Umezu, Keisuke

578

Valderas, Pedro

662

Wang, Chiung-Ying 201 Wang, Jhing-Fa 297, 616 Wang, Jia-chang 297, 616 Wang, Jun-Xuan 352 Wang, Shuenn-Shyang 746 Wang, Xinmei 426 Wang, Ying-Hong 352 Wedum, Petter 271 Wu, Shang-Yao 284 Wustmann, Patrick 143 Yamahara, Hiroyuki 411 Yang, Laurence Tianruo 35 Yau, Stephen S. 3 Yen, Wei 732 Yeo, Sang Soo 591

Author Index Yi, Zaiyao 426 Yu, Zhiwen 131 Zappatore, Marco 119 Zhang, Daqing 187

Zhang, Lin 439 Zhou, Xingshe 131 Zhou, Zhangbing 439 Zou, Deqing 35

763

Ubiquitous Intelligence and Computing: 5th International Conference, UIC 2008, Oslo, Norway, June 23-25, 2008 Proceedings

Autonomic and trusted computing: 5th international conference, ATC 2008, Oslo, Norway, June 23-25, 2008: proceedings

Coordination Models and Languages: 10th International Conference, COORDINATION 2008, Oslo, Norway, June 4-6, 2008, Proceedings

Ubiquitous Intelligence and Computing - UIC 2011

Ubiquitous Intelligence and Computing: Third International Conference, UIC 2006, Wuhan, China, September 3-6, 2006, Proceedings

Intelligence and Security Informatics: IEEE ISI 2008 International Workshops: PAISI, PACCF and SOCO 2008, Taipei, Taiwan, June 17, 2008, Proceedings

Ubiquitous Intelligence and Computing: 4th International Conference, UIC 2007, Hong Kong, China, July 11-13, 2007, Proceedings

Formal Methods for Open Object-Based Distributed Systems: 10th IFIP WG 6.1 International Conference, FMOODS 2008, Oslo, Norway, June 4-6, 2008 Proceedings

Architecture of Computing Systems - ARCS 2008: 21st International Conference, Dresden, Germany, February 25-28, 2008, Proceedings

Distributed Computing in Sensor Systems: 4th IEEE International Conference, DCOSS 2008 Santorini Island, Greece, June 11-14, 2008, Proceedings

International Symposium on Distributed Computing and Artificial Intelligence 2008

CONCUR 2008 - concurrency theory 19th international conference, CONCUR 2008, Toronto, Canada, August 19-22, 2008: proceedings

3rd Symposium of Ubiquitous Computing and Ambient Intelligence 2008 (Advances in Intelligent and Soft Computing)

Python Magazine (June 2008)

Professional Photographer June 2008

ECOOP 2004 - Object-Oriented Programming: 18th European Conference, Oslo, Norway, June 14-18, 2004, Proceedings

Image and Signal Processing: 3rd International Conference, ICISP 2008, Cherbourg-Octeville, July 1-3, 2008, Proceedings

Haptics: Perception, Devices and Scenarios: 6th International Conference, EuroHaptics 2008 Madrid, Spain, June 11-13, 2008, Proceedings

Tests and Proofs: Second International Conference, TAP 2008, Prato, Italy, April 9-11, 2008, Proceedings

Information and Communication Technologies in Tourism 2008: Proceedings of the International Conference in Innsbruck, Austria, 2008

Image Analysis: 16th Scandinavian Conference, SCIA 2009, Oslo, Norway, June 15-18, Proceedings

Agile processes in software engineering and eXtreme programming 9th international conference, XP 2008, Limerick, Ireland, June 10-14, 2008: proceedings

Data Integration in the Life Sciences: 5th International Workshop, DILS 2008, Evry, France, June 25-27, 2008, Proceedings

Theory and Applications of Satisfiability Testing - SAT 2008: 11th International Conference, SAT 2008, Guangzhou, China, May 12-15, 2008, Proceedings

Intelligence and Security Informatics: IEEE ISI 2008 International Workshops: PAISI, PACCF and SOCO 2008, Taipei, Taiwan, June 17, 2008, Proceedings (Lecture ... Applications, incl. Internet Web, and HCI)

Service Availability: 5th International Service Availability Symposium, ISAS 2008 Tokyo, Japan, May 19-21, 2008 Proceedings

Intelligent Tutoring Systems: 9th International Conference on Intelligent Tutoring Systems, ITS 2008, Montreal, Canada, June 23-27, 2008, Proceedings

Requirements Engineering: Foundation for Software Quality: 14th International Working Conference, REFSQ 2008 Montpellier, France, june 16-17, 2008, Proceedings

Mathematics of Program Construction: 9th International Conference, MPC 2008 Marseille, France, July 15-18, 2008 Proceedings

Wireless Internet Communications: 6th International Conference, WWIC 2008 Tampere, Finland, May 28-30, 2008 Proceedings

Inked Magazine, 2008 6, June

Ubiquitous Intelligence and Computing: 5th International Conference, UIC 2008, Oslo, Norway, June 23-25, 2008 Proceedings

Autonomic and trusted computing: 5th international conference, ATC 2008, Oslo, Norway, June 23-25, 2008: proceedings

Coordination Models and Languages: 10th International Conference, COORDINATION 2008, Oslo, Norway, June 4-6, 2008, Proceedings

Ubiquitous Intelligence and Computing - UIC 2011

Ubiquitous Intelligence and Computing: Third International Conference, UIC 2006, Wuhan, China, September 3-6, 2006, Proceedings

Intelligence and Security Informatics: IEEE ISI 2008 International Workshops: PAISI, PACCF and SOCO 2008, Taipei, Taiwan, June 17, 2008, Proceedings

Ubiquitous Intelligence and Computing: 4th International Conference, UIC 2007, Hong Kong, China, July 11-13, 2007, Proceedings

Formal Methods for Open Object-Based Distributed Systems: 10th IFIP WG 6.1 International Conference, FMOODS 2008, Oslo, Norway, June 4-6, 2008 Proceedings

Architecture of Computing Systems - ARCS 2008: 21st International Conference, Dresden, Germany, February 25-28, 2008, Proceedings

Distributed Computing in Sensor Systems: 4th IEEE International Conference, DCOSS 2008 Santorini Island, Greece, June 11-14, 2008, Proceedings

International Symposium on Distributed Computing and Artificial Intelligence 2008

CONCUR 2008 - concurrency theory 19th international conference, CONCUR 2008, Toronto, Canada, August 19-22, 2008: proceedings

3rd Symposium of Ubiquitous Computing and Ambient Intelligence 2008 (Advances in Intelligent and Soft Computing)

Python Magazine (June 2008)

Professional Photographer June 2008

ECOOP 2004 - Object-Oriented Programming: 18th European Conference, Oslo, Norway, June 14-18, 2004, Proceedings

Image and Signal Processing: 3rd International Conference, ICISP 2008, Cherbourg-Octeville, July 1-3, 2008, Proceedings

Haptics: Perception, Devices and Scenarios: 6th International Conference, EuroHaptics 2008 Madrid, Spain, June 11-13, 2008, Proceedings

Tests and Proofs: Second International Conference, TAP 2008, Prato, Italy, April 9-11, 2008, Proceedings

Information and Communication Technologies in Tourism 2008: Proceedings of the International Conference in Innsbruck, Austria, 2008

Image Analysis: 16th Scandinavian Conference, SCIA 2009, Oslo, Norway, June 15-18, Proceedings

Agile processes in software engineering and eXtreme programming 9th international conference, XP 2008, Limerick, Ireland, June 10-14, 2008: proceedings

Data Integration in the Life Sciences: 5th International Workshop, DILS 2008, Evry, France, June 25-27, 2008, Proceedings

Theory and Applications of Satisfiability Testing - SAT 2008: 11th International Conference, SAT 2008, Guangzhou, China, May 12-15, 2008, Proceedings

Intelligence and Security Informatics: IEEE ISI 2008 International Workshops: PAISI, PACCF and SOCO 2008, Taipei, Taiwan, June 17, 2008, Proceedings (Lecture ... Applications, incl. Internet Web, and HCI)

Service Availability: 5th International Service Availability Symposium, ISAS 2008 Tokyo, Japan, May 19-21, 2008 Proceedings

Intelligent Tutoring Systems: 9th International Conference on Intelligent Tutoring Systems, ITS 2008, Montreal, Canada, June 23-27, 2008, Proceedings

Requirements Engineering: Foundation for Software Quality: 14th International Working Conference, REFSQ 2008 Montpellier, France, june 16-17, 2008, Proceedings

Mathematics of Program Construction: 9th International Conference, MPC 2008 Marseille, France, July 15-18, 2008 Proceedings

Wireless Internet Communications: 6th International Conference, WWIC 2008 Tampere, Finland, May 28-30, 2008 Proceedings

Inked Magazine, 2008 6, June

Recommend Documents