Human-Computer Interaction. Ambient, Ubiquitous and Intelligent Interaction: 13th International Conference, HCI International 2009, San Diego, CA, ... Applications, incl. Internet Web, and HCI)

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

5612

Julie A. Jacko (Ed.)

Human-Computer Interaction Ambient, Ubiquitous and Intelligent Interaction 13th International Conference, HCI International 2009 San Diego, CA, USA, July 19-24, 2009 Proceedings, Part III

13

Volume Editor Julie A. Jacko University of Minnesota Institute of Health Informatics MMC 912, 420 Delaware Street S.E., Minneapolis, MN 55455, USA E-mail: [email protected]

Library of Congress Control Number: 2009929048 CR Subject Classification (1998): H.5, I.3, I.7.5, I.5, I.2.10 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/Web and HCI ISSN ISBN-10 ISBN-13

0302-9743 3-642-02579-X Springer Berlin Heidelberg New York 978-3-642-02579-2 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2009 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12707225 06/3180 543210

Foreword

The 13th International Conference on Human–Computer Interaction, HCI International 2009, was held in San Diego, California, USA, July 19–24, 2009, jointly with the Symposium on Human Interface (Japan) 2009, the 8th International Conference on Engineering Psychology and Cognitive Ergonomics, the 5th International Conference on Universal Access in Human–Computer Interaction, the Third International Conference on Virtual and Mixed Reality, the Third International Conference on Internationalization, Design and Global Development, the Third International Conference on Online Communities and Social Computing, the 5th International Conference on Augmented Cognition, the Second International Conference on Digital Human Modeling, and the First International Conference on Human Centered Design. A total of 4,348 individuals from academia, research institutes, industry and governmental agencies from 73 countries submitted contributions, and 1,397 papers that were judged to be of high scientific quality were included in the program. These papers address the latest research and development efforts and highlight the human aspects of the design and use of computing systems. The papers accepted for presentation thoroughly cover the entire field of human–computer interaction, addressing major advances in the knowledge and effective use of computers in a variety of application areas. This volume, edited by Julie A. Jacko, contains papers in the thematic area of Human–Computer Interaction, addressing the following major topics: • • • • • •

Mobile Interaction In-vehicle Interaction and Environment Navigation Agents, Avatars and Personalization Ambient Interaction Affect, Emotion and Engagement Smart and Wearable Materials and Devices

The remaining volumes of the HCI International 2009 proceedings are: • • • • •

Volume 1, LNCS 5610, Human–Computer Interaction––New Trends (Part I), edited by Julie A. Jacko Volume 2, LNCS 5611, Human–Computer Interaction––Novel Interaction Methods and Techniques (Part II), edited by Julie A. Jacko Volume 4, LNCS 5613, Human–Computer Interaction––Interacting in Various Application Domains (Part IV), edited by Julie A. Jacko Volume 5, LNCS 5614, Universal Access in Human–Computer Interaction––Addressing Diversity (Part I), edited by Constantine Stephanidis Volume 6, LNCS 5615, Universal Access in Human–Computer Interaction––Intelligent and Ubiquitous Interaction Environments (Part II), edited by Constantine Stephanidis

VI

Foreword

• • • • • • • • • • •

Volume 7, LNCS 5616, Universal Access in Human–Computer Interaction––Applications and Services (Part III), edited by Constantine Stephanidis Volume 8, LNCS 5617, Human Interface and the Management of Information––Designing Information Environments (Part I), edited by Michael J. Smith and Gavriel Salvendy Volume 9, LNCS 5618, Human Interface and the Management of Information––Information and Interaction (Part II), edited by Gavriel Salvendy and Michael J. Smith Volume 10, LNCS 5619, Human Centered Design, edited by Masaaki Kurosu Volume 11, LNCS 5620, Digital Human Modeling, edited by Vincent G. Duffy Volume 12, LNCS 5621, Online Communities and Social Computing, edited by A. Ant Ozok and Panayiotis Zaphiris Volume 13, LNCS 5622, Virtual and Mixed Reality, edited by Randall Shumaker Volume 14, LNCS 5623, Internationalization, Design and Global Development, edited by Nuray Aykin Volume 15, LNCS 5624, Ergonomics and Health Aspects of Work with Computers, edited by Ben-Tzion Karsh Volume 16, LNAI 5638, The Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience, edited by Dylan Schmorrow, Ivy Estabrooke and Marc Grootjen Volume 17, LNAI 5639, Engineering Psychology and Cognitive Ergonomics, edited by Don Harris

I would like to thank the Program Chairs and the members of the Program Boards of all thematic areas, listed below, for their contribution to the highest scientific quality and the overall success of HCI International 2009.

Ergonomics and Health Aspects of Work with Computers Program Chair: Ben-Tzion Karsh Arne Aarås, Norway Pascale Carayon, USA Barbara G.F. Cohen, USA Wolfgang Friesdorf, Germany John Gosbee, USA Martin Helander, Singapore Ed Israelski, USA Waldemar Karwowski, USA Peter Kern, Germany Danuta Koradecka, Poland Kari Lindström, Finland

Holger Luczak, Germany Aura C. Matias, Philippines Kyung (Ken) Park, Korea Michelle M. Robertson, USA Michelle L. Rogers, USA Steven L. Sauter, USA Dominique L. Scapin, France Naomi Swanson, USA Peter Vink, The Netherlands John Wilson, UK Teresa Zayas-Cabán, USA

Foreword

Human Interface and the Management of Information Program Chair: Michael J. Smith Gunilla Bradley, Sweden Hans-Jörg Bullinger, Germany Alan Chan, Hong Kong Klaus-Peter Fähnrich, Germany Michitaka Hirose, Japan Jhilmil Jain, USA Yasufumi Kume, Japan Mark Lehto, USA Fiona Fui-Hoon Nah, USA Shogo Nishida, Japan Robert Proctor, USA Youngho Rhee, Korea

Anxo Cereijo Roibás, UK Katsunori Shimohara, Japan Dieter Spath, Germany Tsutomu Tabe, Japan Alvaro D. Taveira, USA Kim-Phuong L. Vu, USA Tomio Watanabe, Japan Sakae Yamamoto, Japan Hidekazu Yoshikawa, Japan Li Zheng, P.R. China Bernhard Zimolong, Germany

Human–Computer Interaction Program Chair: Julie A. Jacko Sebastiano Bagnara, Italy Sherry Y. Chen, UK Marvin J. Dainoff, USA Jianming Dong, USA John Eklund, Australia Xiaowen Fang, USA Ayse Gurses, USA Vicki L. Hanson, UK Sheue-Ling Hwang, Taiwan Wonil Hwang, Korea Yong Gu Ji, Korea Steven Landry, USA

Gitte Lindgaard, Canada Chen Ling, USA Yan Liu, USA Chang S. Nam, USA Celestine A. Ntuen, USA Philippe Palanque, France P.L. Patrick Rau, P.R. China Ling Rothrock, USA Guangfeng Song, USA Steffen Staab, Germany Wan Chul Yoon, Korea Wenli Zhu, P.R. China

Engineering Psychology and Cognitive Ergonomics Program Chair: Don Harris Guy A. Boy, USA John Huddlestone, UK Kenji Itoh, Japan Hung-Sying Jing, Taiwan Ron Laughery, USA Wen-Chin Li, Taiwan James T. Luxhøj, USA

Nicolas Marmaras, Greece Sundaram Narayanan, USA Mark A. Neerincx, The Netherlands Jan M. Noyes, UK Kjell Ohlsson, Sweden Axel Schulte, Germany Sarah C. Sharples, UK

VII

VIII

Foreword

Neville A. Stanton, UK Xianghong Sun, P.R. China Andrew Thatcher, South Africa

Matthew J.W. Thomas, Australia Mark Young, UK

Universal Access in Human–Computer Interaction Program Chair: Constantine Stephanidis Julio Abascal, Spain Ray Adams, UK Elisabeth André, Germany Margherita Antona, Greece Chieko Asakawa, Japan Christian Bühler, Germany Noelle Carbonell, France Jerzy Charytonowicz, Poland Pier Luigi Emiliani, Italy Michael Fairhurst, UK Dimitris Grammenos, Greece Andreas Holzinger, Austria Arthur I. Karshmer, USA Simeon Keates, Denmark Georgios Kouroupetroglou, Greece Sri Kurniawan, USA

Patrick M. Langdon, UK Seongil Lee, Korea Zhengjie Liu, P.R. China Klaus Miesenberger, Austria Helen Petrie, UK Michael Pieper, Germany Anthony Savidis, Greece Andrew Sears, USA Christian Stary, Austria Hirotada Ueda, Japan Jean Vanderdonckt, Belgium Gregg C. Vanderheiden, USA Gerhard Weber, Germany Harald Weber, Germany Toshiki Yamaoka, Japan Panayiotis Zaphiris, UK

Virtual and Mixed Reality Program Chair: Randall Shumaker Pat Banerjee, USA Mark Billinghurst, New Zealand Charles E. Hughes, USA David Kaber, USA Hirokazu Kato, Japan Robert S. Kennedy, USA Young J. Kim, Korea Ben Lawson, USA

Gordon M. Mair, UK Miguel A. Otaduy, Switzerland David Pratt, UK Albert “Skip” Rizzo, USA Lawrence Rosenblum, USA Dieter Schmalstieg, Austria Dylan Schmorrow, USA Mark Wiederhold, USA

Internationalization, Design and Global Development Program Chair: Nuray Aykin Michael L. Best, USA Ram Bishu, USA Alan Chan, Hong Kong Andy M. Dearden, UK

Susan M. Dray, USA Vanessa Evers, The Netherlands Paul Fu, USA Emilie Gould, USA

Foreword

Sung H. Han, Korea Veikko Ikonen, Finland Esin Kiris, USA Masaaki Kurosu, Japan Apala Lahiri Chavan, USA James R. Lewis, USA Ann Light, UK James J.W. Lin, USA Rungtai Lin, Taiwan Zhengjie Liu, P.R. China Aaron Marcus, USA Allen E. Milewski, USA

Elizabeth D. Mynatt, USA Oguzhan Ozcan, Turkey Girish Prabhu, India Kerstin Röse, Germany Eunice Ratna Sari, Indonesia Supriya Singh, Australia Christian Sturm, Spain Adi Tedjasaputra, Singapore Kentaro Toyama, India Alvin W. Yeo, Malaysia Chen Zhao, P.R. China Wei Zhou, P.R. China

Online Communities and Social Computing Program Chairs: A. Ant Ozok, Panayiotis Zaphiris Chadia N. Abras, USA Chee Siang Ang, UK Amy Bruckman, USA Peter Day, UK Fiorella De Cindio, Italy Michael Gurstein, Canada Tom Horan, USA Anita Komlodi, USA Piet A.M. Kommers, The Netherlands Jonathan Lazar, USA Stefanie Lindstaedt, Austria

Gabriele Meiselwitz, USA Hideyuki Nakanishi, Japan Anthony F. Norcio, USA Jennifer Preece, USA Elaine M. Raybourn, USA Douglas Schuler, USA Gilson Schwartz, Brazil Sergei Stafeev, Russia Charalambos Vrasidas, Cyprus Cheng-Yen Wang, Taiwan

Augmented Cognition Program Chair: Dylan D. Schmorrow Andy Bellenkes, USA Andrew Belyavin, UK Joseph Cohn, USA Martha E. Crosby, USA Tjerk de Greef, The Netherlands Blair Dickson, UK Traci Downs, USA Julie Drexler, USA Ivy Estabrooke, USA Cali Fidopiastis, USA Chris Forsythe, USA Wai Tat Fu, USA Henry Girolamo, USA

Marc Grootjen, The Netherlands Taro Kanno, Japan Wilhelm E. Kincses, Germany David Kobus, USA Santosh Mathan, USA Rob Matthews, Australia Dennis McBride, USA Robert McCann, USA Jeff Morrison, USA Eric Muth, USA Mark A. Neerincx, The Netherlands Denise Nicholson, USA Glenn Osga, USA

IX

X

Foreword

Dennis Proffitt, USA Leah Reeves, USA Mike Russo, USA Kay Stanney, USA Roy Stripling, USA Mike Swetnam, USA Rob Taylor, UK

Maria L.Thomas, USA Peter-Paul van Maanen, The Netherlands Karl van Orden, USA Roman Vilimek, Germany Glenn Wilson, USA Thorsten Zander, Germany

Digital Human Modeling Program Chair: Vincent G. Duffy Karim Abdel-Malek, USA Thomas J. Armstrong, USA Norm Badler, USA Kathryn Cormican, Ireland Afzal Godil, USA Ravindra Goonetilleke, Hong Kong Anand Gramopadhye, USA Sung H. Han, Korea Lars Hanson, Sweden Pheng Ann Heng, Hong Kong Tianzi Jiang, P.R. China

Kang Li, USA Zhizhong Li, P.R. China Timo J. Määttä, Finland Woojin Park, USA Matthew Parkinson, USA Jim Potvin, Canada Rajesh Subramanian, USA Xuguang Wang, France John F. Wiechel, USA Jingzhou (James) Yang, USA Xiu-gan Yuan, P.R. China

Human Centered Design Program Chair: Masaaki Kurosu Gerhard Fischer, USA Tom Gross, Germany Naotake Hirasawa, Japan Yasuhiro Horibe, Japan Minna Isomursu, Finland Mitsuhiko Karashima, Japan Tadashi Kobayashi, Japan

Kun-Pyo Lee, Korea Loïc Martínez-Normand, Spain Dominique L. Scapin, France Haruhiko Urokohara, Japan Gerrit C. van der Veer, The Netherlands Kazuhiko Yamazaki, Japan

In addition to the members of the Program Boards above, I also wish to thank the following volunteer external reviewers: Gavin Lew from the USA, Daniel Su from the UK, and Ilia Adami, Ioannis Basdekis, Yannis Georgalis, Panagiotis Karampelas, Iosif Klironomos, Alexandros Mourouzis, and Stavroula Ntoa from Greece. This conference could not have been possible without the continuous support and advice of the Conference Scientific Advisor, Prof. Gavriel Salvendy, as well as the dedicated work and outstanding efforts of the Communications Chair and Editor of HCI International News, Abbas Moallem.

Foreword

XI

I would also like to thank for their contribution toward the organization of the HCI International 2009 conference the members of the Human–Computer Interaction Laboratory of ICS-FORTH, and in particular Margherita Antona, George Paparoulis, Maria Pitsoulaki, Stavroula Ntoa, and Maria Bouhli. Constantine Stephanidis

HCI International 2011

The 14th International Conference on Human–Computer Interaction, HCI International 2011, will be held jointly with the affiliated conferences in the summer of 2011. It will cover a broad spectrum of themes related to human–computer interaction, including theoretical issues, methods, tools, processes and case studies in HCI design, as well as novel interaction techniques, interfaces and applications. The proceedings will be published by Springer. More information about the topics, as well as the venue and dates of the conference, will be announced through the HCI International Conference series website: http://www.hci-international.org/

General Chair Professor Constantine Stephanidis University of Crete and ICS-FORTH Heraklion, Crete, Greece Email: [email protected]

Table of Contents

Part I: Mobile Interaction BigKey: A Virtual Keyboard for Mobile Devices . . . . . . . . . . . . . . . . . . . . . Khaldoun Al Faraj, Mustapha Mojahid, and Nadine Vigouroux

3

TringIt: Easy Triggering of Web Actions from a Phone . . . . . . . . . . . . . . . . Vinod Anupam

11

Context Awareness and Perceived Interactivity in Multimedia Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xiao Dong and Pei-Luen Patrick Rau

21

Human Computer Interaction with a PIM Application: Merging Activity, Location and Social Setting into Context . . . . . . . . . . . . . . . . . . . Tor-Morten Grønli and Gheorghita Ghinea

30

CLURD: A New Character-Inputting System Using One 5-Way Key Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hyunjin Ji and Taeyong Kim

39

Menu Design in Cell Phones: Use of 3D Menus . . . . . . . . . . . . . . . . . . . . . . Kyungdoh Kim, Robert W. Proctor, and Gavriel Salvendy

48

Mobile Interfaces in Tangible Mnemonics Interaction . . . . . . . . . . . . . . . . . Thorsten Mahler, Marc Hermann, and Michael Weber

58

Understanding the Relationship between Requirements and Context Elements in Mobile Collaboration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sergio Ochoa, Rosa Alarcon, and Luis Guerrero Continuous User Interfaces for Seamless Task Migration . . . . . . . . . . . . . . Pardha S. Pyla, Manas Tungare, Jerome Holman, and Manuel A. P´erez-Qui˜ nones A Study of Information Retrieval of En Route Display of Fire Information on PDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Weina Qu, Xianghong Sun, Thomas Plocher, and Li Wang A Mobile and Desktop Application for Enhancing Group Awareness in Knowledge Work Teams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Timo Saari, Kari Kallinen, Mikko Salminen, Niklas Ravaja, and Marco Rapino A Study of Fire Information Detection on PDA Device . . . . . . . . . . . . . . . Xianghong Sun, Weina Qu, Thomas Plocher, and Li Wang

67 77

86

95

105

XVI

Table of Contents

Empirical Comparison of Task Completion Time between Mobile Phone Models with Matched Interaction Sequences . . . . . . . . . . . . . . . . . . . Shunsuke Suzuki, Yusuke Nakao, Toshiyuki Asahi, Victoria Bellotti, Nick Yee, and Shin’ichi Fukuzumi

114

Part II: In-Vehicle Interaction and Environment Navigation Nine Assistant Guiding Methods in Subway Design – A Research of Shanghai Subway Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Linong Dai

125

Pull and Push: Proximity-Aware User Interface for Navigating in 3D Space Using a Handheld Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mingming Fan and Yuanchun Shi

133

A Study on the Design of Voice Navigation of Car Navigation System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chih-Fu Wu, Wan-Fu Huang, and Tung-Chen Wu

141

Front Environment Recognition of Personal Vehicle Using the Image Sensor and Acceleration Sensors for Everyday Computing . . . . . . . . . . . . . Takahiro Matsui, Takeshi Imanaka, and Yasuyuki Kono

151

Common Interaction Schemes for In-Vehicle User-Interfaces . . . . . . . . . . . Simon Nestler, Marcus T¨ onnis, and Gudrun Klinker

159

Dynamic Maps for Future Navigation Systems: Agile Design Exploration of User Interface Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Volker Paelke and Karsten Nebe

169

Flight Searching – A Comparison of Two User-Interface Design Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Antti Pirhonen and Niko Kotilainen

179

Agent-Based Driver Abnormality Estimation . . . . . . . . . . . . . . . . . . . . . . . . Tony Poitschke, Florian Laquai, and Gerhard Rigoll Enhancing the Accessibility of Maps with Personal Frames of Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Falko Schmid

189

199

Augmented Interaction and Visualization in the Automotive Domain . . . Roland Spies, Markus Ablaßmeier, Heiner Bubb, and Werner Hamberger

211

Proposal of a Direction Guidance System for Evacuation . . . . . . . . . . . . . . Chikamune Wada, Yu Yoneda, and Yukinobu Sugimura

221

Table of Contents

A Virtual Environment for Learning Aiport Emergency Management Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Telmo Zarraonandia, Mario Rafael Ruiz Vargas, Paloma D´ıaz, and Ignacio Aedo

XVII

228

Part III: Agents, Avatars and Personalisation User Profiling for Web Search Based on Biological Fluctuation . . . . . . . . . Yuki Arase, Takahiro Hara, and Shojiro Nishio Expression of Personality through Avatars: Analysis of Effects of Gender and Race on Perceptions of Personality . . . . . . . . . . . . . . . . . . . . . . Jennifer Cloud-Buckner, Michael Sellick, Bhanuteja Sainathuni, Betty Yang, and Jennie Gallimore User-Definable Rule Description Framework for Autonomous Actor Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Narichika Hamaguichi, Hiroyuki Kaneko, Mamoru Doke, and Seiki Inoue Cognitive and Emotional Characteristics of Communication in Human-Human/Human-Agent Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . Yugo Hayashi and Kazuhisa Miwa

239

248

257

267

Identification of the User by Analyzing Human Computer Interaction . . . R¨ udiger Heimg¨ artner

275

The Anticipation of Human Behavior Using “Parasitic Humanoid” . . . . . Hiroyuki Iizuka, Hideyuki Ando, and Taro Maeda

284

Modeling Personal Preferences on Commodities by Behavior Log Analysis with Ubiquitous Sensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Naoki Imamura, Akihiro Ogino, and Toshikazu Kato A System to Construct an Interest Model of User Based on Information in Browsed Web Page by User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kosuke Kawazu, Masakazu Murao, Takeru Ohta, Masayoshi Mase, and Takashi Maeno

294

304

Adaptive User Interfaces for the Clothing Retail . . . . . . . . . . . . . . . . . . . . . Karim Khakzar, Jonas George, and Rainer Blum

314

Implementing Affect Parameters in Personalized Web-Based Design . . . . Zacharias Lekkas, Nikos Tsianos, Panagiotis Germanakos, Constantinos Mourlas, and George Samaras

320

XVIII

Table of Contents

Modeling of User Interest Based on Its Interaction with a Collaborative Knowledge Management System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jaime Moreno-Llorena, Xavier Alam´ an Rold´ an, and Ruth Cobos Perez

330

Some Pitfalls for Developing Enculturated Conversational Agents . . . . . . Matthias Rehm, Elisabeth Andr´e, and Yukiko Nakano

340

Comparison of Different Talking Heads in Non-Interactive Settings . . . . . Benjamin Weiss, Christine K¨ uhnel, Ina Wechsung, Sebastian M¨ oller, and Sascha Fagel

349

Video Content Production Support System with Speech-Driven Embodied Entrainment Character by Speech and Hand Motion Inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Michiya Yamamoto, Kouzi Osaki, and Tomio Watanabe Autonomous Turn-Taking Agent System Based on Behavior Model . . . . . Masahide Yuasa, Hiroko Tokunaga, and Naoki Mukawa

358

368

Part IV: Ambient Interaction An Interoperable Concept for Controlling Smart Homes – The ASK-IT Paradigm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Evangelos Bekiaris, Kostas Kalogirou, Alexandros Mourouzis, and Mary Panou Towards Ambient Augmented Reality with Tangible Interfaces . . . . . . . . . Mark Billinghurst, Rapha¨el Grasset, Hartmut Seichter, and Andreas D¨ unser Rapid Prototyping of an AmI-Augmented Office Environment Demonstrator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dimitris Grammenos, Yannis Georgalis, Nikolaos Partarakis, Xenophon Zabulis, Thomas Sarmis, Sokratis Kartakis, Panagiotis Tourlakis, Antonis Argyros, and Constantine Stephanidis Challenges for User Centered Smart Environments . . . . . . . . . . . . . . . . . . . Fabian Hermann, Roland Blach, Doris Janssen, Thorsten Klein, Andreas Schuller, and Dieter Spath Point and Control: The Intuitive Method to Control Multi-device with Single Remote Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sung Soo Hong and Ju Il Eom New Integrated Framework for Video Based Moving Object Tracking . . . Md. Zahidul Islam, Chi-Min Oh, and Chil-Woo Lee

377

387

397

407

416

423

Table of Contents

XIX

Object Scanning Using a Sensor Frame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Soonmook Jeong, Taehoun Song, Gihoon Go, Keyho Kwon, and Jaewook Jeon

433

Mixed Realities – Virtual Object Lessons . . . . . . . . . . . . . . . . . . . . . . . . . . . Andreas Kratky

440

New Human-Computer Interactions Using Tangible Objects: Application on a Digital Tabletop with RFID Technology . . . . . . . . . . . . . S´ebastien Kubicki, Sophie Lepreux, Yoann Lebrun, Philippe Dos Santos, Christophe Kolski, and Jean Caelen

446

Context-Aware Cognitive Agent Architecture for Ambient User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Youngho Lee, Choonsung Shin, and Woontack Woo

456

An Embodied Approach for Engaged Interaction in Ubiquitous Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mark O. Millard and Firat Soylu

464

Generic Framework for Transforming Everyday Objects into Interactive Surfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Elena Mugellini, Omar Abou Khaled, St´ephane Pierroz, Stefano Carrino, and Houda Chabbi Drissi mæve – An Interactive Tabletop Installation for Exploring Background Information in Exhibitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Till Nagel, Larissa Pschetz, Moritz Stefaner, Matina Halkia, and Boris M¨ uller

473

483

Relationality Design toward Enriched Communications . . . . . . . . . . . . . . . Yukiko Nakano, Masao Morizane, Ivan Tanev, and Katsunori Shimohara

492

Ultra Compact Laser Based Projectors and Imagers . . . . . . . . . . . . . . . . . . Harald Schenk, Thilo Sandner, Christian Drabe, Michael Scholles, Klaus Frommhagen, Christian Gerwig, and Hubert Lakner

501

Understanding the Older User of Ambient Technologies . . . . . . . . . . . . . . . Andrew Sixsmith

511

Multi-pointing Method Using a Desk Lamp and Single Camera for Effective Human-Computer Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Taehoun Song, Thien Cong Pham, Soonmook Jung, Jihwan Park, Keyho Kwon, and Jaewook Jeon Communication Grill/Salon: Hybrid Physical/Digital Artifacts for Stimulating Spontaneous Real World Communication . . . . . . . . . . . . . . . . . Koh Sueda, Koji Ishii, Takashi Miyaki, and Jun Rekimoto

520

526

XX

Table of Contents

Motion Capture System Using an Optical Resolver . . . . . . . . . . . . . . . . . . . Takuji Tokiwa, Masashi Yoshidzumi, Hideaki Nii, Maki Sugimoto, and Masahiko Inami

536

The Effects of an Anti-glare Sleeve Installed on Fluorescent Tube Lamps on Glare and Reading Comfort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shiaw-Tsyr Uang, Cheng-Li Liu, and Mali Chang

544

Electromyography Focused on Passiveness and Activeness in Embodied Interaction: Toward a Novel Interface for Co-creating Expressive Body Movement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Takabumi Watanabe, Norikazu Matsushima, Ryutaro Seto, Hiroko Nishi, and Yoshiyuki Miwa

554

Part V: Affect, Emotion and Engagement An Integrated Approach to Emotion Recognition for Advanced Emotional Intelligence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Panagiotis D. Bamidis, Christos A. Frantzidis, Evdokimos I. Konstantinidis, Andrej Luneski, Chrysa Lithari, Manousos A. Klados, Charalambos Bratsas, Christos L. Papadelis, and Costas Pappas Addressing the Interplay of Culture and Affect in HCI: An Ontological Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Emmanuel G. Blanchard, Riichiro Mizoguchi, and Susanne P. Lajoie Love at First Encounter – Start-Up of New Applications . . . . . . . . . . . . . . Henning Breuer, Marlene Kettner, Matthias Wagler, Nathalie Preuschen, and Fee Steinhoff

565

575

585

Responding to Learners’ Cognitive-Affective States with Supportive and Shakeup Dialogues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sidney D‘Mello, Scotty Craig, Karl Fike, and Arthur Graesser

595

Trust in Online Technology: Towards Practical Guidelines Based on Experimentally Verified Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christian Detweiler and Joost Broekens

605

Influence of User Experience on Affectiveness . . . . . . . . . . . . . . . . . . . . . . . . Ryoko Fukuda

615

A Human-Centered Model for Detecting Technology Engagement . . . . . . James Glasnapp and Oliver Brdiczka

621

Relationship Learning Software: Design and Assessment . . . . . . . . . . . . . . Kyla A. McMullen and Gregory H. Wakefield

631

Table of Contents

XXI

Relationship Enhancer: Interactive Recipe in Kitchen Island . . . . . . . . . . . Tsai-Yun Mou, Tay-Sheng Jeng, and Chun-Heng Ho

641

ConvoCons: Encouraging Affinity on Multitouch Interfaces . . . . . . . . . . . . Michael A. Oren and Stephen B. Gilbert

651

Development of an Emotional Interface for Sustainable Water Consumption in the Home . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mehdi Ravandi, Jon Mok, and Mark Chignell Influences of Telops on Television Audiences’ Interpretation . . . . . . . . . . . Hidetsugu Suto, Hiroshi Kawakami, and Osamu Katai

660 670

Extracting High-Order Aesthetic and Affective Components from Composer’s Writings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Akifumi Tokosumi and Hajime Murai

679

Affective Technology, Affective Management, towards Affective Society . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hiroyuki Umemuro

683

Bio-sensing for Emotional Characterization without Word Labels . . . . . . Tessa Verhoef, Christine Lisetti, Armando Barreto, Francisco Ortega, Tijn van der Zant, and Fokie Cnossen An Affect-Sensitive Social Interaction Paradigm Utilizing Virtual Reality Environments for Autism Intervention . . . . . . . . . . . . . . . . . . . . . . . Karla Conn Welch, Uttama Lahiri, Changchun Liu, Rebecca Weller, Nilanjan Sarkar, and Zachary Warren Recognizing and Responding to Student Affect . . . . . . . . . . . . . . . . . . . . . . Beverly Woolf, Toby Dragon, Ivon Arroyo, David Cooper, Winslow Burleson, and Kasia Muldner

693

703

713

Part 6: Smart and Wearable Materials and Devices Usability Studies on Sensor Smart Clothing . . . . . . . . . . . . . . . . . . . . . . . . . Haeng Suk Chae, Woon Jung Cho, Soo Hyun Kim, and Kwang Hee Han Considering Personal Profiles for Comfortable and Efficient Interactions with Smart Clothes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . S´ebastien Duval, Christian Hoareau, and Gilsoo Cho Interaction Wearable Computer with Networked Virtual Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jiung-yao Huang, Ming-Chih Tung, Huan-Chao Keh, Ji-jen Wu, Kun-Hang Lee, and Chung-Hsien Tsai

725

731

741

XXII

Table of Contents

The Impact of Different Visual Feedback Presentation Methods in a Wearable Computing Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hendrik Iben, Hendrik Witt, and Ernesto Morales Kluge

752

Gold Coating of a Plastic Optical Fiber Based on PMMA . . . . . . . . . . . . . Seok Min Kim, Sung Hun Kim, Eun Ju Park, Dong Lyun Cho, and Moo Sung Lee

760

Standardization for Smart Clothing Technology . . . . . . . . . . . . . . . . . . . . . . Kwangil Lee and Yong Gu Ji

768

Wearable ECG Monitoring System Using Conductive Fabrics and Active Electrodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Su Ho Lee, Seok Myung Jung, Chung Ki Lee, Kee Sam Jeong, Gilsoo Cho, and Sun K. Yoo Establishing a Measurement System for Human Motions Using a Textile-Based Motion Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Moonsoo Sung, Keesam Jeong, and Gilsoo Cho

778

784

A Context-Aware AR Navigation System Using Wearable Sensors . . . . . . Daisuke Takada, Takefumi Ogawa, Kiyoshi Kiyokawa, and Haruo Takemura

793

Emotional Smart Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Akira Wakita, Midori Shibutani, and Kohei Tsuji

802

Novel Stretchable Textile-Based Transmission Bands: Electrical Performance and Appearance after Abrasion/Laundering, and Wearability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yoonjung Yang and Gilsoo Cho Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

806

815

BigKey: A Virtual Keyboard for Mobile Devices Khaldoun Al Faraj, Mustapha Mojahid, and Nadine Vigouroux University of Toulouse, IRIT, 31062 Toulouse, France {alfaraj,mojahid,vigourou}@irit.fr

Abstract. This paper describes BigKey virtual keyboard for mobile devices, designed to make the keys of virtual keyboard easier to acquire. The tiny size of keys makes efficient selection difficult. To overcome this drawback, we propose to expand key size that corresponds to next character entry. The proposed solution helps to facilitate the selection task by expanding the next entry. Moreover the prediction system reduces the visual scanning time to find letters that one is looking for. Users’ performance study showed that they were 25.14% faster and more accurate with BigKey virtual keyboard than with normal virtual keyboard. Keywords: Virtual keyboard, text input, PDAs, expanding targets, letter prediction.

1 Introduction An efficient text entry method for mobile devices is becoming one of the most prominent challenges in the world of mobile computing. The shrinking size of handheld devices has resulted in a keyboard that is not as convenient as that of desktop computer. Personnel Digital Assistants (PDAs) and smart phones equipped with a touch screen and a stylus, in general, tend to have alternative text input techniques like handwriting recognition technology and virtual keyboard on the screen. Handwriting recognition systems largely help to overcome the screen-space constraints of mobile computing products. However, user learning of making character strokes is required and that is not always easy, especially for novice users. Virtual keyboard (sometimes called on-screen keyboard) is a reproduction of a hardware keyboard on the screen of computing devices. It was originally designed for people with disabilities to access to computers and for some special needs as well. Also the mobile devices equipped with a touch screen and a stylus, have used it as another text entry solution. The virtual keyboard of handheld device has less number of keys (60) as compared to desktop keyboard (105). It was reduced by modes of labeled keys to enter numbers and special characters. However, the accurate selection of smaller keys still remains difficult. It requires a great amount of attention as user also has to focus on what he/she is writing. J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 3–10, 2009. © Springer-Verlag Berlin Heidelberg 2009

4

K. Al Faraj, M. Mojahid, and N. Vigouroux

A combination of gesture and virtual keyboard has been constructed like Quickwriting [1] and Shark [2]. The main goal of this concept is to allow word level entry with keyboard. Although many investigations have been devoted to find the most efficient text entry method for mobile devices by changing character arrangement and key shapes of virtual keyboard [3] [4]. Less attention has been paid to the key size that we consider as an essential element of efficient mobile text inputting. In this paper, we discuss our BigKey virtual keyboard principle that aims to optimize user’s performance. Subsequently, we will present our preliminary results obtained from a formal study of the proposal.

2 BigKey Virtual Keyboard Users of PDAs and smart phone feel difficulty in using the tiny size of keys while using virtual keyboard for small screen. Our primary design objective is to break this main obstacle that limits user’s performance. The principle of BigKey virtual keyboard is the expanding key size corresponding to the next character entry. This system is primarily designed for the mobile devices such as Ultra Mobile PC (UMPC), PDAs, smart phones and so forth, using their virtual keyboard. However this design is applicable to any computing device supporting the same input pattern. McGuffin and Balakrishnan have proposed an interface design of one-dimensional arrays of widgets consisting of button strip [5]. It is based on expanding a target when the pointer approaches it to facilitate its selection. To solve the drawback of a sideways motion resulting from expanding target, some overlapping between adjacent buttons is allowed. This principle is much more similar to that was proposed by Dasher [6]. Cirrin is a continuous stylus-based text entry technique [7]. The letters of English alphabet are arranged on the circumference of a circle by using the common sequences in English to minimize stylus travel. A word is entered by pressing and moving the stylus over the letters. Expanding targets idea explained above has been applied to Cirrin in order to improve user’s performance [8]. The experimental results indicate that Cirrin is more fast but less accurate. This is due to the occlusion and overlapping between neighboring buttons. Another application of expanding targets principle to virtual keyboard with QWERTY layout, called Fisheye, has been proposed [9]. The aim is always to make selection task easier for PDAs. The character is selected by lifting up the stylus when it is over the key. While analyzing the selection model, we noticed that the successive selection of characters on touch-screen with stylus is accomplished by using three-dimensional mode (3D mode). In other words, user has to lift the stylus between every selection of character (sequence of stylus up and stylus down); hence the stylus moves in threedimensional space. Whereas using the mouse as a pointing device makes selection task in two-dimensional mode (2D mode). Note that the pointing device used to evaluate the measure of performance of expanding targets design mentioned earlier was the mouse.

BigKey: A Virtual Keyboard for Mobile Devices

5

Considering the 3D mode, expanding targets when the stylus is close to them is rarely occurred because of the third dimension (lifting up the stylus). In other words, the expansion of target is mostly occurred when the stylus is over the target. In this case, there is no advantage to expand this key as the stylus is exactly over the key and user can select it if it is expanded or not. Moreover, with each animation of Fisheye keyboard, user has to re-compute the target’s position that makes it more difficult to acquire and more attention is required. According to Fitts’ law [10], the time “MT” to acquire a target of width “W” which lies at a distance or amplitude “A” is given by the relationship [11]:

M MT = a + b log 2 (

A + 1) W

(1)

Where a and b are constants determined through linear regression. The logarithmic term is called the index of difficulty (ID) and is measured in “bits”. As character selection is made from key-to-key, according to Fitt’s law, larger the size of the next key to be selected, shorter the time will be required to acquire it. Our BigKey system is based on two processes: first is to predict the next character entry based on the users previous input, second is to expand the corresponding predicted keys. The keys expand as a function of their letter probability of entry: the most probable the next letter, the largest the key is. To build our prediction system, we have employed tables of single letter and diagram frequency Counts proposed by [12]. Four expanded keys are maximum number used in our BigKey implementation as shown in figure 1.

Fig. 1. BigKey virtual keyboard while selecting the next entry for word “the”

The BigKey system offers two main advantages. On the one hand, it facilitates character selection according to Fitt’s law. On the other hand, it reduces the time of keyboard visual scanning to find the letters that one is looking for, especially for novice users.

6

K. Al Faraj, M. Mojahid, and N. Vigouroux

It could also be effective for the people with motor impairments who need to reduce the acquisition target time. Furthermore, this would help immensely persons with Alzheimer’s disease. As a result our design model makes the selection task easier through expanding the targets of next entry when the stylus is over the previous entry. The regular QWERTY keyboard layout is used as for the user’s familiarity with its layout. Moreover it was originally designed to keep commonly used letter combination farther from each other. In this way the overlapping between expanded keys is not occurred. Unlike hardware keyboard where one can type over by using several fingers. While using a virtual keyboard, one has to carry out all movements by using only a pointing device. In our BigKey proposed solution, the expanded keys can be considered as the most probable keys that user always puts his fingers over. Comparing the most probable next letter predictions for each letter with others, an intersection is mostly detected especially for vowels which are typically proposed as predictions. In hardware keyboard, the most accessible keys are those that are under the fingers. Similarly in our Bigkey system, the expanded keys are those that are the most accessible like the initial fingers configuration, even if the user does not type them when the prediction does not give the desired result.

3 Experiment The aim of this study is to verify our hypothesis: the proposed animation of virtual keyboard had significant effects on user performance for text entry task. 3.1 Subjects Nine volunteers (3 female) from our university campus participated in this study. Users average 27.11 years of age (ranging from 24 to 33 years). They were novice stylus-based text input users. All users had normal or corrected eyesight and all were right-handed to use the stylus as a pointing device. 3.2 Apparatus Users conducted the study on a Sony VAIO UMPC with a 1.2GHz Core Solo processor and 512MB of RAM. The display size was 4.5" SVGA TFT and ran at a resolution of 600 × 800. The pointing device used in our experiment was the stylus. The experiment included the three following virtual keyboards: The virtual keyboard without expanded keys (No-BigKey), the BigKey virtual keyboard with one expanded key (One-BigKey), and the BigKey virtual keyboard with four expanded keys (Four-BigKey). Table 1 shows the key size used in our study, the normal key had the same size of that is used for PDA virtual keyboard. Besides the key of the first most probable letter had the same size of that is used for the virtual keyboard of desktop computer. For each virtual keyboard, the program reads a series of 10 phrases ranging from 16 to 43 characters [13]. All virtual keyboards are built in .NET C#.

BigKey: A Virtual Keyboard for Mobile Devices

7

Table 1. The key size of BigKey virtual keyboard

Key

Size

Normal

18 × 16 pixels

The first most probable letter

26 × 24 pixels (+ 44.44 %)

The second most probable letter

24 × 22 pixels (+ 33.33 %)

The third most probable letter

22 × 20 pixels (+ 22.22 %)

The fourth most probable letter

20 × 18 pixels (+ 11.11 %)

3.3 Procedure The experiment consisted of two parts: training session followed by testing session. The first session consisted of entering the sentence “the quick brown fox jumps over the lazy dog” by using Four-BigKey virtual keyboard. Then, in the testing session each participant completed three sentence tasks using three virtual keyboards. Participants were divided into three-person groups to perform tasks in a different order. The first performed the experiment in following order: No-BigKey, OneBigKey and then Four-BigKey. While the second followed the order: One-BigKey, Four-BigKey and then No-BigKey. The succession of tasks for the third group was as following: Four-BigKey, One-BigKey and then No-BigKey. In this way, the task order had no impact on the results. The same phrases were used for all tasks but were in a different order for each one so that user cannot anticipate the phrase with the other tasks. Participants were instructed to enter the phrases “as quickly and accurately as possible”. They could make errors and corrections.

4 Results and Discussion In order to evaluate the efficiency of text entry technique, two essential metrics are available up till now. The text entry speed expressed in words per minute (wpm) or in characters per second (cps), and the accuracy during and after the text entry task. 4.1 Text Entry Speed The analysis of entry speed yields a significant result in favor of Four-BigKey. Comparing three virtual keyboards, the No-BigKey speed was the slowest 20.84 wpm, the OneBigKey was faster 23.66 wpm, and the Four-BigKey was the fastest 26.08 wpm, as shown in figure 2. The average improvement of speed is 25.14 % with Four-BigKey.

8

K. Al Faraj, M. Mojahid, and N. Vigouroux

This study shows that the fastest text entry speed was achieved with increasing the number of expanded keys. As letter prediction does not always give the intended result, increasing the number of expanded keys over one, was necessary. The question that remains still to be answered: What is the optimal number of expanded keys for each letter?

Fig. 2. Text entry speed for three virtual keyboards

4.2 Accuracy In our study, participants were allowed to enter phrases naturally, thus they may commit errors and make corrections. We measured errors made during text entry and the errors left in transcribed string using Corrected Error Rate and Not Corrected Error Rate metrics respectively [14]. Figures 3&4 show that participants made more corrections than leaving errors for presented text in all tasks. However, error rates were not significantly different between all tasks.

Fig. 3. Corrected error rate for three virtual keyboards

BigKey: A Virtual Keyboard for Mobile Devices

9

Fig. 4. Not corrected error rate for three virtual keyboards

Comparing the three virtual keyboards, the analysis yields the least error rate in favor of the Four-BigKey (see figures 3&4). These results suggest that keys were easier to acquire when expanding them through Four-BigKey.

5 Conclusion and Future Work We have shown that the expanding targets based on letter prediction can be an effective means of making targets easier to acquire for the virtual keyboard of handheld devices. This design offered 25.14% better speed during text entry for the BigKey virtual keyboard over the normal virtual keyboard and higher accuracy at the same time. On the basis of these preliminary results, we are conducting detailed experimentation to explore the optimal number and size of predicted keys. In the future, we are planning to study the efficiency of the BigKey system for people with motor impairment who need to reduce the fatigue of target acquisition. We also intend to explore the impact of expanding the next entry on the recall of the word completion for people with Alzheimer’s disease.

References 1. Perlin, K.: Quikwriting: Continuous Stylus-Based Text Entry. In: UIST 1998, pp. 251– 316. ACM Press, San Francisco (1998) 2. Zhai, S., Kristensson, P.O.: Shorthand Writing on Stylus Keyboard. In: CHI 2003, pp. 97– 104. ACM Press, Ft. Lauderdale (2003) 3. MacKenzie, I.S., Soukoreff, R.W.: Text Entry for Mobile Computing: Models and Methods Theory and Practice. In: Human-Computer Interaction, vol. 17(2), pp. 147–198. Lawrence Erlbaum, Mahwah (2002) 4. Zhai, S., Hunter, M., Smith, B.A.: The Metropolis keyboard: An exploration of quantitative for graphical keyboard design. In: UIST 2000, pp. 119–128. ACM Press, San Diego (2000)

10

K. Al Faraj, M. Mojahid, and N. Vigouroux

5. McGffin, M., Balakrishnan, R.: Acquisition of expanding targets. In: CHI 2002, pp. 57– 64. ACM Press, Minneapolis (2002) 6. Ward, D.J., Blackwell, A.F., Mackay, D.J.C.: Dasher: A Data Entry Interface Using Continuous Gestures and Language Models. In: UIST 2000, pp. 129–137. ACM Press, San Diego (2000) 7. Mankoff, J., Abowd, G.D.: Cirrin: A Word-Level Unistroke Keyboard for Pen Input. In: UIST 1998, pp. 213–214. ACM Press, San Francisco (1998) 8. Cechanowicz, J., Dawson, S., Victor, M., Subramanian, S.: Stylus based text input using expanding CIRRIN. In: Proceedings of the Working Conference on Advanced Visual Interface, AVI 2006, pp. 163–166. ACM Press, New York (2006) 9. Raynal, M., Truillet, P.: Fisheye keyboard: Whole keyboard displayed on PDA. In: Jacko, J.A. (ed.) HCI 2007. LNCS, vol. 4551, pp. 452–459. Springer, Heidelberg (2007) 10. Fitts, P.M.: The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology 47(6), 381–391 (1954) 11. MacKenzie, I.S.: Fitts’ Law as a Research and Design Tool in Human-Computer Interaction. In: Human-Computer Interaction, vol. 7(1), pp. 91–139. Lawrence Erlbaum, Mahwah (1992) 12. Mayzner, M.S., Tresselt, M.E.: Tables of Single-Letter and Diagram Frequency Counts for Various Word-Length and Letter-Position Combinations. Psychonomic Monograph Supplements 1(2), 13–32 (1965) 13. MacKenzie, I.S., Soukoreff, R.W.: Phrase set for evaluating text entry techniques. In: CHI 2003, pp. 754–755. ACM Press, Ft. Lauderdale (2003) 14. Soukoreff, R.W., MacKenzie, I.S.: Metrics for text entry research: An evaluation of MSD and KSPC, and a new unified error metric. In: CHI 2003, pp. 113–120. ACM Press, Ft. Lauderdale (2003)

TringIt: Easy Triggering of Web Actions from a Phone Vinod Anupam Anexas Inc, 67 Shields Ln, Bridgewater NJ 08807, USA [email protected]

Abstract. Much information that is of interest to mobile users is available on the Web, yet is difficult to access for most users. We introduce a novel method for users to interact with network-connected computers using their phones, and describe a system called TringIt that implements the method. TringIt enables users to trigger Web actions by simply dialing specific numbers – an action that we call a Phone Click. TringIt can be used out-of-the-box by any phone user. The Phone Click is most useful from mobile phones that can receive messages in response to the click. TringIt enables users to easily initiate interaction with businesses and content owners by simply dialing numbers discovered in offline media (e.g. print, TV, radio) as well as online media (e.g. Web, SMS, MMS.) It makes every mobile phone a more compelling information, interaction and participation device. Keywords: Phone Click, Tring, Dial-to-Click, Call Triggered Messaging, Userto-Application Signaling, SMS/MMS Click-through, Dial-able hyperlinks.

1 Introduction Dialing phone numbers is the easiest thing that users can do from a mobile phone. Users manually dial numbers, dial them out of address books, speed dial them, voicedial them, and even "click" on phone numbers in messages and Web pages to dial them. Phone-car integration systems (including e.g. Microsoft Sync) allow users to place calls hands-free in their automobiles. So far, users have been able to do one of two things by dialing a number: set up a voice call (to interact via voice with another user or with an Interactive Voice Response system) or set up a dial-up connection (to interact with a remote system via data modulated over the voice channel). In this paper, we describe a new way in which users can interact with networkconnected computers using their phones. We introduce the notion of a “Phone Click” – the triggering of a Web action in response to the dialing of a number. We describe a system called TringIt that enables any mobile phone user to interact with Web applications in this easy-to-use yet powerful way, and discuss how it is used to easily request information. The paper is organized as follows. In Section 2 we discuss several capabilities of mobile phones, some of which are leveraged by the new technique. In Section 3 we describe how phone signaling is used for user-to-application signaling, and describe some applications. Finally, we summarize our conclusions and discuss upcoming work. J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 11–20, 2009. © Springer-Verlag Berlin Heidelberg 2009

12

V. Anupam

2 Background and Motivation The mobile phone is the most widely used connected device. Over 3.9 billion (and growing) mobile phone users worldwide connect to the "network" - including the Public Switched Telephone Network (PSTN) and the Internet - via their mobile phones. Importantly, every mobile phone – even a relatively basic feature-phone – is a very sophisticated, network-connected computer. For most users, however, it is still difficult and/or expensive to request and receive information in their phone – the device is vastly under-utilized, compared to its potential! Most users use their mobile phone just to make voice calls and to send and receive messages. While the number of mobile users who access the Internet from their mobile phones is steadily increasing, adoption is still low in most markets (other than Japan and South Korea, which have seen significant adoption.) In the US, about 16% of subscribers use applications that access the Internet from their mobile phones [9]. Adoption is even lower in other mature markets - e.g., 10% in France, 7% in Germany – and especially in emerging markets - e.g., 2% in India, 7% in China. Many emerging mobile Web/Internet based applications have been unable to reach critical mass because limited adoption of mobile Internet leaves little room for network effects to kick in. Key barriers to adoption include: • Cost - User-incurred cost of data service is a key issue. • Usability - Mobile web browsing poses significant usability challenges. Mobileoptimized Web sites are still more the exception than the rule, and most mobileoptimized applications are not yet widely used. It is reasonable to posit that information solutions that work for all users, are easy to use and also are sensitive to user-incurred cost have a higher likelihood of adoption than those that are not. Mobile phone subscribers typically pay for calls that they place from their mobile phone, for messages that they send from their phone as well as for data connections initiated from their phone. In many parts of the world (e.g. Europe, Africa and Asia), incoming calls and incoming messages are free to the user, while in other parts of the world (e.g. USA) subscribers pay for incoming calls as well as incoming messages. Messaging uptake and usage has historically been high in markets where calls are/were significantly more expensive than messaging. This can be attributed to the subscriber's desire to minimize cost of communication, even at the cost of poor usability (e.g. triple-tapping to create messages.). Let us quickly look at the main communication channels available to users in their mobile phone in terms of their reach, their relative cost and their key deficiencies. 2.1 Phone Calls Voice calls dominate user-generated traffic in mobile phone networks, and continue to be the mainstay of mobile network operator revenue worldwide. Users call to interact with other users and businesses. Phones are optimized for voice calls, and provide capabilities like address books, speed dialing, voice dialing etc. to streamline communication via voice calls.

TringIt: Easy Triggering of Web Actions from a Phone

13

Businesses widely use phone numbers (often toll free numbers, like 1-800 numbers in the US) that users can call to speak with representatives, customer service agents etc. to receive information and to trigger interactions. While most voice calls are person-to-person calls, use of Interactive Voice Response (IVR) [7] systems is the second largest use case. Sophisticated IVR applications are widely used worldwide – reducing the need for expensive human agents. Technologies like VoiceXML [12] allow users to interact with these systems using voice and/or touch-tone input. IVR systems interconnected to the Web provide a powerful channel for information retrieval by all phone users. Universal reach (any phone user can make voice calls) is a key strength of the voice call. Additionally, phone numbers are familiar, and easily disseminated via both offline and online channels. However, there are some key deficiencies of voice as a channel for requesting and receiving information. Voice is the most expensive channel in most markets. Information received via voice does not persist in the phone for subsequent use (the user must either remember what he was told by the other party, or must transcribe it onto another medium - e.g., by writing something down on paper.) IVR systems are somewhat cumbersome to navigate. Finally, voice interfaces are not optimal for use in noisy environments – typical of mobile usage. 2.2 Messaging Messaging is the second most frequently used communication channel from mobile phones. Users transmit and receive billions of messages every day. Messaging is analogous to email – messages created by a sending party are transmitted using a store-and-forward metaphor towards the target party. SMS (Short Message Service) [1] is the most popular form of messaging. Since it is limited to a small payload of 160 7-bit characters, SMS is frequently used to send short messages and status updates. MMS (Multimedia Message Service) [10] is a high-capacity messaging channel that has support for larger messages possibly containing images and/or short video clips. While MMS message size limits are sometimes imposed by operators, typical limits are upwards of 100K bytes. Messaging initially offered an economical communication channel alternative to the much more expensive voice call. The benefit of lower cost outweighed poor usability - the fairly cumbersome process of multi-tapping messages via a phone's numeric keypad. However, messaging has some key strengths. It works in every mobile phone. It is less intrusive than voice calls. It is very easy to receive messages in phones. And much application infrastructure now exists allowing messaging systems to be interfaced to Web systems for a variety of applications. However, messaging has drawback: it is still somewhat cumbersome to create a message by multi-tapping. A typical call to action displayed in a magazine or TV ad, or transmitted via SMS/MMS is of the form "TEXT PIZZA to short code 123456 to receive a coupon.” It takes about 18 button presses to react to that call to action on a typical phone. And the vocabulary for interacting with different application servers varies greatly, imposing a cognitive load. Collectively, these present a usability barrier for users who want to respond to the call to action.

14

V. Anupam

2.3 USSD Unstructured Supplementary Services Data [2] is a communication channel available in GSM networks. Users can manually (or via an in-phone client application) create and send USSD messages, and can receive/review USSD messages via a notifier application. Unlike SMS, which uses transaction-based store-and-forward communication, USSD uses session based point-to-point communication and is therefore much faster. The key strengths of USSD are its wide deployment on GSM networks (every GSM phone is capable of sending and receiving USSD messages) and its low cost. The key drawbacks there are no standard vocabularies for interacting with application servers to request information, so messages are cumbersome to create. Additionally, USSD application server infrastructure is not as widely deployed as SMS application server infrastructure. And USSD does not work in CDMA networks. 2.4 Mobile Internet/Web Internet access from mobile phones is a powerful communication channel. Modern mobile networks provide high bandwidth and low latency Internet access. Most modern mobile phones have a mobile Web browser built in, and mobile-ready Web sites can provide a compelling interactive user experience. However, the mobile phone is a very constrained device, with a small screen and limited input capabilities. The browser in a typical mobile phone is significantly less sophisticated than a PC browser (but this is changing with high end phones like the Apple iPhone). Most Web sites are not mobile ready, leading to a very poor user experience when accessed from a mobile phone. These factors collectively make it a low-reach information channel - most mobile phone users do not use the mobile Web. To address usability concerns in mobile phones, a variety of in-phone client applications are being built and deployed. Clients that let users interact with email as well as instant messaging (IM) systems are available in many phones, and have seen some uptake. However, application infrastructure that enables interfacing these channels with Web servers is limited. Emerging mobile widget solutions (like Yahoo Go! [13]) provide the user a mobile-optimized information browsing experience, but have seen limited uptake. Such solutions, however, do provide infrastructure for interfacing with Web applications.

3 TringIt: Dial to Click Dialing a number is the easiest thing that a user can do from a mobile phone. And every mobile phone can receive messages. We exploit these attributes to create a solution that is both easy to use and usable by the entire mobile phone user population. The key idea is to use telephone signaling for application signaling, and to use messaging for information delivery. The PSTN (Public Switched Telephone Network) is the interconnection of POTS (Plain Old Telephony System) networks and PLMN (Public Land Mobile Network) networks operated by multiple network operators. To support VoIP (Voice Over IP), the PSTN interconnects to the Internet.

TringIt: Easy Triggering of Web Actions from a Phone

15

Modern phone networks use separate signaling and "data" channels and can be thought of as two parallel networks - the Common Channel Signaling network that uses protocols like SS7 (Signaling System 7) [6] to set up and tear down calls, and the "media" network that carries audio "data." Many networks now have all-IP cores. From the user perspective, however, the details are irrelevant. Users simply care about the fact that the call gets set up to the appropriate party when they place a call. A call in a telephone network proceeds in two stages: • A "signaling" stage in which the phone network locates and rings the called phone and provides feedback to the calling phone. • A "communication" stage involving voice transport between the calling party and the called party, possibly with data modulated over the voice channel. Importantly, the telephone network transmits two key pieces of information in the signaling stage of a call - the calling number and the called number. Dialed Number Identification Service (DNIS) [11] is a telephone network service that lets a subscriber of the service determine which telephone number was actually dialed by a caller. Automatic Number Identification (ANI) [4] is a telephone network service that permits subscribers of the service to capture the telephone number of a calling party. Via such services, call signaling automatically transmits a small amount of information from the calling party’s network to the called party’s network. 3.1 Similarity between Phone Signaling and Web Requests Interestingly, a user's interaction to place a call via the telephone network can be likened to Web request. We can think of a phone number as a URL, the phone as a browser, and the PSTN as an amalgamation of Internet infrastructure (name servers, routers etc.) and Web servers. A user uses his phone (the browser) to manually enter a number (a URL) or to speed-dial a number (a "browser bookmark"). The phone (the browser) transmits a request into the phone network. The request includes information about the called party (the URL) and the calling party (a Header in the request). The phone network attempts to route the call to the appropriate party (like the browser uses DNS to identify the IP address of the target server) and delivers signaling information to the called party (like a Web request being delivered to the server). If there is an error during this process (e.g., invalid number) appropriate feedback is provided to the calling party (like a “404 Not Found” error in HTTP). Otherwise, the phone starts the "voice client" that allows the user to listen to ringing tones. If the call is answered by the called party, all subsequent interaction happens via bi-directional streaming over a voice channel that is set up as a side effect of successful signaling. (This is equivalent to the browser launching a helper application that supports bidirectional voice.) Placing a phone call, therefore, can thus be thought of as a special kind of Web "click" from a phone. So far, this click has had very limited range of behavior - the click either results in the voice call being established or failing. (Setting up of a dialup connection is a special case, where a voice channel is first established, and the established channel is used to modulate/demodulate data over a carrier signal.).

16

V. Anupam

TringIt generalizes the use of telecom signaling, enabling it to drive any Internetconnected application. In particular, it enables rich phone + Internet converged applications [8] that work with any phone. TringIt enables mobile phone users to interact with Web applications in a powerful yet easy-to-use way by enabling Web actions to be triggered by simply dialing numbers - "Dial to Click!" TringIt is most useful for users who do not yet use the mobile Internet - it uses SMS/MMS to deliver information to users in response to the received phone call. However, TringIt is also useful for mobile phone users who currently use the mobile Internet. Numbers can be discovered via any media - online, TV, print etc. In a few button presses from an idle screen - the phone number + SEND, the user can "connect" with the real world entity associated with that number. A mobile phone with Internet access can be driven to the appropriate Web site. 3.2 Signaling Numbers: Phone Numbers as URLs As described in the preceding section, dialing a number to place a phone call can be likened to making an HTTP request. A phone number can therefore be thought of as a URL! So far, these URLs have simply been used to trigger the setting up of a voice call. We know that any action on the Web can be represented as a URL [5] - data needed for that action can either be encoded into the URL used for an HTTP GET request, or can be stored in a descriptor that is used to generate an HTTP POST request to the appropriate server. A phone number can be associated with such a URL by simply maintaining a lookup table mapping the phone number to the URL. If the system can detect when the phone number is called, it can use the lookup table to retrieve the associated URL and can trigger the appropriate Web action. This fundamentally changes what users can do by calling a number! The behavior of such a phone number is different from a regular phone number that is used to set up a voice call. We refer to this kind of number as a "Phone Signaling Number" (more concisely, "Signaling Number.” Any Web action can be associated with a Signaling Number, and thus be triggered by a simple phone call from any phone - wired or mobile. Importantly, this requires no new software in the phone, and thus works out-of-the-box with every phone in use on every network. If the associated Web action causes the generation and transmission of e.g. an SMS/MMS message to the phone, the calling user can dynamically receive "feedback" for the call – possibly containing requested information! The numeric form of this new kind of URL is ideal for easy input on the phone. Its compact and familiar form makes it easy to display and disseminate via any medium - books, magazines, newspapers, flyers, packaging, receipts, TV, radio, video and audio content, Web sites, email, SMS/MMS messages etc. And conventional letters-to-number mnemonics (1-777-4PIZZA to represent 1-777-474992) can make some subset of the numbers self-describing. These numeric URLs – “dial-able hyperlinks” – can easily be direct dialed from an idle phone screen. They can be stored in phone address books and can be speed-dialed and voice-dialed. They can be communicated from user to user. They can be embedded in SMS/MMS/email messages or in Web content and can be easily clicked by the receiving user.

TringIt: Easy Triggering of Web Actions from a Phone

17

3.3 Tring: A Phone Click As described earlier, when a call is being set up in the phone network, the calling party’s network transmits a small amount of information to the called party. This information can be used to create tremendous value. If the called party is associated with multiple phone numbers, the calling party can signal different things to the called party, based on the number it calls. If the calling party and called party can maintain state - i.e. remember information about previous transmissions, the interaction gets richer. E.g. if the calling party calls different numbers in some sequence it can convey additional information to the called party. Additionally, if the calling party and called party have shared context - i.e. some persistent (non-transient) data that both parties share (that is possibly transmitted and shared between the calling party and the called party using some separate offline mechanism) - the interaction gets even richer. The shared context allows specific meaning to be attributed to calls to and from different numbers, and the calling and called parties can be aware of that meaning. The ability to exchange user-to-application signals in the presence of state as well as shared context enables very compelling applications. TringIt enables the act of placing a call to a Signaling Number to be interpreted as the intent of the calling party to trigger the associated action registered by the called party. The associated Web action can equivalently be triggered by the called party or by any entity along the signaling path between the calling party and the called party. This makes the humble phone call much more versatile. A simple call can be used to trigger any Web action that has been associated with the number – a “Phone Click.”

TringIt Server Mobile Phone Network

1

3

Phone Network

Internet

2

Web Server Fig. 1. The Versatile “Phone Click”

Figure 1 depicts the flow of a typical Phone Click. In Step 1, signaling information travels over the mobile phone network and the PSTN to a TringIt Server when a call is placed to a Signaling Number. The TringIt Server looks up and triggers the corresponding Web action in Step 2, over the Internet. In Step 3, the TringIt Server transmits a message to the calling phone via the Internet and mobile phone network. The TringIt Server is a network-based server that interfaces to the phone network to receive Phone Clicks - calls that are used to trigger Web actions. Details of the TringIt Server are outside the scope of this paper, and are being published separately

18

V. Anupam

[3]. At a high level, the TringIt Server interconnects to the PSTN via physical and logical connections. It receives and terminates phone calls aimed at Signaling Numbers and triggers appropriate Web actions in response. Any phone number can potentially be used as a Signaling Number – the TringIt server just needs to be able to receive the call. Calls can be directed at the TringIt Server by routing or forwarding. For best usability, however, Signaling Numbers should come from a separate part of the phone number-space so that they are visually identifiable as numbers that trigger Web actions (as opposed to those that set up calls.) This can be achieved, e.g., by using numbers from a separate unused area code or country code, to create a concept like 1-800 numbers – users “know” that calls to 1-800 numbers are toll free. An appropriate prefix can be allocated and assigned by the appropriate phone number administration authority that controls and manages the phone numbering space. 3.4 Network-Based Context and Personalization One of the key challenges for non-voice use of mobile phones is usability. Portability and power constraints impose the requirement that the screen be small. A majority of phones only have a small set of buttons (typically 15-20), most representing digits for dialing numbers (and via techniques that use multiple taps per key, allow users to input alphabets and symbols as well). While mobile phones have been optimized for numeric and voice input, non-voice and non-numeric input is still cumbersome. Usability of interaction from such constrained devices can be greatly improved by maintaining as much information as possible in the network and allowing users and application servers to use that information easily as needed. We refer to this userrelevant information in the network as "Network Context" or simply context. TringIt stores commonly used user profile information like email addresses, postal addresses, preferences etc. in the network. The user maintains information in the context through a rich, non-phone mechanism – a Web portal accessed via an Internetconnected PC. The user can also maintain context information directly from the mobile phone - via a voice portal, SMS portal, phone client etc. if desired. The mobile phone differs from a Web browser in a PC in a very important way. It is the only device that most users carry with them almost all the time yet rarely share with other users - making it an ideal device to deliver a personalized interactive experience. As described earlier, signaling information that is transmitted across the phone network to set up a phone call typically includes the calling party number and the called party number. TringIt uses information derived from the calling party number as part of the data used to trigger the Web action. Awareness of the calling party's "identity" is used to provide personalized services triggered by the Phone Click – TringIt uses ANI information as a key to look up profile data in the context database. Importantly, ANI information is transmitted by default - and requires no manual intervention on the part of the calling user. The calling user not only indicates to TringIt what action he wants to trigger (based on the called number) but also simultaneously, and with no extra effort, provides information that can be used to personalize his request. This greatly improves usability for the calling user. TringIt uses well-known and understood techniques and best practices for maintaining context

TringIt: Easy Triggering of Web Actions from a Phone

19

information in the cloud, and to deal with issues of scalability, reliability, security, access control etc. Maintaining personalizable network context has a significant additional benefit – it enables multi-modal interaction. TringIt allows the user to dynamically intermix interaction via multiple input channels - e.g. via voice, by sending an SMS/MMS message, by clicking a link on a Web page or by using an application, and via Phone Clicks - the user can choose the channel that works best based on dynamic usability and price considerations. 3.5 Internet Access Not Required The mobile phone differs from a Web browser in a PC in some other key ways. By virtue of built-in technologies (e.g. a messaging client, address book), a mobile phone provides “data persistence” and is also “spontaneously reachable” from the network – making it an ideal portable receiver of information delivered by Web applications. TringIt leverages these features to make the mobile phone much more useful, even without Internet access. TringIt enables any mobile phone user to easily use the phone as an input device to trigger Web applications – like a programmable universal remote control – that then deliver information via messaging. The Phone Click is also useful for users with mobile Internet access. The incoming message can be a rich email message containing hyperlinks to Web sites, or a WAP Push that drives an in-phone Web browser to a particular mobile Web site to start a browsing session, or can wake up an in-phone application, triggering it to perform some action. A key contribution is the elimination of the need for end-to-end HTTP from the phone. Users can request information via a Phone Click. Application servers can retrieve Web documents on behalf of the user, transform the results appropriately to fit into an SMS or MMS message, and transmit the message to the user. Users can allow received information to persist in the mobile phone for as long as required, possibly eliminating subsequent need to re-fetch that information. TringIt makes it possible for a user to request news, weather and traffic updates, stock quotes etc. by simply dialing the corresponding number. The information request is personalized using profile data stored in the network. By simply adding numbers and content providers, TringIt makes it possible for users to easily request directions to a hotel, information about a product, daily specials at a restaurant, check order status etc. Like a Web click, the possibilities are limitless.

4 Conclusion The Phone Click uniquely enables a class of applications that work by simply calling numbers – without requiring any additional input over the telephony media channel. These applications automatically trigger Web actions and possibly send information to users in response to a call. By using information that is delivered over the signaling channel as well as any context information available in the network (relevant to the calling number, the called number and the application server) very compelling applications can be created - the caller simply needs to be aware of the action that will be triggered by the application server when he calls that number. The Phone Click can

20

V. Anupam

be employed to enable useful services for all phone users, not just users with mobile Internet access. Importantly, the Phone Click complements all existing channels for communication from a mobile phone - voice, messaging, USSD and mobile Web, making them all more powerful. This user-to-application signaling capability is a broadly useful enabler. It enables all mobile phone users to economically and easily initiate contact and interact with businesses, brands and information providers via their mobile phone. Users simply need to know the number to dial, and numbers can easily be discovered via traditional offline and online channels. TringIt eliminates key usability and cost “friction,” enabling mobile phone users to economically and easily initiate contact and interact with information providers via their mobile phone. It makes the mobile phone a more powerful tool and unlocks its potential as the ultimate information, interaction and participation device. Future Work. While the act of dialing numbers to initiate a voice call is very familiar to users, the notion of triggering Web actions by doing so is not. The authors plan to study the usability barriers associated with the Phone Click via experiments and focus groups. The authors are working on simplifying discovery of Signaling Numbers and on simplifying browsing/review of information delivered by applications – existing software in mobile phones offers significant scope for improvement – and are designing an in-phone client application that will further simplify the user experience. The authors are also developing cloud-based infrastructure that will enable businesses, brands and content providers to easily integrate the Phone Click into their applications.

References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.

3GPP TS 03.40 – Technical Realization of the Short Message Service (SMS) 3GPP TS 03.90 – Unstructured Supplementary Service Data (USSD) Anupam, V.: Using Phone Signaling for Application Signaling (in preparation) Automatic Number Identification, http://en.wikipedia.org/wiki/Automatic_number_identification Berners-Lee, T., Cailliau, R., Luotonen, A., Nielsen, H.F., Secret, A.: The World-Wide Web. Comm. ACM 37(8), 76–82 (1994) Dryburgh, L., Hewitt, J.: Signaling System No.7 (SS7/C7): Protocol, Architecture and Services. Cisco Press (2004) Interactive Voice Response, http://en.wikipedia.org/wiki/IVR Kocan, K.F., Roome, W.D., Anupam, V.: A Novel Software Approach for Service Brokering in Advanced Service Architectures. Bell Labs Technical Journal 11(1), 5–20 (2006) Nielsen Mobile: Critical Mass - The Worldwide State of the Mobile Web (July 2008) OMA Multimedia Messaging Service V1.3 Dialed Number Information Service, http://en.wikipedia.org/wiki/DNIS W3C Voice Extensible Markup Language (VoiceXML) 2.1 Yahoo Go!, http://mobile.yahoo.com/go

Context Awareness and Perceived Interactivity in Multimedia Computing Xiao Dong1 and Pei-Luen Patrick Rau2 1

Industrial & System Engineering Department, University of Minneosta, Minnesota, USA [email protected] 2 Industrial Engineering Department, Tsinghua University, Beijing, P.R. China [email protected] H

H

H

H

Abstract. Context awareness and perceived interactivity are two factors that might benefit mobile multimedia computing. This research takes mobile TV advertisements as a scenario and verifies the impacts of perceived interactivity and its interaction with context awareness. Seventy-two participants were recruited and an experiment was conducted in order to identify those impacts. The main findings indicated the following: (1) the effect of high perceived interactivity advertisement is significantly better than the effect of low perceived interactivity advertisement; (2) the interaction of context awareness and perceived interactivity has a significant influence on the effect of mobile TV advertising. Keywords: Context awareness, perceived interactivity, mobile TV advertising.

1 Introduction Mobile multimedia has emerged as the hottest growth area in wireless services. It brings a host of new features and functions into the wireless market, providing advanced forms of communication, entertainment and productivity [1]. Mobile operators are investing considerably in broadcasting mobile TV with fully fledged services in various countries throughout Asia as well as other large scale trials around the world [2]. Although mobile advertising and TV advertising have been studied for many years, few researchers have extended their studies to include TV advertising on the platform of mobile devices. With the development of network and hardware capacity, mobile devices can be used to watch TV programs. The purpose of this study was to identify user perception and response to mobile multimedia services. The result can be a guidance of advertisement designers and a base for further studies.

2 Literature Review 2.1 Mobile Multimedia Mobile TV fulfills the growing need of entertaining and staying informed on the move. Mobile TV builds on established consumer behavior: end-users are familiar J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 21–29, 2009. © Springer-Verlag Berlin Heidelberg 2009

22

X. Dong and P.-L.P. Rau

with the concept of television and, with the continued need for mobility, the benefits of this new medium are clear. Mobile TV is enhanced by the element of interactivity, which adds value to the user experience and makes it a richer entertainment option. 2.2 Mobile Advertising Compared with traditional advertising media, mobile advertising can promote sales of goods and services, create brand images and product awareness (branding), disseminate information using a personally relevant and context-aware approach, support direct business transactions and encourage customer interaction [3,4]. In recent years, mobile advertising has been enhanced by the processing capability of handheld devices and the development of networks. Consequently, some innovative advertising methods are found in everyday life. For instance, mobile games and MMS advertising have emerged as creative advertisement venues. However, mobile advertisers must be very careful not to risk privacy issues and exhaust customer tolerance. We are of the opinion that it is the users and not the media designers or the market that are the ultimate determinants of effectiveness. 2.3 Context The context is frequently determined by user location [5,6]. Hence, depiction and association of user location is pivotal to a context-sensitive advertising system. Potential application fields can be found in areas such as travel information, shopping, entertainment, event information and different mobile professions [5]. Another reason for the importance of location is that it is easier to identify and measure compared with other context components. It can be measured with different positioning systems, such as embedded GPS modules, mobile phones which can be located by the telecom operator of a network, or service points utilizing WLAN, Bluetooth, or infrared technologies. 2.4 Perceived Interactivity Interactivity is generally believed to be a multi-dimensional construct [7,8,9,10], but there is no general consensus regarding the nature and the content of the dimensions. Based on constructs identified in internet studies and analysis of the characteristics of mobile communication, a model of interactivity for mobile advertisements was constructed by Gao et al. [11], comprising user control, direction of communication, synchronicity, connectedness, playfulness, and interpersonal communication. They also stated that different mobile advertising tools might differ in these dimensions due to the different communication style each tool has. For example, message push-ads might allow less user control, but an included reply option will give customers a convenient channel to respond; mobile banners are less intrusive compared with pushads, but they might be ignored or assumed to be only decorative images. User control is conceptualized as the degree of user intervention that is required to operate the system [12]. Dholakia et al. refer to user control as the extent to which an individual can choose the content, timing and sequence of a communication to change his/her viewing experience [13]. It is taken as the core component of interactivity by some researchers [10,14]. Two constructs identified by Steuer [10], range and

Context Awareness and Perceived Interactivity in Multimedia Computing

23

mapping, actually describe two aspects of control. The former refers to the number of options the environment provides the user to modify the task flow and the environment, and the latter refers to the extent to which the controls and manipulations in a computer-mediated environment are similar to controls and manipulations in the real word. This paper manipulates interactivity by adding different user controls to the advertisement.

3 Hypotheses and Methodology 3.1 Hypotheses Hypothesis 1: High perceived interactivity advertisements will have better advertising effectiveness(better memory, better attitudes towards the ads and brand, higher purchase intention) than low perceived interactivity advertisements. For message advertising, user control options are important. Control choice/range and mapping have been traditionally considered as fundamental constructs of interactivity [15,10]. Users can respond by replying to the message directly, call back with a provided telephone number, or visit another source linked in the message. Users also want the control and manipulation in computer-mediated environments to be similar to those in the real world. The more a user can control the options provided, the more interactive the customer perceives the advertisement to be. The more similar the mediated environment is to the real world, the more interactive the customer perceives the advertisement to be. Studies on Internet advertising interactivity have found that there is a strong correlation between perceived interactivity and advertising effectiveness in terms of attitude towards brand, attitude towards ads, and purchase intention [16,17,18]. Previous study also suggested that higher interactivity helps the customer experience “flow” during the interaction [19] and the consequences of the “flow” experience are increased learning and perceived behavioral control. Therefore it was hypothesized that high perceived interactivity has a positive influence on advertising effectiveness. Hypothesis 2: The interaction between interactivity and context-awareness will have an influence on mobile advertising effectiveness. While context-awareness ads give users higher involvement and make the ads more relevant, interactivity provides the customer a chance to communicate with the company, to search for further information or to disseminate information to others conveniently and quickly. As Kannan et al. [20] have already pointed out, it is critical to provide the customer a chance to respond at the point of purchase or usage immediately when sending context-aware advertisements. Immediately redeemable m-coupons, callback numbers, or simply a message requiring a reply from the customer are hypothesized as most likely to exert influence when sent in a contextaware manner compared to when sent in a context-irrelevant manner. Context awareness and interactivity are hypothesized to have an interaction effect on advertising effectiveness.

24

X. Dong and P.-L.P. Rau

3.2 Experiment Participants. Seventy-two participants (36 female and 36 male) from Universities in Beijing voluntarily took part in the experiment. They were randomly assigned to six groups with the combination of two-level context awareness and three-level perceived interactivity. The participants were all undergraduate and graduate students with no prior knowledge about the tasks to be performed during the experiment. The participants’ ages ranged from 20 to 36 years (mean=24, S.D. =2.26) and 55% of the participants had used mobile phones for more than five years. In addition, 98% of the participants had previously received SMS advertisements, and 32% of the participants had received MMS advertisements. 90% of the participants had more than five years of Internet experience, while 32% of the participants had experience connecting to the Internet via mobile devices. Experimental design and variables. Independent variables are perceived interactivity and its interaction with context awareness. Dependent variables were advertising effectiveness which consisting of memory of the advertisement, attitude towards the advertisement and brand, and purchase intention. Memory was measured by a free recall and recognition test. The ads attitude [8], brand attitude [21], user involvement [22] and purchase intention was measured using scales from other researches. Procedures. Each participant was tested individually. They were asked to complete a demographic and technology (internet, mobile services and TV advertisements) usage questionnaire. Then the participants were given an introduction to the experiment’s procedure and their tasks. A practice task was provided to let the participants view sample mobile TV advertisements and make sure they knew how to use the experiment devices (PDA-ASUS A620, KONKA TPC880). During the experiment, all participants visited five different scenarios (mall, bookstore, cell phone market, McDonald’s, and the IE building at Tsinghua University) with a predefined sequence. In each scenario, they did two information seeking tasks and viewed mobile TV advertisements on experiment devices after each task. According to different groups, context awareness and perceived interactivity features were embedded into mobile TV advertisements. Upon completion of all tasks, a free-recall test and an advertisement recognition test were given; the participants were not informed prior to the task that these tests would be applied. Then participants were asked to finish a post-test questionnaire, which measured the user’s attitude towards the ads, brand attitude, perceived interactivity, as well as purchase intention.

4 Results and Discussions In this section we present the reliability of each measure and the results of test hypotheses one and two. The internal consistencies for the questionnaire responses, using Cronbach’s a, were 0.95 for the advertisement attitude questionnaire, 0.95 for the brand attitude questionnaire and 0.87 for the involvement with the advertisement questionnaire, and 0.78 for the perceived interactivity questionnaire.

Context Awareness and Perceived Interactivity in Multimedia Computing

25

4.1 The Effect of Perceived Interactivity on Advertisement It was hypothesized in this study that high perceived interactivity advertisements would have better advertising effectiveness than low perceived interactivity advertisements. High range and high mapping are two different kinds of high level perceived interactivity. They are compared with low perceived interactivity separately to identify whether this hypothesis is true. After the experiment, the data showed that there were significant differences between exposure time to mobile TV advertisements in each group. This factor was used as a covariate in the ANCOVA process. The results can be found in Tables 1 and 2. Table 1. Data for Testing Hypothesis Three (High range×Low interactivity) Perceived interactivity High range

Low interactivity

Variable

Mean

SD

Mean

SD

P value

Ad free recall

2.93

1.73

1.77

1.05

Ad recognition

7.33

1.88

6.13

2.25

0.04*

Ad attitude

4.80

0.52

4.24

0.53

0.00*

0.003*

Brand attitude

4.87

0.53

4.28

0.41

0.00*

Purchase intention

4.69

0.69

4.08

0.49

0.00*

Table 2. Data for Testing Hypothesis Three (High mapping×Low interactivity) Perceived interactivity High range

Low interactivity

P value

Variable

Mean

SD

Mean

Ad free recall

2.82

1.46

1.77

1.05

0.006*

Ad recognition

7.46

2.43

6.13

2.25

0.02*

Ad attitude

4.60

0.65

4.24

0.53

0.02*

Brand attitude

4.62

0.57

4.28

0.41

0.01*

Purchase intention

4.35

0.70

4.08

0.49

0.10

SD

From these results we can see that hypothesis one is supported. High perceived interactivity advertisements will have better advertising effectiveness than low perceived interactivity advertisements. This was consistent with past studies. Cho and Leckenby [16] measured participants' intention to interact with a target (banner) ad and found positive relations (correlation coefficients ranging between .30 and .75) between intention to interact with the ad and attitudes toward the ad, attitudes toward the brand, and purchase intention. Yoo and Stout [18] also achieved similar results. McMillan and Hwang’s [17] study demonstrated that interactivity and involvement

26

X. Dong and P.-L.P. Rau

with the subject of a site were two possible predictors of positive attitude towards the Web site, and perceived interactivity accounted for more of the variance in attitude than did involvement. Analysis of relationships among the variables in the study suggested that the control sub-dimension of perceived interactivity had the strongest correlation with attitude toward the Web site. 4.2 Joint Effect of Context-Aware and Perceived Interactivity It was hypothesized in this study that the interaction between interactivity and context awareness influences mobile advertising effectiveness. As mentioned above, we also included exposure time to mobile TV advertisement in each group as the covariate in the ANCOVA test. The results showed that the interaction between perceived interactivity and context awareness significantly influenced the subject’s attitude towards mobile TV advertisements (F=4.183, p=0.019), their attitude towards brands (F=5.011, p=0.009), and their purchase intention (F=7.732, p=0.001). Although the interaction between context awareness and perceived interactivity have no significant influence on advertisement free recall and advertisement recognition, the p value is quite close to the significant level p=0.05 (F=2.22, p=0.117 for free recall; F=2.793, p=0.068 for recognition). Table 3. The Effect of Interactivity under the Context-aware Condition

Variable Ad free recall Ad recognition Ad attitude Brand attitude Purchase intention

Perceived interactivity High range High mapping Mean SD Mean SD 3.47 2.07 3.31 1.68 7.92 1.51 8.50 2.02 5.14 0.40 4.68 0.72 5.20 0.38 4.64 0.66 5.13 0.52 4.28 0.83

Low interactivity Mean SD 1.55 0.87 5.75 2.42 4.13 0.52 4.19 0.43 3.88 0.46

F value

P value

5.16 6.20 9.72 11.86 12.45

0.01 0.01 0.00 0.00 0.00

Table 4. The Effect of Interactivity under the Context-irrelevant Condition

Variable Ad free recall Ad recognition Ad attitude Brand attitude Purchase intention

Perceived interactivity High range High mapping Mean SD Mean SD 2.39 1.15 2.33 1.03 6.75 2.09 6.42 2.43 4.46 0.40 4.52 0.58 4.54 0.44 4.61 0.49 4.26 0.57 4.41 0.57

Low interactivity Mean SD 1.99 1.20 6.50 2.11 4.35 0.54 4.37 0.39 4.27 0.46

F value

P value

0.43 0.07 0.34 0.94 0.30

0.65 0.93 0.71 0.40 0.75

After examining the effect of interactivity in different context conditions (Tables 3 and 4), we found that when mobile advertisements are distributed in a context-aware manner, the interactivity of ads has a positive influence on the advertising effectiveness in terms of advertisement memory, advertisement attitude, brand attitude, and purchase intention. However, when mobile ads are distributed in a context-irrelevant way, the interactivity of advertisements has no significant influence on advertisement memory,

Context Awareness and Perceived Interactivity in Multimedia Computing

27

advertisement attitude, brand attitude and purchase intention. This finding is partially contradictory with previous literatures in Internet advertising, where it is generally agreed that perceived interactivity has positive influences on advertising effectiveness [16,17,18]. The main reason for the lack of significant effects of interactivity in the context-irrelevant situation may be the lower message involvement in the contextirrelevant condition. First, low involvement leads to a lower information processing level, which may cause the subject to neglect the interaction options of the advertisement or even the advertisement itself. Second, lower involvement with the advertisement content also results in lower motivation to respond to the ad [23]. The lower motivation to respond makes it less critical to provide interaction options, since they do not seek interaction options from the beginning. When the same advertisements were sent in a context-aware manner, however, the effects of the interactivity of mobile advertisements on advertising effectiveness became significant, as expected in hypotheses one. The results in Tables 5, 6 and 7 show that when the interactivity of mobile ads is low, to send them in a context-aware manner could result in even worse advertising Table 5. The Effect of Context Awareness to High Range Advertisements

Variable Ad free recall Ad recognition Ad attitude Brand attitude Purchase intention

Context awareness Location relevant Location irrelevant Mean SD Mean SD 3.47 2.07 2.39 1.15 7.92 1.51 6.75 2.09 5.14 0.40 4.46 0.40 5.20 0.38 4.54 0.44 5.13 0.52 4.26 0.57

F value

P value

2.46 2.46 17.55 15.50 15.13

0.13 0.13 0.00 0.00 0.00

Table 6. The Effect of Context Awareness to High Mapping Advertisements

Variable Ad free recall Ad recognition Ad attitude Brand attitude Purchase intention

Context awareness Location relevant Location irrelevant Mean SD Mean SD 3.31 1.68 2.33 1.03 8.50 2.02 6.42 2.43 4.68 0.72 4.52 0.58 4.64 0.66 4.61 0.49 4.28 0.83 4.41 0.57

F value

P value

2.97 5.21 0.37 0.02 0.19

0.10 0.03 0.55 0.90 0.67

Table 7. The Effect of Context Awareness to Low Interactivity Advertisements

Variable Ad free recall Ad recognition Ad attitude Brand attitude Purchase intention

Context awareness Location relevant Location irrelevant Mean SD Mean SD 1.55 0.87 1.99 1.20 5.75 2.42 6.50 2.11 4.13 0.52 4.35 0.54 4.19 0.43 4.37 0.39 3.88 0.46 4.27 0.46

F value

P value

1.06 0.66 0.98 1.05 4.14

0.31 0.43 0.33 0.32 0.05

28

X. Dong and P.-L.P. Rau

effectiveness than to send them in a context-irrelevant manner, in terms of memory, advertisement attitude, brand attitude and purchase intention. However, when the interactivity of mobile ads is high, it is better to send them in a context-aware manner so as to promote the advertising effectiveness in terms of memory, advertisement attitude, brand attitude and purchase intention. In this study, we also found that the high range effect was greater than the high mapping effect as the indicator of perceived interactivity. The finding that context awareness with low interactivity ads has a negative influence on advertisement effectiveness is surprising. We propose two reasons to address this. (1) Location-based services and advertisements should be provided with great care so as not to invade users’ privacy, since handsets are very personal devices. The low interactivity ads only “broadcast” information about products and services. It is possible that the feeling of being invaded by such ads might become even stronger when users detect that their cell phones are being spammed just because they happen to walk past a particular store. (2) Mobile TV advertisement is characterized by its rich media features. Context awareness may influence advertisement effectiveness through other factors, such as interactivity. But further study is required to identify their relationship.

5 Conclusion and Future Study The effect of perceived interactivity is clear in this study. High perceived interactivity advertisements have better advertising effectiveness than low perceived interactivity advertisements. This finding is consistent with former research on internet and mobile message advertisement. One guideline of perceived interactivity based on this research ought to be formed to give mobile TV advertisement business market instructions. The effects of context awareness on mobile TV advertising effectiveness take place according to different levels of interactivity. With high interactive advertisements, contextual advertising information does increase user response effectively and results in a more accepting attitude. Therefore, when the goal of a mobile advertising campaign is to generate responses, the context in which the response options are given to the users are of importance and must be taken into consideration by the mobile marketer. However, with “broadcasting” advertisements, the user’s attitude towards the brand and the consequent purchase intention were impaired by context aware advertisements rather than improved. This interaction style is needed to be thoroughly studied by future studies.

References 1. Robert, W.S.: Mobile multimedia goes Net-centric. Electronic Engineering Times. Manhasset: Mar 5(1156), 78–79 (2001) 2. Kenton, O., April, S.M., Alex, V.: Consuming Video on Mobile Devices. In: Proc. CHI 2007, pp. 857–866. ACM Press, New York (2007)

Context Awareness and Perceived Interactivity in Multimedia Computing

29

3. Dickinger, A., Haghirian, P., Murphy, J., Scharl, A.: An Investigation and Conceptual Model of SMS Marketing. In: Proc. 37th Hawaii International Conference on System Sciences 2004, p. 10031.2. IEEE Computer Society Press, Los Alamitos (2004) 4. Yunos, H., Gao, J.: Wireless Advertising’s Challenges and Opportunities. Computer 36, 30–37 (2003) 5. Kaasinen, E.: User needs for location-aware mobile services. Personal and Ubiquitous Computing 7, 70–79 (2003) 6. Younghee, J., Per, P., Jan, B.: DeDe: Design and Evaluation of a Context-Enhanced Mobile Messaging System. In: Proc. of CHI 2005. ACM Press, New York (2005) 7. Wu, G.: Perceived Interactivity and Attitude Toward Web Sites. In: Proc. Conference of the American Academy of Advertising 1999. American Academy of Advertising (1999) 8. Ha, L., James, E.: Interactivity Reexamined: An Analysis of Business Web Sites. In: Proc. Conference of the American Academy of Advertising (1998) 9. Liu, P., Shrum, L.: What is interactivity and is it always such a good thing? Implications of definition, person, and situation for the influence of interactivity on advertising effectiveness. Journal of Advertising 31, 53–64 (2002) 10. Steuer, J.: Defining Virtual Reality: Dimensions Determining Telepresence. Journal of Communication 42(4), 73–93 (1992) 11. Gao, Q., Rau, P.L.P., Salvendy, G.: Measuring perceived interactivity of mobile advertisements. Behavior & Information Technology (2006) 12. Van der Heijden, H.: Ubiquitous computing, user control, and user performance: conceptual model and preliminary experimental design. In: Proc. Tenth Research Symposium on Emerging Electronic Markets, pp. 107–112 (2003) 13. Dholakia, R., Zhao, M., Dholakia, N., Fortin, D.: Interactivity and Revisits to Websites: A Theoretical Framework, http://ritim.cba.uri.edu/wp/ 14. Bezjian-Avery, A., Calder, B., Lacobucci, D.: New Media Interactive Advertising vs.Traditional Advertising. Journal of Advertising Research 38(94), 23–32 (1998) 15. Coyle, J., Thorson, E.: The Effects of Progressive Levels of Interactivity and Vivideness in Web Marketing Sites. Journal of Advertising 30(3), 65–78 (2001) 16. Cho, C.-H., Leckenby, J.: Interactivity as a measure of advertising effectiveness: Antecedents and Consequences of Interactivity in Web Advertising. In: Proc. Conference of the American Academy of Advertising (1999) 17. McMillan, S., Hwang, J.S.: Measures of Perceived Interactivity: An Exploration of Communication, User Control, and Time in Shaping Perceptions of Interactivity. Journal of Advertising 31(3), 41–54 (2002) 18. Yoo, C.Y., Stout, P.: Factors Affecting User’s Interactivity with the Web site and the Consequences of User’s Interactivity. In: Proc. Conference of the American Academy of Advertising (2001) 19. Hoffman, D., Novak, T.: Marketing in Hypermedia Computer-Mediated Environments: Conceptual Foundations. Journal of Marketing 60(3), 50–68 (1995) 20. Kannan, P., Chang, A., Whinston, A.: Wireless commerce: marketing issues and possibilities. In: Proc. International Conference on System Science 2001. IEEE Computer Society Press, Los Alamitos (2001) 21. Li, H., Bukovac, J.L.: Cognitive Impact of Banner Ad Characteristics: An Experimental Stud. Journalism and Mass Communication Quarterly 76(2), 341–353 (1999) 22. Norris, C.E., Colman, A.M.: Context effects on recall and recognition of magazine advertisements. Journal of Advertising 21(3), 37–46 (1992) 23. Petty, R.E., Cacioppo, J.T.: Attitudes and Persuasion: Classic and Contemporary Approaches. Westview Press (1981)

Human Computer Interaction with a PIM Application: Merging Activity, Location and Social Setting into Context Tor-Morten Grønli and Gheorghita Ghinea The Norwegian School of Information Technology, Schweigaardsgt. 14, 0185 Oslo, Norway School of Information Systems, Computing and Mathematics, Brunel University, Uxbridge UB8 3PH, London, United Kingdom [email protected], [email protected]

Abstract. Personal Information Managers exploit the ubiquitous paradigm in mobile computing technology to integrate services and programs for business and leisure. Recognizing that every situation is constituted by information and events, this context will vary depending on the situation users are in, and the tasks they are about to commit. The value of context as a source of information is highly recognized and for individual dimensions context has been both conceptually described and prototypes implemented. The novelty in this paper is a new implementation of context by integrating three dimensions of context: social information, activity information and geographical position. Based on an application developed for Microsoft Window Mobile these three dimensions of context are explored and implemented in an application for mobile telephone users. Experiment conducted show the viability of tailoring contextual information in three dimensions to provide user with timely and relevant information. Keywords: PIM, context, context-aware, Microsoft pocket outlook, ubiquitous computing, HCI.

1 Introduction Personal Information Managers (PIM’S) exploit the ubiquitous paradigm in mobile computing technology to integrate services and programs for business and leisure. Activities performed with PIMs range from plotting appointments and tasks in a calendar to the automatic information exchange between mobile devices, different device synchronizations and base station communication. In every situation in our daily life, an independent context of information and events is defined. This context will vary depending on the situation users are in, and the tasks they are about to commit. Nonetheless, despite the fact that the challenge of defining context has already been addressed [1-3] and that its value as a source of information is recognised, the link between context and PIMs is still an immature and unexplored area of research. This paper addresses the use of context in an everyday mobile phone based PIM application which makes use of context based information to enhance the user experience. J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 30–38, 2009. © Springer-Verlag Berlin Heidelberg 2009

Human Computer Interaction with a PIM Application

31

The unique use of context in this PIM application combines activities, social information and geographical information.

2 Background Developers and researchers agree that context is an important factor when designing new applications. With PIM devices becoming increasingly widespread and in daily use by a large population, this opens interesting possibilities for development. Such applications would potentially be used daily by people in their homes or at their workplace, especially bearing in mind that people carry mobile devices and thereby the application with them almost 24 hours a day. Recent arguments state the possibility for business travelers and other mobile workers to leave the laptop at home and shift entirely to mobile devices because of their increased capacity and convenient size. Context in mobile applications has been looked at by more than a few researchers [1,10]. For example, Ludford et al. [8] looked at the use of context to provide useful information to the user on their mobile phone. In their work, context is based on the location and/or time of the day. This partly makes use of daily available context information, which, however, is not instantly fed back into the system as parameters for information retrieval. Efforts have also been made to make use of context as a tool for supporting business travelers in [8]. The definition of context here as the user’s planned activity in combination with the location is quite interesting. This, because it generates quite a lot of information about the user, but information is of reduced interest if we have no way of making use of it. Zhou et al. [11] have also demonstrated the use of context sensitive information for the real estate business. These are just two examples out of many and one could only imagine many other possible scenarios. On an overall basis, though, the use of context in applications is often missing or single dimensional. This focus should be changed, since automated PIM applications which take into account the total context of the user would possibly be able to not only support the everyday tasks of the user, but also improve efficiency and ease the work of the user by automatically tailoring information to the user’s needs and/or adapting the application to the user’s current setting. The CityFlocks application [1] is one step in this direction, however it falls short of offering a full solution to the problem. The widespread use of small mobile devices (PIMs) has, as shown, forced researchers and developers to consider context as an important criteria for highly mobile systems. The notion of context-aware computing is generally the ability for the devices to adapt their behavior to the surrounding environment, hence enhancing usability [7]. Towards this goal, Dey and Abowd [2] state that if we understand context fully in a given environment and setting, we would then be able to better choose what contextaware behaviours to sustain in applications. This could lead to more realistic applications and thereby applications more meaningful to users. This is also exemplified by Edwards [4] when he uses context information to build an application. In his application different layers represent different sources of information and they can be reused in later settings. Edwards argues that context is a major part of our daily life and that

32

T.-M. Grønli and G. Ghinea

computing with support for sharing and using contextual information (context-aware computing) would improve user interaction. Indeed when viewing people rather than systems as consumers of information, a new infrastructure is needed.

3 Design Our application interacts with the user by presenting information relevant to the users’ context. To be able to do this, system design and functionality are split into three main modules, each of which generate context-aware information and responds to different user interactions. This enables a precise definition of elements and thereby tailoring of information to be displayed according to the users’ actual needs. One of these main modules handles activity, one handles social context and the third handles the geographical location. The input from all three sources together provides the foundation for the user context computation. By this operationalization of context and contextaware concepts we are able to create a user context. This user context is then the foundation upon which the application takes action to present information. We now proceed to provide further details of each of the three modules involved in our PIM application. 3.1 Social Context This module computes the foundation upon which the social context is determined. Social context will naturally differ tremendously based upon each situation and for each individual. Still, it is possible to use one social context in common by several people by choosing concepts that are interpreted the same by most people. This is achieved through building a taxonomy of concept terms illustrated in Table 1 below. Table 1. Taxonomy of social context

Categories:

Sub categories:

Leisure

Work

Shopping Cinema Spare time Food Culture

Meeting Preparation Own time Travelling Phone meeting

Travel Train Car Tube Foot Transport

In our approach, building on the description of activities from Prekop and Burnett [9] information of the social context in the application are stored and meta-tagged as Pocket Outlook activities / appointments. These tags are based on the above taxonomy and are implemented for the user through extending the standard Pocket Outlook category interface (Figure 1). Thereby the user can enrich (tag) the activity with category tags through a familiar interface and thereby greatly increase the familiarity of the system.

Human Computer Interaction with a PIM Application

33

Fig. 1. Taxonomy interface

3.2 Geographical Location Module This module calculates location based on input from the internal Global Positioning System (GPS) in the device. When a valid connection through the GPS device to a GPS satellite occurs, it returns location information to the application. The input from the GPS is then parsed and the actual location retrieved by inspecting the longitude and latitude coordinates. These coordinates are then mapped to a one of 16 specific zones (in our case, Oslo, Norway). As the user is moving, information about the current zone is updated and stored in the application. This is done by letting the device interact with the running applications data sources and although the user not actually is feeding any information into the device, the physical moving around is sufficient for context-aware exploitation of user information. 3.3 Activity Module This module communicates with the Microsoft Pocket Outlook storage on the mobile device and retrieves appointments and activities. The module accesses the Pocket Outlook storage directly and also listens to system events generated in this storage. The user interacts with the activity module through the familiar Pocket Outlook interface, and attaches one or more of the category terms as described previously. In doing this, almost unknowingly, the user improves the quality of activity information and thus eases the use of the PIM application.

4 Implementation The application prototype is designed for and implemented on a Pocket PC device [6] (HTC 3600 phone) using the Microsoft Windows Mobile 6.0 operating system. The application is programmed in Microsoft .NET Compact Framework with C# as

34

T.-M. Grønli and G. Ghinea

implementation language. Geographical position is acquired through the use of GPS and activities and appointments are acquired through Microsoft Pocket Outlook. All data on the device are kept continuously up to date by computer synchronization with Microsoft Outlook 2007. This device is also chosen since it contains powerful enough hardware and has a decent enough storage area to be suitable for software development.

5 User Evaluation The PIM application was evaluated with a test group of 15 users who undertook a set of social and work-related activities whilst navigating a route through central Oslo (Figure 2).

Fig. 2. Suggested route through city

Users had to complete an evaluation questionnaire after the testing shown in Table 2 below. For all questions, users were asked to state whether they agreed or not to each statement. Measurement scale: Strongly Disagree (SD), Disagree (D), Mildly Disagree (MD), Mildly Agree (MA), Agree (A) and Strongly Agree (SA). Each possible answer from SD to SA was mapped to a number from 1 to 6, respectively, and the responses thus received were analyzed using a T-test (Table 3). We will in the following sections elaborate on the implications of our evaluation exercise.

Human Computer Interaction with a PIM Application

35

Table 2. Questionnaire

1 2 3

The information provided by the reminder system correctly matched my current location The information provided when I was “Sightseeing in old town” was incorrect. The summary of blocked events I received after appointment “DNB Nor Solli plass” was useful

4 5

The system provides duplicated information I liked the fact that the application is integrated with Outlook

6 7

The reminder system is useful I would use this application in my daily life

User responses to the evaluation questionnaire are summarized in Table 3 below. Table 3. T-Test results

Question 1

Mean Response 4.80

T-Value

P

8.088

0.000

2

2.47

-1.054

0.310

3

3.73

1.585

0.135

4

2.73

-0.654

0.524

5

5.47

12.854

0.000

6

5.20

12.602

0.000

7

3.93

3.287

0.005

From Table 3 we can see that all answers display a positive bias (Questions 2 and 4 are in fact negative statements about the application, therefore the negative bias here, reflects positive user statements). In question 1, 14 out of 15 answered that information displayed did match their current location and the responses are statistically significant. This would indicate a correct computed context and correct displayed information on an overall basis. For question number two, 11 out of 15 respondents were negative to that the information being displayed in one appointment was incorrect. This indicates that 11 got, at least partly, displayed correct information and four had incorrect or no information displayed. There is thus a strong bias towards negative answers, a few (4) positive answers and no middle values (MA / MD). This polarization of results however leads to the data for this question not to be statistically significant.

36

T.-M. Grønli and G. Ghinea

Fig. 3. Questionnaire results

As described, the application prototype is adaptable to different scenarios and user settings, but a context dependent application needs to be tailored to the users’ need when being deployed for a real life setting, e.g. the initial categories and their weight need to be configured in accordance with the findings of Prokop and Burnett [9]. When these issues are taken care of, the user experience might improve and increase even more the positive trend in answers to question six and seven. Moreover, as shown by Zhou et al. [11], information tailoring is an important task to help users interpret data. Our application focuses on tailoring by having minimal information displayed at the same time, when new messages are shown, thereby easing the users’ interpretation. Currently calendars and applications based on these do not take multi contextual information into account [7] as they often only reproduce the information already available there. Thereby, this can at worst lead to incorrect display of data and at the best a reproduction of data in a new interface. Our developed PIM application greatly differentiates from this by only displaying information based on the computed user context, given by the three factors social context, location and activity / appointment. Earlier approaches that have made use of calendar data from Pocket Outlook, often end in using Pocket Outlook data together with a simple timeline (i.e. [5]). In our approach, the use of Pocket Outlook data is extended to not only retrieve and display the data, but also to add extra meta-information to the appointments. Results from question five show that all respondents stated they liked the integration of the developed PIM application with the Outlook calendar. This is important because it shows they had no problems entering an appointment in one application, and having information displayed in another application (the prototype). In the evaluation exercise, generation of information was tightly connected with the actual task at hand and participants were asked to judge whether or not they found the application useful (question six). Our results show that all 15 users involved in the evaluation thought the

Human Computer Interaction with a PIM Application

37

application gave useful value. This would indicate that the reminder system is an application with practical use for the users. One other side of usefulness is the behaviour of the device and the application. Therefore each was asked to evaluate these parameters as well in question six. As a final question, after the test, the users were asked to state whether or not they would like to use this application in their daily life. As the t-test shows, the results for this question are statistically significant and one should note this indicates that the users found value in using the application in a daily life.

6 Concluding Remarks Context and context-awareness have long been acknowledged as important and have generated considerable research effort. However, integration into PIMs has so far been limited and the perspective has often been single-dimensional. In this article, the main aspects of the design, implementation and evaluation of an application prototype which integrates context / context-awareness into a PIM from a novel threedimensional perspective combining social-, geographical- and activity information have been presented. User evaluation of the proof of concept displayed a strong positive bias, highlighting its potential usefulness and applicability. Based on the developed prototype, we have shown the viability and usefulness of our approach and we do believe that tailoring information in the manner described in this paper takes the PIM concept one step further towards the ideal of providing tailored and timely information to mobile information users everywhere.

References 1. Bilandzic, M., Foth, M., Luca, A.: CityFlocks: Designing Social Navigation for Urban Mobile Information Systems. In: Proceedings ACM Designing Interactive Systems (2008) 2. Dey, A.K., Abowd, G.: Towards a Better Understanding of Context and ContextAwareness. In: Gellersen, H.-W. (ed.) HUC 1999. LNCS, vol. 1707, p. 304. Springer, Heidelberg (1999) 3. Dourish, P.: What we talk about when we talk about context. Journal of Personal and Ubiquitous Computing 8, 19–30 (2004) 4. Edwards, K.: Putting Computing in Context. ACM Transactions on Computer-Human Interaction 12(4), 446–474 (2005) 5. Hertzog, P., Torrens, M.: Context-aware Mobile Assistants for Optimal Interaction: a Prototype for Supporting the Business Traveler. In: Proceedings of the 9th international conference on Intelligent User Interfaces, pp. 256–258 (2004) 6. HTC, The HTC 3600 (2007), http://www.htc.com/product/03-product_p3600.htm (June 1, 2007) 7. Kaasinen, E.: User needs for location-aware mobile services. ACM Personal and Ubiquitous Computing 7(1), 70–79 (2003) 8. Ludford, P., Rankowski, D., Reily, K., Wilms, K., Terveen, L.: Because I carry my cell phone anyway: functional location-based reminder applications. In: Proceedings of Conference on Human Factors in Computing Systems, April 2006, pp. 889–898 (2006)

38

T.-M. Grønli and G. Ghinea

9. Prekop, P., And Burnett, M.: Activities, Context and Ubiquitous Computing. Journal of Computer Communications, Special Issue on Ubiquitous Computing 26, 1168–1176 (2003) 10. Rodden, T., Cheverest, K., Davies, K., Dix, A.: Exploiting context in HCI design for mobile systems. In: Workshop on Human Computer Interaction with Mobile Devices (1998), http://www.dcs.gla.ac.uk/~johnson/papers/mobile/HCIMD1.html 11. Zhou, M., Houck, K., Pan, S., Shaw, J., Aggarwal, V., Wen, Z.: Enabling ContextSensitive Information Seeking. In: Conference of Intelligent User Interfaces, January / February 2006, pp. 116–123 (2006)

CLURD: A New Character-Inputting System Using One 5-Way Key Module Hyunjin Ji1 and Taeyong Kim2 1 CLURD, 211-903, Hyundai Apt. Guui 3-dong, Kwangjin-gu, Seoul, Korea 2 School of Journalism & Communication, Kyunghee University, Hoegi-dong, Dongdaemun-gu, Seoul, Korea [email protected], [email protected]

Abstract. A character inputting system using one 5-way key module has been developed for use in mobile devices such as cell phones, MP3 players, navigation systems, and remote controllers. All Korean and English alphabet characters are assembled by two key clicks, and because the five keys are adjacent to each other and the user does not have to monitor his/her finger movements while typing, the speed of generating characters can be extremely high and its convenience is also remarkable. Keywords: Character Input, Typing, 5-way Key Module, Mobile Device, Keyboard, Wearable Computer.

1 Background People use cell phones, MP3 players, navigation systems, and remote controllers almost every day. Since these devices have to be small enough to hold and carry, only a limited number of keys can be installed in them. A challenge rises in that we have an increasing level of need to input text data using these devices. Therefore, with a few exceptions like Blackberry, which has 26 separate keys assigned for 26 English alphabet characters, device manufacturers has employed various methods that make possible for the users to input all characters conveniently with a small number of keys. The oldest and most popular method should be the one we have in our telephones. This is a sort of 'toggle' method, in which each numerical key from '2' to '9' is made to correspond to 3-4 character keys, so that a desired character is selected and inputted. Fig. 1 is the key layout of this method.

1 2 ABC 4 GHI 5 JKL 7 PQRS 8 TUV 0

3 DEF 6 MNO 9 WXYZ

Fig. 1. This may be the most widely applied character inputting system as it is installed in most of the traditional telephones J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 39–47, 2009. © Springer-Verlag Berlin Heidelberg 2009

40

H. Ji and T. Kim

A serious limitation of this method is experienced when a user tries to input a word like ‘feed.’ Since the characters ‘d,’ ‘e,’ ‘f’ are assigned to the same key, the user has to either wait awhile or input a splitter signal in order to input ‘e’ after ‘f.’ It is because the phone does not understand whether the user attempts to input a new character, which happens to be assigned to the same key, or change the inputted character to the next character assigned to the same key. In Korea, since the 'toggle' method requires too many times of manipulation when characters are inputted, to thus consume much time, other methods for inputting characters have been developed. The 'chun-ji-in' method and the 'naragul' method are two well-known examples. However, even with these widely-adopted methods in Korea, complicated and burdensome key manipulations are needed for inputting, and therefore characters cannot be speedily inputted. A source of the problem is that, like the old method of inputting English alphabet characters mentioned above, the two methods use 11 or 12 keys that are spread widely over the surface of the device. Another is that, in case of chun-ji-in, a splitter is required for the same reason that the old English inputting method is inconvenient, and in case of naragul, two modifier keys have to be used very frequently to change an inputted character into others that have the same root. Because of these limitations, a user has to move his/her fingers around the 11 or 12 keys busily and click as many as three keys located apart to input one character.

2 The CLURD System In order to overcome the limitations of the methods mentioned above, a new method of inputting characters using a 5-way key module has been developed and named “CLURD,” which stands for Center-Left-Up-Right-Down. 5-way key module comprises a center key disposed in the center, and an upper key, a lower key, a left key and a right key which are disposed in the top, bottom, left and right sides of the center key, respectively. Typical examples of the module are shown in Fig. 2.

Fig. 2. Typical 5-way key modules that can adopt the CLURD system

Even though there are only five keys in the key module, the CLURD system does not use the ‘toggle’ method. Instead, the system makes it a rule to assemble a character with two key clicks. Theoretically, a total of 25 combinations can be created with two clicks of five keys(5x5). If these 25 combinations are assigned to 24 Korean characters and a space datum respectively, it should be feasible to input Korean words

CLURD: A New Character-Inputting System Using One 5-Way Key Module

41

with no additional splitter or modifier keys. The combinations for the 24 Korean characters are illustrated in Figs. 3 & 4. The only unassigned combination is CenterCenter, and this is used to input a space datum. Incidentally, the data generator turns

ㄱ ㄷ ㅂ ㅅ

ㅈ

ㄲ ㄸ ㅃ ㅆ

ㅉ

‘ ’, ‘ ,’ ‘ ,’ ‘ ,’and ‘ ’ into ' ', ' ', ' ', ' ', and ' ,' respectively when the second key is pressed longer than a predetermined time. As shown in Figs. 3 & 4, the combinations are not randomly assigned to characters. Rather, the combination for each character is determined based on the geometric shapes of the keys. (See the first round-shape key module in Fig. 2.) The shapes of five keys are apparently different, and if the shapes of the two keys to be clicked are combined, they should resemble the shape of each of the 24 Korean characters fairly closely.

Fig. 3. CLURD Combinations for 10 Korean Vowels

Fig. 4. CLURD Combinations for 14 Korean Consonants

42

H. Ji and T. Kim

The CLURD system works for English alphabet characters in the same logic. As in the case of Korean, the combinations are thoughtfully assigned to characters so that they resemble the shapes of the characters as closely as possible. As shown in Fig. 5, the data generator assembles 'n', 'D', 'U', and 'C' among the English alphabetical characters if a signal generated by clicking each key combination of Up-Center, RightCenter, Down-Center, and Left-Center is input. The data generator assembles 'm', 'B', 'W', and 'E' if a signal generated by clicking each key combination of Up-Up, RightRight, Down-Down, and Left-Left is input.

Fig. 5. CLURD Combinations for English characters – Group 1

Fig. 6. CLURD Combinations for English characters – Group 2

CLURD: A New Character-Inputting System Using One 5-Way Key Module

43

Fig. 7. CLURD Combinations for English Characters – Group3

The Figs. 6 and 7 illustrate the key combinations of 'A', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'O', 'P', 'Q', 'R', 'S', 'T', 'X' and 'Z'. The data generator assembles 'V' and 'Y' as it does 'U' and 'T', respectively, however the second key should be pressed longer than a predetermined time. In the case that the language mode is set to be the Chinese Pinyin, ' ' is generated instead of 'V'.

Ü

3 Functionality and Usability of the CLURD System The CLURD system is compared with the existing methods, that is, the 'chun-ji-in' Korean character inputting method which is used in mobile phone terminals produced in Samsung Electronics Co., Ltd., the 'naragul' Korean character inputting method which is used in mobile phone terminals produced in LG Electronics Co., Ltd., and the Korean character inputting method which is used in mobile phone terminals produced in SK Telecom Co., Ltd., and illustrated in Table 1. In Table 1, the numeric values are theoretical input time expressed in units of milliseconds (ms; 1/1000 seconds). The numeric values in the parentheses means the number of the input typographical hits. These data in the specification have been Table 1. Theoretical comparison with the existing methods in light of typing speed and number of key clicks

Sentence to be input

사랑해 지금 전화해줘 늦을 것 같으니까 조금만 기다려

Samsung

LG

SK

CLURD

8975(13) 19800(28)

6740(9) 19070(27)

6805(10) 18935(24)

6480(16) 14580(36)

40115(54)

36435(46)

37350(50)

29970(74)

44

H. Ji and T. Kim

cited from the paper of Kim S., Kim K., Myung R. [1], and the data of CLURD are calculated in the same manner by the present authors. The CLURD system uses five keys that are adjacent to one another and located below a thumb of the user. Thus, a time consumed for moving fingers is greatly reduced compared with the existing methods, and the user’s gaze needs not to be shifted between the screen and keys because the user can easily locate five adjacent keys forming a ‘+’ shape located under his/her thumb. Thus, although the number of typographical hits increases by 50%, the total input time greatly decreases in comparison with the existing methods. Here, since the above-indicated times are theoretical input times calculated by a formula, and are presented for a purpose of comparison, an input can be performed within a shorter or longer time in all systems described. Also, the CLURD system has a merit in that the user is capable of inputting characters with the same hand holding his/her phone. The CLURD system can be used, though some level of caution is necessary for a safety reason, even when the user walks, drives a car, or puts the hand taking hold of the mobile device under table or in his/her pocket. The CLURD system does not have the kinds of problems that the existing methods have. That is, it does not need a splitter or a modifier key. The users only have to recall the combinations and input in the device using the key module. Also, the fact that the CLURD system has the space datum included in the key combinations improves its usability, according to the testers who experienced the system for a few days.

4 Limitation As the CLURD system works based on the predetermined key combinations for Korean and English characters, the users need to invest some time and efforts to Table 2. Results of memory test using an instructional animation (Korean)

CLURD: A New Character-Inputting System Using One 5-Way Key Module

45

memorize the combinations. Thanks to the fact that the combinations are not random, but associated with the shapes of the characters fairly closely, the memory burden seems not significant. Table 2 is the results of a memory test with junior high school students.

5 Augmentation of the CLURD System For the case in which the user of the CLURD system decides not to monitor the screen while typing for any reason, it should be better to add some sort of function notifying him/her in the case in which an error has occurred. Because the user’s vision is likely to have been allocated to another task in such a situation, it should be wise to utilize another sensory modality. A short vibration may be a good idea in case of a cell phone. Also, if the CLURD system is used as the main communicating device by speech-handicapped persons, it can be combined with a text to speech (TTS) module which converts texts into sounds.

6 Devices Using the CLURD System In the case of cell phone, PDA or wearable computer, the CLURD system can be used as a substitute for a regular keypad/keyboard. In the case of the remote controller for a digital TV set, the CLURD system is used for the purpose of searching program titles, inputting identification and password, or writing in the user’s address to order a product. Some examples of devices(cell phone, TV remote, door lock, typing mouse) are shown in Fig. 8. Other examples, though not shown, may include watch phones, wearable computers, and so on. For watch phone, the 5-way key module can be installed on the surface of the window in the form of transparent thermal touch pad. A watch phone can be used as an inputting device for

Fig. 8. Cell phone, remote controller for digital TV set, door lock, typing mouse with the CLURD system installed (Conceptual illustrations or prototype models)

46

H. Ji and T. Kim

Fig. 9. The on-screen keypad and the palm-size full-function keyboard with the CLURD system installed (Available in market); The palm-size keyboard was developed with a financial support from the Korea Agency for Digital Opportunity and Promotion

wearable computers with a wireless connection to the CPU. The on-screen keypad program and the palm-size keyboard shown in Fig. 9 are the first official products in market (introduced in April 2009) that adopt the CLURD system.

7 Conclusion As described above, CLURD is a character inputting system suitable for a mobile device where a sufficient number of keys cannot be arranged due to space restriction. Besides the fact that this character inputting system takes up little space on the surface of a device, the system has remarkable edges that other existing methods cannot materialize. First, it requires only one or two fingers for operation, and this allows the user to allocate the other hand and/or the rest of fingers to other concurrent tasks. This also means that the physically handicapped people who can use only one hand or one finger can type characters speedily. In fact, a Korean lady who is partly paralyzed and thus can use only one hand has passed an official computer literacy test that requires a fair speed of typing only with this palm-size keyboard. Another edge of the CLURD system is that the user does not have to look at the keys while typing since his/her finger knows where the five keys are located. This allows him/her to monitor whether he/she makes any typo by fixing his/her gaze to the screen. The CLURD system is also useful to the blind, especially when they take a note reading a book in Braille. They may read a book with left hand and type in a computer using the CLURD keyboard. (Taking a note on paper is meaningless for them because they cannot read it later.) A possible application for the mute is a hand-held device in which the CLURD system is installed together with a TTS system so that when the user presses the TTS button the device reads what he/she has just typed in the device using the CLURD system. The CLURD system can be nicely installed in the form of a virtual keypad in touch-screen interface. This has been confirmed with the on-screen keypad program designed for tablet computers and the prototype model of typing mouse. The time and efforts needed to get familiar with the key combinations may be the only limitation with the CLURD system. A good sign was that most of the testers (over 2,000 people) who have used add-on software designed for cell phone stated

CLURD: A New Character-Inputting System Using One 5-Way Key Module

47

that the time and efforts needed for learning the combinations were quite minimal and its functionality and usability were remarkable enough for them to willingly make such an investment.

Reference 1. Kim, S., Kim, K., Myung, R.: Hangul input system’s physical interface evaluation model for mobile phone. Journal of Korean Institute of Industrial Engineers 28(2), 193–200 (2002)

Menu Design in Cell Phones: Use of 3D Menus Kyungdoh Kim1, Robert W. Proctor2, and Gavriel Salvendy1,3 1

School of Industrial Engineering, Purdue University, 315 N. Grant St., West Lafayette, IN 47907 USA [email protected] 2 Department of Psychological Sciences, Purdue University, 703 Third St., West Lafayette, IN 47907 USA [email protected] 3 Department of Industrial Engineering, Tsinghua University, Beijing, P.R. China [email protected]

Abstract. The number of mobile phone users has been steadily increasing due to the development of microtechnology and human needs for ubiquitous communication. Menu design features play a significant role in cell phone design from the perspective of customer satisfaction. Moreover, small screens of the type used on mobile phones are limited in the amount of available space. Therefore, it is important to obtain good menu design. Review of previous menu design studies for human-computer interaction suggests that design guidelines for mobile phones need to be reappraised, especially 3D display features. We propose a conceptual model for cell phone menu design with 3D displays. The three main factors included in the model are: the number of items, task complexity, and task type. Keywords: cell phones, menu design, 3D menu, task complexity, task type.

1 Introduction The number of mobile phone users has been steadily increasing due to the development of microtechnology and human needs for ubiquitous communication. People use mobile phones to communicate with their friends, family, and business partners, and also to obtain information through the mobile Internet. Moreover, people use embedded mobile phone features such as games, cameras and wireless Internet for various purposes of entertainment and shopping. Due to increasing features, mental workload of using cell phones has increased. Ling et al. [1] prioritized the design features and aspects of cell phones based on users’ feedback to optimize customers’ satisfaction. Although physical appearance and body color of cell phones had considerable influence on overall user satisfaction, menu design features also played a significant role. Therefore, obtaining a good menu design in cell phones is an important issue. There has been a lot of research about menu design for computers. When it comes to menu dimensions, many researchers have concluded that performance time and errors increase as the hierarchical levels of the menu structure increase [2, 3]. With J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 48–57, 2009. © Springer-Verlag Berlin Heidelberg 2009

Menu Design in Cell Phones: Use of 3D Menus

49

regard to menu type, hierarchical menus are more accurate and faster than fisheye menus [4]. Three-dimensional (3D) displays show many items of a menu at the same time, so they may give the same effect as a broader menu [5]. With regard to adaptability, computer menus that can be customized by users have been shown to be better than ones that adapt automatically [6]. Research on cell phone menu design is relatively recent. Geven, Sefelin, and Tscheligi [7] concluded that narrow hierarchies performed better than broader hierarchies in mobile devices, contrary to menu design in computers. With respect to menu type, Gutwin and Fedak [8] found that people were able to carry out a web navigation task better with the fisheye view than with alternatives. For adaptability, results have been similar to those for computer displays. Customized menus produced better performance and evaluation than the traditional static menu [9]. But, a lack of studies about 3D display for cell phones was found, and in this paper 3D design research is investigated in more detail. At this point, there are no standard interaction devices or interfaces used in 3D environments, and there is a lack of specific best practice guidelines to develop these 3D designs. 3D design is able to convey more information than text or twodimensional (2D) images, and it enhances the usability of the limited screen on a typical wireless device. Interactive 3D can therefore be used to remove some of the complexity and clutter present on menu systems of today’s handsets. 3D icons can be animated to show activity or changes in status, and the depth dimension can be utilized to show urgency or relative importance [10]. Therefore, new standards should be developed to allow personal digital assistants (PDAs) and mobile devices to render 3D applications. Review of previous menu design studies for human-computer interaction suggests that design guidelines for mobile phones need to be reappraised, especially 3D display features. To this end, the main objective of this paper is to propose an overall framework to develop mobile phone menu design guidelines regarding 3D displays. We review menu design components for computers in section 2 and investigate menu design factors for cell phones in section 3. Strengths and weaknesses of 3D design factors are considered in section 4. We compare menu design factors in section 5 and conclude after explaining a model of cell phone menu design in section 6.

2 Menu Design in Computers 2.1 Menu Dimension Many of the early studies of menu design for computers focused on the cognitive factors of a menu’s hierarchical structure and the structure’s impact on end users’ behaviors and performance in retrieving information. Out of this research, studies about whether it is better to have a broad or deep design have been conducted. Jacko et al. [2] suggested three components of hierarchical menu design: menu dimension, task complexity, and user knowledge structure. The results about the menu dimension supported that both performance time and errors increased as the levels of the menu structure increased. That is, depth in an information structure increases the likelihood of navigational errors and also increases performance time [11].

50

K. Kim, R.W. Proctor, and G. Salvendy

Seppala and Salvendy [3] also drew the conclusion that a broader mode of data presentation is more effective than a deeper one. Because searching back and forth through the menu system decreases the speed and accuracy of performance, the broader menu has better performance in the case of a personal computer. This is because increased depth involves additional visual search and decision-making, and greater uncertainty as to the location of target items due to the increased number of menu frames [12]. In other words, as the depth increases and the number of responses needed while going through a menu tree increases, more time for decision making and responding is required [3]. 2.2 Menu Type Menu structure can be classified as hierarchical and fisheye [4]. Fisheye is a menu display method that shows a region of the menu at high magnification, while items before and after that region are shown at gradually reduced sizes. Hornbaek and Hertzum [4] provided evidence that, for finding known items, conventional hierarchical menus were more accurate and faster than fisheye menus. Also, participants rated hierarchical menus as more satisfying than fisheye menus. For browsing tasks, the menus did not differ with respect to accuracy or selection time. Fisheye interfaces have an advantage in that they can accommodate many menu items in a limited amount of screen space by showing part of an information space at high magnification, while other parts are shown at low magnification to provide context. However, performance remained worse with fisheye menus than with hierarchical menus because the latter impose lower mental demands on users [4]. Within a hierarchical menu, cascading and indexed menus can be compared [13]. Participants searched three types of menu layouts: categorical index; horizontal cascading; vertical cascading. Search time differences between the three menu layouts were detected that strongly favored the index menu. One possible reason for this result is that the items in the index menus were in closer proximity. Another is that the index menus were centrally located on the screen, and thus would have been easier to see and acquire. 2.3 Adaptability Some commercial applications now have adaptable interfaces. For example, the Start Menu in Microsoft Windows XPTM has an adaptive function that provides automatically generated shortcuts to frequently used applications. Microsoft Office also provides Smart Menus, which are an adaptive mechanism where infrequently used menu items are hidden from view. Understanding these interfaces through strong empirical and theoretical studies is particularly important, because adaptive interfaces are now being introduced into productivity software and used by an increasing number of people [14]. Mitchell and Shneiderman [15] compared dynamic vs. static menus using a menu-driven computer program. Subjects who used adaptive dynamic menus for the first set of tasks were significantly slower than those who used static menus. Moreover, 81% of the subjects preferred working with static menus to working with dynamic menus. This preference

Menu Design in Cell Phones: Use of 3D Menus

51

likely is because dynamic menus can slow down first-time users, at least until they become accustomed to this interaction style. Findlater and McGrenere [6] compared the measured and perceived efficiency of three menu conditions: static, adaptable and adaptive. They found that users generally preferred the customizable version to the adaptive menus. In terms of performance, adaptive menus were not faster than either of the other conditions. User-driven customization is a more viable approach for personalizing user interfaces than systemdriven adaptation. The static menu was found to be significantly faster than the adaptive menu, and the adaptable menu was found to be significantly faster than adaptive menu under certain conditions. But, in terms of accuracy, there were no differences. However, the majority of users preferred the adaptable menu overall and ranked it first for perceived efficiency. Therefore, this study suggests that system-driven adaptation is not helpful.

3 Menu Design in Cell Phones 3.1 Menu Dimension As screens become smaller, the information they display changes more extensively with each scrolling action, making it more difficult to refocus on the page. In this way, screen size affects the navigation behavior and perceptions of mobile phone users [11]. Therefore, the breadth of information structures should be adapted to anticipated screen size. The advantage of depth is that it encourages funneling; the disadvantage is that it induces errors and increases the number of page transactions. On the other hand, the advantage of breadth is that it reduces navigation errors and the number of page transactions; the disadvantage is that it leads to crowding. Therefore, a user encountering greater depth has fewer options to process on a single page. Thus, the cognitive load on the user is reduced. Findings consistently have suggested an advantage of employing a deeper menu structure to achieve better user performance and accuracy. Geven et al. [7] showed that people perform better with narrow hierarchies than with broader hierarchies on small screens. Contrary to computers, where many options are usually presented at once, it is better to use a layered design in cell phones. Huang [16] showed that users prefer a less extensive menu structure on a smallscreen device. This result supports the recommendation of not having a broad menu structure on a small screen. With less space to display information, designers of cell phones tend to chunk menu items of a broader menu into several pages or screens. This chunking requires end-users to employ more scrolling operations, maintain more information in working memory, and engage in more searching and navigation behaviors. The consequence is to reduce the speed and accuracy in use of the menus. The following describes the two suggestions that Huang [16] developed: (1) Reduce both breadth and depth of the menu. (2) Instead of displaying only a limited number of items on one screen, include more menu items and options in one page.

52

K. Kim, R.W. Proctor, and G. Salvendy

Dawkins [9] also suggests that filling the screen as much as possible without requiring scrolling should be the ideal breadth of the menu. 3.2 Menu Type and Adaptability Many of the current visualization methods aimed at small screens rely on distorting the view. The viewpoint information is manipulated in a way that enables seeing important objects in detail, and the whole information space can be displayed at once with very low amount of detail [17]. The rubber sheet is one of view distortion techniques that allow the user to choose areas on the screen to be enlarged. Zooming and zoomable user interfaces (ZUI) are another way of presenting large information spaces even on a small screen. Combs and Bederson [18] studied image browsers and found that their system, based on a ZUI method (as well as 2D thumbnail grid), outperformed 3D browsers in terms of retrieval time and error rate. Displaying the overview and the detail at the same time is also more beneficial than the traditional linear format because the global context allows faster navigation [8]. Gutwin and Fedak [8] found that people were able to carry out a web navigation task better with the fisheye view. Some phones are already being designed with a fisheye display for selected items to be salient and clear. Therefore, a fisheye menu may be better than a 2D hierarchical menu. In computers, users can create folders, reorder the layout, and make shortcuts. But a mobile phone has limited screen size and a small input device. Moreover, telecommunication carriers want the buttons to be used for their wireless Internet service. They are therefore reluctant to offer many customization functions to users. In other words, mobile phones do not provide enough adaptation functions. Dawkins [9] evaluated personalized menus alongside a traditional static menu structure based on user preference and performance. He concluded that customized menus had better performance and evaluation than the traditional static menu. Therefore, customers seem to want more customization functions in their cell phones from the perspectives of performance and satisfaction.

4 3D Design 4.1 Benefits of 3D Design Human information-processing has evolved to recognize and interact with a 3D world. And the 3D design space is richer than the 2D design space, because a 2D space is part of 3D space. It is always possible to flatten out part of a 3D display and represent it in 2D [19]. Therefore, it is unsurprising that 2D interfaces have performed relatively poorly. For example, Ware and Franck [20] conducted an experiment that was designed to provide quantitative measurements of how much more (or less) can be understood in 3D than 2D. Results showed that the 2D interface was outperformed by 3D interfaces. These results provide strong reasons for using advanced 3D graphics for interacting with a large variety of information structures [20].

Menu Design in Cell Phones: Use of 3D Menus

53

The 3D interfaces make it possible to display more information without incurring additional cognitive load, because of pre-attentive processing of perspective views (e.g., smaller size indicates spatial relations at a distance). An ability to recognize spatial relations based on 3D depth cues makes it possible to place pages at a distance (thereby using less screen space) and understand their spatial relations without effort [21]. As described before, there are many 3D depth cues that can be provided to facilitate spatial cognition. The most obvious of these are perspective view and occlusion. Using these cues, the user gets the advantages of a 3D environment (better use of space, spatial relations perceived at low cognitive overhead, etc.). 3D allows larger menu items than the screen size. This would be a desirable feature for small screens that have a restricted screen resolution and size [22]. The effect of 3D is to increase the effective density of the screen space in the sense that the same amount of screen can hold more objects, which the user can zoom into or animate into view in a short time. It seems reasonable that 3D can be used to maximize effective use of screen space [23], especially in cell phones for which the screens are small screens. The use of 3D models on the Internet is gaining popularity, and the number of 3D model databases is increasing rapidly because 3D interfaces enable a more natural and intuitive style of interaction [24]. Since the use of 3D models is becoming more common on various cellular phone web sites, development of algorithms that retrieve similar information will be important in cell phone menu design [25]. 4.2 Weaknesses of 3D Design Creating a 3D visualization environment is considerably more difficult than creating a 2D system with similar capabilities. As the study of Cockburn and McKenzie [26] suggests, one should not assume that use of 3D provides more readily accessible information. In determining whether to implement a 3D display, designers should decide whether there are enough subtasks that would benefit from 3D representations. The complexity and the consistency of the user interface for the whole application must also be weighed in the decision. In the study of Ware [19], 3D navigation methods took considerably longer than 2D alternatives. Even if somewhat more information can be shown in 3D than in 2D, the rate of information access may be slower, and 3D applications may have greater visual complexity than 2D applications [27]. People often find it difficult to understand 3D spaces and to perform actions in them. It is clear that simply adapting traditional WIMP (windows, icons, menus, and pointers) interaction styles to 3D does not provide a complete solution to this problem. Rather, novel 3D user interfaces, based on interactions with the physical world, must be developed. Jones and Dumais [28] have suggested that little significant value is provided by adding physical location information to the storage and subsequent retrieval of a document over and above simply providing a semantic label for the same purposes. 4.3 Direct Comparison between 2D and 3D Few prior studies have directly compared 2D and 3D interactive systems. Also, there is a surprising lack of empirical research into the benefits (or costs) that are produced

54

K. Kim, R.W. Proctor, and G. Salvendy

by moving from 2D to 3D. Cockburn and McKenzie [29] compared subject’s efficiency in locating files when using Cone-Trees (a 3D technique for exploring hierarchical data structures) and when using a ‘normal’ folding tree interface similar to that used in Windows Explorer. Results showed that the subjects took longer to complete their tasks when using the cone interface. They rated the cone interface as poorer than the normal one for seeing and interacting with the data structure. Also, Cockburn and McKenzie [26] showed no significant difference between task performance in 2D and 3D, but a significant preference for the 3D interfaces. Recently there has been a growth of interest in 3D interactive systems for everyday ‘desktop’ computing applications, such as document and file management. However, the relative value of the third visual dimension in cell phone menu design has not previously been evaluated.

5 Models for Cell Phone Menu Design Jacko et al. [2] proposed modifications to an information-processing model developed by Salvendy and Knight [30]. In this model, three constructs of hierarchical menu retrieval were proposed: menu dimension, task complexity, and knowledge structure. Figure 1 illustrates a version of Jacko et al.’s [2] information-processing model extended to cell phone menu retrieval operation. The model takes advantage of the natural 3D human information-processing capabilities for cell phone menu interfaces, with distinctions similar to those identified by Jacko et al. The three main factors for cell phone menu design within 3D display included in the model are: the number of items, task complexity, and task type. Cell phones support more features such as broadcasting, mobile wallet and health condition sensor, etc. This is consistent with an issue raised by Norman [31], which is “a tendency to add to the number of features that a device can do, often extending the number beyond all reasons” (p. 173). With human cognitive limitations, a cell phone with too many features may overwhelm users due to its complexity [1]. Under these circumstances, it is important to investigate how the number of items can influence 3D menu design in cell phones. The number of items could influence menu dimensions, resulting in effects on perception, cognition, and motor response time. In this way, the number of items is an important characteristic of a virtual menu that will influence the item selection time. Moreover, inclusion of many menu items may decrease the usability of a 2D display solution. Therefore, deciding whether or not to use 3D design should depend on the number of items per menu screen. Task complexity can impact performance and satisfaction of 3D menu design because in a 3D environment the spatial relationships are perceived at low cognitive overhead [22]. Thus, performing a complex task may be better in a 3D environment than in a 2D environment. On the other hand, a 3D display sometimes has greater visual complexity. Therefore, direct comparisons between 2D and 3D menus for different levels of task complexity are needed. Task type influences the perceptual information required, the cognition operations involved in using that information, and necessary motor responses. Experiments need to be conducted to validate the proposed conceptual model.

Menu Design in Cell Phones: Use of 3D Menus

55

Fig. 1. Modified Information-processing Model for Cell Phone Menu Operation

6 Conclusion The widespread use of cell phones for a variety of purposes provides evidence that they are shifting from just a communication tool to being an integral part of people’s everyday life. It is important to study cell phone menu design because, though menu design plays a crucial role in cell phone usability, little work exists on developing cell phone menu design. Three factors were identified that may influence performance of menu retrieval tasks with 2D and 3D displays in cell phones: the number of items, task complexity, and the type of tasks. These three factors are included in the proposed conceptual model for cell phone menu design with 3D displays. Research designed to validate this model should provide insights into the human information-processing requirements of various cell phone menu interfaces.

References 1. Ling, C., Hwang, W., Salvendy, G.: A survey of what customers want in a cell phone design. Behaviour & Information Technology 26, 149–163 (2007) 2. Jacko, J.A., Salvendy, G., Koubek, R.J.: Modelling of menu design in computerized work. Interacting with Computers 7, 304–330 (1995) 3. Seppala, P., Salvendy, G.: Impact of depth of menu hierarchy on performance effectiveness in a supervisory task: computerized flexible manufacturing system. Human Factors 27, 713–722 (1985)

56

K. Kim, R.W. Proctor, and G. Salvendy

4. Hornbaek, K., Hertzum, M.: Untangling the usability of fisheye menus. ACM Trans. on Computer-Human Interaction, Article 6, 1–32 (2007) 5. Dachselt, R., Ebert, J.: Collapsible cylindrical trees: a fast hierarchical navigation technique. In: Information Visualization, INFOVIS 2001, pp. 79–86 (2001) 6. Findlater, L., McGrenere, J.: A comparison of static, adaptive, and adaptable menus. In: Proceedings of the 2004 conference on Human Factors in Computing Systems, pp. 89–96 (2004) 7. Geven, A., Sefelin, R., Tscheligi, M.: Depth and breadth away from the desktop: the optimal information hierarchy for mobile use. In: Proceedings of the 8th conference on Human-Computer Interaction with Mobile Devices and Services, pp. 157–164 (2006) 8. Gutwin, C., Fedak, C.: Interacting with big interfaces on small screens: a comparison of fisheye, zoom, and panning techniques. In: Proceedings of the 2004 conference on Graphics Interface, pp. 145–152 (2004) 9. Dawkins, A.L.: Personalized Hierarchical Menu Organization for Mobile Device Users. Vol. Master. North Carolina (2007) 10. Beardow, P.: Enabling Wireless Interactive 3D. article retrieved from the Superscape Plc (June 2004), http://www.superscape.comin 11. Chae, M., Kim, J.: Do size and structure matter to mobile users? An empirical study of the effects of screen size, information structure, and task complexity on user activities with standard web phones. Behaviour and Information Technology 23, 165–181 (2004) 12. Jacko, J.A., Salvendy, G.: Hierarchical menu design: Breadth, depth, and task complexity. Perceptual and Motor Skills 82, 1187–1201 (1996) 13. Bernard, M., Hamblin, C.: Cascading versus Indexed Menu Design. Usability News 5 (2003) 14. Gajos, K.Z., Czerwinski, M., Tan, D.S., Weld, D.S.: Exploring the design space for adaptive graphical user interfaces. In: Proceedings of the working conference on Advanced Visual Interfaces, pp. 201–208 (2006) 15. Mitchell, J., Shneiderman, B.: Dynamic versus static menus: an exploratory comparison. ACM SIGCHI Bulletin 20, 33–37 (1989) 16. Huang, S.C.: Empirical Evaluation of a Popular Cellular Phone’s Menu System: Theory Meets Practice. Journal of Usability Studies, 136–150 (2006) 17. Hakala, T., Lehikoinen, J., Aaltonen, A.: Spatial interactive visualization on small screen. In: Proceedings of the 7th international conference on Human Computer Interaction with Mobile Devices & Services, pp. 137–144 (2005) 18. Combs, T.T.A., Bederson, B.B.: Does Zooming Improve Image Browsing? In: Proceedings of the fourth ACM Conference on Digital Libraries, pp. 130–137 (1999) 19. Ware, C.: Information Visualization: Perception for Design. Morgan Kaufmann, San Francisco (2004) 20. Ware, C., Franck, G.: Evaluating stereo and motion cues for visualizing information nets in three dimensions. ACM Transactions on Graphics 15, 121–140 (1996) 21. Robertson, C.M., Larson, K., Robbins, D.C., Thiel, D., van Dantzich, M.: Data mountain: using spatial memory for document management. In: Proceedings of the 11th annual ACM Symposium on User Interface Software and Technology, pp. 153–162 (1998) 22. Rekimoto, J.: Tilting operations for small screen interfaces. In: Proceedings of the 9th annual ACM Symposium on User Interface Software and Technology, pp. 167–168 (1996) 23. Robertson, C.S.K., Mackinlay, J.D.: Information visualization using 3D interactive animation. Communications of the ACM 36, 57–71 (1993)

Menu Design in Cell Phones: Use of 3D Menus

57

24. Molina, J.P., Gonzalez, P., Lozano, M.D., Montero, F., Lopez-Jaquero, V.: Bridging the Gap: Developing 2D and 3D User Interfaces with the IDEAS Methodology. In: Jorge, J.A., Jardim Nunes, N., Falcão e Cunha, J. (eds.) DSV-IS 2003. LNCS, vol. 2844, pp. 303–315. Springer, Heidelberg (2003) 25. Suzuki, M.T., Yaginuma, Y., Sugimoto, Y.Y.: A 3D model retrieval system for cellular phones. In: IEEE International Conference on Systems, Man and Cybernetics, vol. 4, pp. 3846–3851 (2003) 26. Cockburn, A., McKenzie, B.: 3D or not 3D?: evaluating the effect of the third dimension in a document management system. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 434–441 (2001) 27. van Dam, A.: Post-WIMP user interfaces. Communications of the ACM 40, 63–67 (1997) 28. Jones, W.P., Dumais, S.T.: The spatial metaphor for user interfaces: experimental tests of reference by location versus name. ACM Transactions on Information Systems 4, 42–63 (1986) 29. Cockburn, A., McKenzie, B.: An Evaluation of Cone Trees. People and Computers XIVUsability Or Else! 425–434 (2000) 30. Salvendy, G., Knight, J.: Psychomotor work capabilities. In: Salvendy, G. (ed.) Handbook of Industrial Engineering, pp. 1–5 (1982) 31. Norman, D.A.: The Psychology of Everyday Things. Basic Books, New York (1988)

Mobile Interfaces in Tangible Mnemonics Interaction Thorsten Mahler, Marc Hermann, and Michael Weber Institute of Media Informatics University of Ulm, Ulm, Germany {thorsten.mahler,marc.hermann,michael.weber}@uni-ulm.de

Abstract. The Tangible Reminder Mobile brings together tangible mnemonics with ambient displays and mobile interaction. Based on the Tangible Reminder Project we present a new interface for mobile devices that is capable of viewing and editing data linked to real world objects. An intelligent piece of furniture equipped with RFID-sensors and digitally controlled lighting keeps track of appointments linked to real world objects that are placed in its trays. The mobile interface now allows the complete waiving of classic computer interaction for this ambient shelf. Instead, by implementing the toolglas metaphor, the mobile interface can be used to edit and view linked data to objects.

1

Introduction

Mark Weiser [1] formulated in 1991 his vision of ubiquitous computing stating the goal of bringing together the virtual world and the real world in a constant intertwining. The direct result for human computer interaction is that the interaction with real life artefacts can affect the virtual representations, thus making virtual objects easily graspable. According to Holmquist et al. [2] a lot of research is done in this domain focussing on different aspects depending on the primary goal they pursue: graspable interface, tangible interface, physical interface, embodied interface, to name just a few. Whatever the name, they are all unified by their common goal, to enrich the interaction with virtual objects by physicality. One of the first projects to describe this linkage between physical objects and virtual representations is Ishii’s and Ullmer’s paper ”Tangible Bits” [3]. Based on their observation with coupling everyday objects and virtual information, ”Bits and Atoms”, they tackle an interesting question: how can physical artefacts become interfaces for virtual objects and, from our point of view even more important, how can these interfaces be recognized? Their solution for this problem is the introduction of virtual light or virtual shadow. Both concepts show the information in a glow or shadow around the physical object in question. For example the software on the new Microsoft surface [4] table makes use of this concept. However, the visualization of information into the very vicinity of real objects requires a technically very well equipped environment. The Project Urp J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 58–66, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Mobile Interfaces in Tangible Mnemonics Interaction

59

[5] uses a large display underneath the objects in order to view the virtual shadows. The SenseBoard uses a projector to display detailed information on a ”View Details” command puck when it is placed over an item on the SenseBoard [6]. The AuraOrb provides newsticker information when a eyecontact sensor detects the focus of a user on it [7]. But if neither the environment nor the object itself is equipped with displaying technology, the augmentation of the objects remain invisible. The solution for this problem is the use of other metaphors, for example the see-through metaphor or the magic lens [8] [9]. The information linked to a real object is not projected into the real world. Rather a mobile device shows the information linked directly on its screen. A mobile device also compensates the non-ubiquity of intelligent environments through mobility [10]. In the Tangible Reminder Project we now make use of this technique. The next section gives an overview of the first prototype of the Tangible Reminder. After that we will focus on the concept of the mobile interface and finally present the whole system with this new interface.

2

The Tangible Reminder

The Tangible Reminder is a device combining the benefits of ambient displays and tangible interaction with personal objects. It is designed as a tool to keep track of appointments and deadlines and particularly to remind the user on upcoming events. As shown in Fig. 1. the ambient display subsystem in our prototype consist of three trays in which freely chosen RFID-tagged objects can be placed. According to the urgency of the appointment linked to every object the trays are colored green, yellow or red. Showing the appropriate color, the display remains calm, avoiding distraction from other tasks until the deadline is due. After that it begins flashing, grasping the user’s attention. [11] A main idea was to let the user choose personal objects with specific associations in the topic of the appointments. We use RFID technology to identify the objects in the trays. As RFID-Tags have gotten cheaper and smaller, a user can tag any object he wants to. Choosing a personal object makes it easier to mentally link it with appointments. A study of van den Hoven et al. showed that nearly all personal souvenirs bear a mass of information for the owner [12]. The memories associated with the object can give a clue when spotting it in the tray, a mnemonic to remind the topic of the appointment. If more information is needed, the user has to interact with the system. In our first prototype we established a non-ubiquitous way of allowing this ”easy transition to more in-depth information”, one of the heuristics both for graphical user interfaces [13] and ambient displays [14]. If additional information was desired, the object had to be removed from the tray and put on an interaction platform connected with an ordinary computer or laptop. The information was then displayed on the monitor. Also, before usage an object had to be technically linked to an appointment and additional information like addresses, images etc. The inspection system and the input system were the same in the first prototype,

60

T. Mahler, M. Hermann, and M. Weber

Fig. 1. The ambient display subsystem of the Tangible Reminder, with an object in each tray, showing the different colors for different states

so the user was able to change linked information when an object was placed on the platform. This form of inspection and input on traditional computers is not very ubiquitous, so we will present an alternative to that approach here.

3

A Concept for Mobile Interfaces for Tangible Mnemonics

One goal of tangible interaction is the masking of computer interaction by implicit actions with real objects. But this masking is also the problem for tangible interaction systems application in real life. This masking hinders the use of ready-mades, objects present in everyday life as it clouds the linkage between the real object and the virtual data. Neither the linkage itself, i.e. which data an object represents can be seen nor it is clear how this data can be edited. Both issues are solved in the original Tangible Reminder system by braking with the paradigm of implicit interaction by simply using a laptop as editing and viewing station. While this is true for objects linked to absolute appointments, the first steps to further pushing the computer in the background is taken. The introduction of relative appointments supersede the explicit computer interaction. Instead, the simple act of putting an object in the tray already triggers the

Mobile Interfaces in Tangible Mnemonics Interaction

61

appointment. For instance, the simple act of putting a special tea cup into the Tangible Reminder shelf results in an alert 3 minutes in the future. Clearly, this way of editing linked data is an improvement. Nevertheless, the visualization of the linkage and thus the display of the data still involves the computer. To overcome this drawback, Ishii and Ullmer [3] have proposed to use digital shadows or digital light. Whilst this metaphor is clearly interesting and integrates nicely into the real world, it is also very demanding as it needs a lot of special hardware and sensor integration which is only present in special rooms today. Instead we decided to tackle the challenges of displaying linkage and editing linked data via the use of mobile devices. Not only can they be easily carried to the real object in question, but they also perfectly support the see-through metaphor of Bier et al. [8] A portable device in this case acts as a magical toolglas, showing the link to a real object when placed over it. In the display the user can see the linked data and manipulate it if desired. This metaphor is easy to understand and to use. Nevertheless, the decision in favor of small and portable devices does not completely hide the computer. It rather shifts the interaction part to a small device. This device, though a complete computer as well, is perceived as being much simpler and easier to use. It does not cloud the computer as such but it clouds its complexity [15]. With the adoption of small computing devices, especially cell-phones, in time even smart-phones this approach is getting even more appealing.

4

The Tangible Reminder Mobile

The ambient display subsystem (see Fig. 1) of the original Tangible Reminder follows Weiser’s vision of implicit computer interaction and fits nicely into the surrounding. It brings together small personal objects acting as mnemonics, with a calm but also demanding interface if needed. Therefore, we decided to keep this part of the Tangible Reminder unchanged but to completely replace the input and inspection subsystem. 4.1

Interaction with the Tangible Reminder Mobile

To interact with an object in the Tangible Reminder Mobile system it is sufficient to simply put the PDA, containing the new mobile input and inspection subsystem, near an object enhanced with an RFID. Via the integrated RFIDReader the Tangible Reminder Mobile recognizes that the object nearby can be associated with virtual data. It queries the database and retrieves the stored data. This data is shown on the display pane. Fig. 2 depicts the scanning of a globe, which is in this case associated with a journey to San Diego. Besides the time additionally the reminder period is shown. The simple act of moving the Tangible Reminder Mobile near an object shows its linkage and capacities. The act of posing the Tangible Reminder Mobile over an object results in showing the virtual content of a real object as if watched through a magic lens for digital data.

62

T. Mahler, M. Hermann, and M. Weber

Fig. 2. Scanning the globe for the associated appointment

4.2

The Mobile Input and Inspection Subsystem

The subsystem on the PDA serves for two purposes: It shows the data linked to a certain object plus it is capable of editing this data or adding data if not yet stored with an object. Suitable objects contain a RFID-Chip simply sending an ID whenever a reader comes into close vicinity. With the readers we are using this reading distance is limited to about 5 cm which is very small. The reader therefor has to be posed directly over the object and its RFID to automatically raise the inspection screen. As we are dealing with appointments and reminders, the inspection screen shows the name of the appointment together with the exact date and the period after which a reminder should occur. Additionally a photograph of the associated real object can be shown. Fig. 3 shows this dialog in detail. In order to add or edit data stored with an object, the edit screen has to be opened. There all relevant data can be entered. To keep the interface usable we decided for special input fields like date-pickers to keep the pen interaction simple. The only text-field capable of free text is the appointment name. All other data can only be modified via controls. This better meets the accuracy of

Mobile Interfaces in Tangible Mnemonics Interaction

Fig. 3. The mobile device running the Tangible Reminder Mobile program, displaying the appointment view

63

Fig. 4. The appointment form allows for changing the associated appointment and the way of reminding

pen interaction on mobile devices as well as it reduces input errors. Fig. 4 shows an example for the input screen. Here the appointment linked with the globe is changed from an appointment on the 24th of December to a journey in June. 4.3

System Design and Changes Compared to the Original Tangible Reminder

To allow for the nice and simple magic-lens approach we decided to port only the input and inspection subsystem to the PDA while leaving the underlying concept unchanged to further support the display subsystem. The original Tangible Reminder makes use of RFID-Chips to recognize objects and to link them to appointments. There are different ways of making a small device capable of reading RFIDs. Instead of using an extra device, we equipped our PDA with a SD-Card-RFID-Reader. This solution integrated the Reader into the PDA and keeps the system small instead of using separated devices. The laptop in the original Tangible Reminder did not only work as an input device, it also kept the data stored for the linked objects. This data was retrieved by the display system to control the reminder functions of the shelf. The new Tangible Reminder system separates this earlier intertwining and divides the system according to their functions. The virtual data is now stored on a server

64

T. Mahler, M. Hermann, and M. Weber

that provides all domain related functions via a web-service. This service can be contacted and controlled by the display subsystem of the shelf as well as the mobile magic-lens subsystem. Thus, either subsystem has to deal only with the exact functions it has to fulfill. We rather make the virtual space underlying the real world accessible independent of environmental hardware by lending real objects a mobile interface, in our case rendering the Tangible Reminder system completely independent of traditional computer interaction.

5

Conclusion

With the Tangible Reminder Mobile system we presented a mobile interface for an ambient mnemonic system, the Tangible Reminder. This system brings together ambient displays with tangible interaction by reminding a user on appointments previously linked to everyday objects, functioning as mnemonics. To establish the linkage between real life objects and virtual data we needed a classic graphical computer interface in the original system. The Tangible Reminder Mobile System now provides an interface on a PDA making the Laptop as input device unnecessary. It thereby renders the Tangible Reminder system to an intelligent piece of furniture not recognized as computer interface. The mobile interface itself makes the usage of the Tangible Reminder more natural by implementing the magic lens metaphor. The smaller device not only makes the classic interface disappear, it also keeps the interface seemingly simpler. It combines the tools for information display and data manipulation and makes the interface mobile like the user chosen real life mnemonic objects. The interface therefor bridges the gap between the real and the virtual world and solves the problems of data manipulation and display in an elegant way. No classic computer interface is needed, just a PDA is used, a device that gets more and more common. The switch to this mobile interface is another step towards real implicit interaction with everyday objects with no computers visible, towards the vision of natural interaction with virtual data and ubiquitous computing alike.

6

Future Work

The Tangible Reminder Mobile system is just a step on the way to implicit and computer-less machine interaction. Further development is needed and planned in mobile interface improvement and in the field of implicit interaction. The integration of camera images to literally implement a see-through tool could make the interface more intuitive. However, from our point of view the benefit is marginal unless the system recognizes and marks the detected object which is a hard problem without marker usage. The next step therefor would be to recognize objects visually and superimpose digital data directly in the video image, implementing the digital shadow metaphor of Ishii and Ullmer [3].

Mobile Interfaces in Tangible Mnemonics Interaction

65

Another direction the Tangible Reminder can evolve is the use of real life artefacts that allow for programming by combination and handling of tool and mnemonic. This way, the computer could be made completely invisible and virtually superfluous. Yet, the problem of information display has to be solved. This could be done by switching to another media, sound and voice feedback for instance or attaching a display or projector to give visual feedback. Both extensions will make the Tangible Reminder an even more integrated intelligent piece of furniture not recognized as a common computer interface, lowering the inhibitions for computer usage.

Acknowledgments Many thanks to the programming team of the Tangible Reminder mobile project, Sung-Eun Kang and Michele Pinto, who worked on different aspects of mobile device portation and hardware integration as part of a practical course during their studies at the Institute of Media Informatics, University of Ulm.

References 1. Weiser, M.: The computer for the twenty-first century. Scientific American 265, 94–104 (1991) 2. Holmquist, L.E., Schmidt, A., Ullmer, B.: Tangible interfaces in perspective: Guest editors’ introduction. Personal Ubiquitous Comput. 8(5), 291–293 (2004) 3. Ishii, H., Ullmer, B.: Tangible bits: towards seamless interfaces between people, bits and atoms. In: CHI 1997: Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pp. 234–241. ACM Press, New York (1997) 4. Microsoft Corporation: Microsoft Surface (2008), http://www.microsoft.com/surface/ 5. Underkoffler, J., Ishii, H.: Urp: a luminous-tangible workbench for urban planning and design. In: CHI 1999: Proceedings of the SIGCHI conference on Human Factors in Computing Sstems, pp. 386–393. ACM Press, New York (1999) 6. Jacob, R.J.K., Ishii, H., Pangaro, G., Patten, J.: A tangible interface for organizing information using a grid. In: CHI 2002: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 339–346. ACM Press, New York (2002) 7. Altosaar, M., Vertegaal, R., Sohn, C., Cheng, D.: Auraorb: using social awareness cues in the design of progressive notification appliances. In: OZCHI 2006: Proceedings of the 20th Conference of the Computer-Human Interaction Special Interest Group (CHISIG) of Australia on Computer-Human Interaction: Design: Activities, Artefacts and Environments, pp. 159–166. ACM Press, New York (2006) 8. Bier, E.A., Stone, M.C., Pier, K., Buxton, W., DeRose, T.D.: Toolglass and magic lenses: the see-through interface. In: SIGGRAPH 1993: Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques, pp. 73–80. ACM, New York (1993) 9. Stone, M.C., Fishkin, K., Bier, E.A.: The movable filter as a user interface tool. In: CHI 1994: Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pp. 306–312. ACM Press, New York (1994)

66

T. Mahler, M. Hermann, and M. Weber

10. Mahler, T., Weber, M.: Mobile Device Interaction in Ubiquitous Computing. In: Advances in Human-Computer Interaction, pp. 311–330. In-Tech Education and Publishing (October 2008) ISBN 978-953-7619-15-2 11. Hermann, M., Mahler, T., de Melo, G., Weber, M.: The tangible reminder. Intelligent Environments. In: 3rd IET International Conference on Intelligent Environments, IE 2007, pp. 144–151 (September 2007) 12. van den Hoven, E., Eggen, B.: Personal souvenirs as ambient intelligent objects. In: sOc-EUSAI 2005: Proceedings of the 2005 Joint conference on Smart Objects and Ambient Intelligence, pp. 123–128. ACM Press, New York (2005) 13. Nielsen, J., Molich, R.: Heuristic evaluation of user interfaces. In: CHI 1990: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 249–256. ACM Press, New York (1990) 14. Mankoff, J., Dey, A.K., Hsieh, G., Kientz, J., Lederer, S., Ames, M.: Heuristic evaluation of ambient displays. In: CHI 2003: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 169–176. ACM Press, New York (2003) 15. Maeda, J.: The Laws of Simplicity (Simplicity: Design, Technology, Business, Life). MIT Press, Cambridge (2006)

Understanding the Relationship between Requirements and Context Elements in Mobile Collaboration Sergio Ochoa1, Rosa Alarcon2, and Luis Guerrero1 1

Universidad de Chile, Computer Science Department {sochoa,luguerre}@dcc.uchile.cl 2 Pontificia Universidad Catolica de Chile, Computer Science Department [email protected]

Abstract. The development of mobile collaborative applications involves several challenges, and one of the most important is to deal with the always changing work context. This paper describes the relationship between these applications requirements and the typically context elements that are present in mobile collaborative work. This article presents a house of quality which illustrates this relationship and shows the trade-off involved in several design decisions. Keywords: Context Elements, Software Requirement, Mobile Collaboration, House of Quality.

1 Introduction One of the most challenging activities when developing software applications is to understand how user needs are satisfied by the application’s functionality. Groupware applications are not an exception, and although determining groupware users' needs in advance and creating the correspondent software may seem uncomplicated, the experience proves that it is not the case and that groupware users will refuse to use collaborative systems that do not support their needs in a wider context [Gru94; Luf00; Orl92; Suc83]. In addition, typical groupware applications have been criticized for being decontextualized, they do not consider the complex conditions (e.g., social, motivational, political or economic factors), where software will be executed causing users to reject the system [Gru94, Lju96, Bar97]. The problem is that most groupware developers usually focus almost exclusively on the analysis and specification of functional requirements neglecting the non-functional ones. Furthermore, users may be unaware of their needs in a wider context (e.g. social, physical, etc.), so that, by simply asking them to identify their requirements, software developers may not obtain the appropriate information [Ack00]. User’s needs are translated into functional requirements and some functions are implemented to satisfy such requirements. However, the appropriateness of such functionality is mostly evaluated later, when the software has been built. Misconceptions at that J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 67–76, 2009. © Springer-Verlag Berlin Heidelberg 2009

68

S. Ochoa, R. Alarcon, and L. Guerrero

point in the software life cycle are costly, which adds to the costs of groupware testing. Hence, new techniques are required in order to understand in advance the impact of the intended functionality on user’s needs in a wider context. One of such techniques is the Software Quality Function Deployment (QFD). QFD can be considered as a set of methods that allows contrasting the customer requirements with the product functionality and can be applied at each product development stage [Cha03, Haa96]. QFD aims to understand how users’ needs are intended to be satisfied; it is a consumer-driven process. Typically, QFD is applied in four stages: product planning, parts deployment, process and control planning and production planning. Each step produces a matrix that serves as an input to the subsequent steps. Due to its shape, the matrix is called a House of Quality (HOQ) [Kus99]. A HOQ relates customer requirements with technical descriptions of the product, including design activities. That is to say, the HOQ captures the essentially non-linear relationships between functions offered by the system and customers needs. Conflictive features become apparent and trade-offs are identified early in the design process. QFD has been used successfully for identifying customers and their needs [Mar98] considering user satisfaction as a measure of product quality. The use of formal tools that accompanies software development has been proven to be significant in various industries and some researches had accounted the usefulness of QFD for the design of groupware applications [Ant05, Gle04]. However, due to the complexity of groupware design, we believe that technical descriptions are not enough for analyzing whether user’s requirements are met or not. As stated by Grudin [Gru94] and Ackerman [Ack00], among others, groupware users needs go beyond technical functionality and involves various contexts of analysis, such as social, technological, and physical context. Based on a framework for contextual design derived by the authors [Ala06], the QFD technique and the authors’ experience, we present a correspondence matrix that shows the relationship between typical mobile groupware requirements and context elements inclusion. The analysis shows that trade-offs appear early during design, and some context have up to 9 relationships. Our aim is to provide groupware developers with formal software techniques that help them to reduce software costs while enriching its quality. We believe that such quality is strongly related with the contextualization degree of the application. Section 2 presents the context elements that typically are involved in mobile groupware applications. Section 3 describes the groupware requirements involved in the development of these tools. Section 4 presents the derived HOQ as well as an analysis of the relationships between user’s requirements and context elements. Finally, section 5 presents the conclusions and future work.

2 Context Elements The authors defined a set of context elements that are briefly explained below. Context elements should be considered during the development of collaborative mobile applications and are represented in the columns of the HOQ shown in figure 1.

Understanding the Relationship between Requirements and Context Elements

69

Readiness to use IT. This context element allows determining the group members’ preparation for using Information Technology tools. Users’ experience, readiness to use technology and learning will influence the kind of interaction dialogues, interfaces, protocol design options and even the project feasibility. Previous formal context (e.g., rules and regulations). This context element assists on characterizing users’ information needs, as well as the actions the group should perform in order to conform to such regulations. Previous informal context (e.g., social conventions and protocols). Unlike formal contexts, social conventions naturally emerge during everyday users’ interactions. They cannot be imposed and they constitute a frame for understanding each other behavior and purposes. Work practice tools. Every work practice community usually develops its own tools. These tools are not necessarily supported by technology. Provided they mediate social interactions, they can assist the analyst to understand the current underlying workflow. Group members interaction. This context element helps identify general interaction scenarios among group members in order to determine which of them require mobile support. Such interaction must consider users’ communication needs for data and/or voice transfer. Mobility Type: mobile coupled. This context element represents a type of mobility that can be present in a collaboration scenario. Group members performing mobile collaboration activities in a synchronous way are considered as carrying out mobile coupled activities. Mobility Type: mobile uncoupled. This context element represents to the asynchronous work carried out by mobile workers during a collaboration process. Communication requirements. Communication can be direct or mediated; public, private or a mixture; broadcast or multicast. This context element represents the communication requirements of a mobile collaboration activity. Communication strategies constrain the coordination strategies that can be applied. Coordination requirements. Coordination elements and policies are contextual element that needs to be identified. Some of these elements are: support for session management; floor control administration; user roles support; shared information handling. Activity criticality. It is important to determine the urgency of achieving the activity goals and the activity importance for the user. These criteria may influence the choice of communication and coordination strategies. Activity duration. Except for the case of mobile phones, activity duration in mobile collaboration based on PDAs, notebooks or Tablet PCs can be critical as it could be restricted by battery life. This context element identifies the activity duration and the requirement of power supply.

70

S. Ochoa, R. Alarcon, and L. Guerrero

Organizational structure (rigid/flexible). The organizational structure will influence the group needs for coordination and control policies. A rigid organization requires formal coordination with strict control, but flexible organizations must quickly react to environmental changes. This context element represents the type structure of the organization that host mobile workers. Collaboration policies/rules/norms/conventions. Every organization develops a series of social protocols, policies, rules and norms that regulate its workflow. It is important to identify which are the social rules that may be relevant for the intended collaborative application. Group size. Group size matters. Research in groupware has pointed out the importance of group size for the success of the coordination, communication and collaboration strategies. Most groupware design elements will be affected by the group size. Roles. An appropriate identification of roles will help developers to design useful applications. Otherwise, the collaborative mediation process could not be well supported. Clearly it may have a meaningful impact on the group performance. Group structure. The relationships among roles will define the group structure. An understanding of the group structure and the relationship between it and the organizational structure could be useful to design the interaction policies to support collaboration. Demographics. It is also important to take into account the users’ characteristics, e.g., their age, gender, race, and language may influence the application design. Usability of the application will probably be improved when considering this context element. Physical space. This element represents the available space for deploying and operating the collaborative mobile application. The smaller/less comfortable/less stable the physical available space is, the less likely is to use large or heavy computing devices. Adverse environmental conditions. This context element represents physical conditions such as noise, light, number of people around and distracting factors. These factors impose restrictions over the type of user interface to be used for interacting with the collaborative application. Safety and privacy. These are two important context elements to consider during the application design in case of mobile applications being used in public spaces. Handheld devices are especially appropriate for use in public spaces. User location (positioning). Traditionally in groupware, it refers to users’ location within the virtual environment and is known as location awareness. Current technology lets users locate the partners in the physical world. Power supply. The activity duration is in direct relation with this context element. The analysis of this element helps developers to identify if the power autonomy of the selected mobile device is enough to support each activity.

Understanding the Relationship between Requirements and Context Elements

71

Communication capability. This context element represents the availability of networking infrastructure in the work scenario. This element also includes the communication bandwidth that is possible to get in the physical scenario for supporting the mobile collaboration activity. Uptime effort. A mobile device may need short start-up time, e.g., when users have little time periods to carry out work or when quick reactions are required. This element represents the effort to leave available the mobile application. Transportability. It is important to identify those activities requiring mobility and to estimate the effort the users are able to spend while transporting the devices. Computing power. This element represents the processing and storage capacities required for a mobile computing device. Based on that, more than one device type can be selected to support activities with different requirements.

3 Computer Supported Mobile Collaboration Requirements Based on a literature review and the authors’ experience, this section describes general requirements of collaborative mobile solutions. These issues will be useful to help understand the type of applications and capabilities required to work in a specific scenario. Next, a brief explanation of each requirement is presented. Interoperability. The interoperability of a collaborative mobile application has two faces: communication capability and interaction services of the mobile units. Communication capability involves the threshold, protocols and infrastructure used to support the communication process among mobile units [Ald06, Kor01, Tsc03]. The structure and meaning of all information shared among the applications should be standardized in order to support any kind of interoperability. Multimedia support. If the application requires capturing, managing and/or transmitting heavyweight data types, such as image, video or audio, smaller the device size more limited will be the solution. The features of each device limit the quality and quantity of data that is able to capture, store and transmit. All road. Typically the nomadic work makes the work context changes periodically, therefore the groupware application has to be context-aware, and also it has to consider as much work scenarios as possible. Robustness. Nomadic work requires an important effort of the persons that use the computer-based applications. Several distracting events and interruptions are happening around them. Therefore, if the mobile groupware application is not robust and able to consider these distracting factors, then the users could not utilize the application to support the nomadic work. Autonomy. Typically, the nomadic workers carry out loosely-coupled work. It means they work autonomously and collaborate on-demand. Such autonomy must be provided by the software tool; therefore it must avoid using centralized resources.

72

S. Ochoa, R. Alarcon, and L. Guerrero

Usable or usefulness. The functionality provided by the tool, the design of the user’s interfaces, and mobile computing device utilized to perform a mobile collaborative activity, influence the usability of the solution in the work field. These three elements must be considered during the application design in order to improve the impact of the solution. Synchronous/asynchronous work. Mobile collaborative applications require synchronous/asynchronous communication capabilities depending on the type of activity to be supported (synchronous or asynchronous). If asynchronous communication is required, every mobile computing machine is able to provide such support based on minimal network availability. On the other hand, if synchronous communications is required, a permanent and stable communication service should be provided independently of the environment the user is located [Sar03]. Mobile phones supported by cellular networks are typically the best option for synchronous communication provided their large coverage range and good signal stability [Mal02]. However, these networks have a limited bandwidth. Another option is to provide synchronous communication capabilities to mobile applications using a Wi-Fi communication infrastructure [Rot01, Kor01]. Although the bandwidth is better than cellular networks, Wi-Fi signal stability depends on the physical environment where it is deployed [Ald06]. Furthermore, this type of networks has a limited coverage range [Mal02]. Portability (transportability). If the application requires to be used on the move, transportability is a strong requirement. Typically, the way to address this issue is through the mobile computing device chose to support the collaborative work. Smaller the device size the more transportable is the device. However, the device size reduction implies restrictions at least on the screen size and input capability [Kor01]. Privacy. If the privacy is an important requirement, mobile computing devices usually have small screens, and thus, they provide better privacy protection than notebooks and tablet PCs if data displayed on screen needs to be hidden from other people in public spaces. Furthermore, the physical distance between the user and the handheld device during the interactions is shorter than the distance between a user and his/her notebook or tablet PC. Another privacy consideration in mobile collaboration is the visibility of the users and users’ actions in MANETs or public networks [Kor01]. Ensuring accuracy of location information and users’ identities, and establishing private communication could be a critical issue in some cases [Che00]. Long time support (battery life). Activity duration in mobile collaboration provide a strong requirement on the type of device can be used to support it. Many researchers have identified the battery life as critical to support mobile collaboration [Kor01, Gue06]. However, the use of context-information provides a way to optimize the use of power supply resulting in a longer lasting battery life [Che00, Hak05]. On the other hand, it is always possible to carry extra batteries when PDAs, notebooks or Tablet PCs are used. Activity duration is not so critical in the case of mobile phones because these devices are able to work for many hours without being re-charged [Hak05].

Understanding the Relationship between Requirements and Context Elements

73

Capability to be deployed. Handheld devices are easy to deploy and carry, and also they require low user’s attention and have short start-up time. These features allow fast reaction from the users; such speed could be critically needed in these physical environments. Mobility. Users’ mobility on a physical environment depends on the features of the physical environment where the users are located and the current environmental conditions. A user equipped with a mobile computing device can be traveling, wandering and visiting [Kri00]. Traveling is defined as the process of going from one place to another in a vehicle. Wandering, in turn, refers to a form of extensive local mobility where an individual may spend considerable time walking around. Finally, visiting refers to stopping at some location and spending time there, before moving on to another location. Sarker and Wells report that “the optimal size of a device associated with wandering was necessarily lower than an acceptable device size when visiting or traveling” [Sar03]. Performance. The processing power needed for certain mobile applications can exceed what handheld devices can currently offer [Kor01, Gue06]. However, in case of PDAs, it is possible to find commercial devices with CPU speeds higher than 500 Mhz. The processing power limitation of these devices becomes visible, e.g., while processing multimedia information. Although every mobile computing device is able to address basic multimedia needs, just notebooks and tablet PCs are able to handle strong multimedia requirements, such as support for 3D games. Storage. Storage restrictions have been reported in the literature, especially related to handheld devices [Kor01]. However, these devices keep improving their storage and memory capacities. Last versions of these devices allow mobile applications to manage and store complex data types, even simple multimedia information. Data input. A possible requirement for a mobile collaborative application is the need for massive data entry. Typically, the mobile computing device used to support the solution will play a key role. PDAs and mobile phones use pen-based data input, which is slow, but also useful to support short annotations [Buy00, Sar03]. On the other hand, notebooks and tablet PCs are the most appropriate devices to support data intensive processes using the keyboard.

4 House of Quality The correspondence matrix, also called House of Quality (HOQ), has typically three parts (Fig. 1): customer requirements (leftmost rectangle), technical descriptions (upper rectangle), and relationships between customer requirements and technical descriptions (centered rectangle). In addition, the grey line shows the direction in which each relationship should be enhanced in order to improve the application’s capability to support the mobile work.

74

S. Ochoa, R. Alarcon, and L. Guerrero

Fig. 1. House of Quality

Analyzing the matrix it is possible to see that around a 30% of the intersections between rows and columns have some kind of relationship. It means each design decisions should be made carefully. The positive relationships must be increased and enhanced, and the negative ones should be minimized and neutralized. In addition, applications with high degree of interoperability among various software tools as well as coupled interaction pose the major challenges as they consume several resources (storage, bandwidth, battery) and may compromise application’s robustness, mobility, and performance. Authors expect this matrix helps developers to make fast and accurate decisions during the development process. When a design decision has to be made, the designer will evaluate their alternatives, against the HOQ, in order to determine which is the most appropriate. Therefore, the proposed tool not only systematizes and facilitates the decision making process, but also it makes it cheaper and expedite.

5 Conclusions and Future Work The article presents the typical users’ requirements and work contexts that are present in the development of mobile groupware applications. The paper also presents and analyzes the relationship among these two set of components.

Understanding the Relationship between Requirements and Context Elements

75

The analysis shows that trade-offs appear early during the application design. In addition, such analysis allows designers to easily identify the context variables that should be monitored in order to detect a work context change, or improve the users’ interaction paradigm. Our aim was to provide a tool (the HOQ) that allows mobile groupware developers to improve software quality, in term of usability and effectiveness, through the improvement of the decision making process at the design time. We believe that product’s quality is strongly related with the contextualization degree of the mobile application. As a next step, we are analyzing in detail three mobile groupware applications, in order to show how the HOQ can be used to support particular design decisions, and also to show the impact these decisions have in the products’ usefulness. If the authors assumptions becomes true, this proposal could have an important impact in the development of mobile groupware applications.

Acknowledgements This work was partially supported by Fondecyt (Chile), grant Nº 11060467, and LACCIR grants No. R0308LAC001 and No. R0308LAC005.

References 1. Ackerman, M.S.: The Intellectual Challenge of CSCW: The Gap Between Social Requirements and Technical Feasibility. Human Computer Interaction 15(2/3), 179–204 (2000) 2. Alarcón, R., Guerrero, L., Ochoa, S., Pino, J.: Analysis And Design of Mobile Collaborative Applications Using Contextual Elements. Computers and Informatics 25(6), 469–496 (2006) 3. Aldunate, R., Ochoa, S., Pena-Mora, F., Nussbaum, M.: Robust Mobile Ad-hoc Space for Collaboration to Support Disaster Relief Efforts Involving Critical Physical Infrastructure. ASCE Journal of Computing in Civil Engineering. American Society of Civil Engineers (ASCE) 20(1), 13–27 (2006) 4. Ramirez, J., Antunes, P., Respício, A.: Software Requirements Negotiation Using the Software Quality Function Deployment. In: Fukś, H., Lukosch, S., Salgado, A.C. (eds.) CRIWG 2005. LNCS, vol. 3706, pp. 308–324. Springer, Heidelberg (2005) 5. Bardram, J.: I Love the System -I just don’t use it! In: Proc. of International ACM SIGGROUP Conf. on Supporting Group Work, Phoenix, US, pp. 251–260 (1997) 6. Buyukkokten, O., Garcia-Molina, H., Paepcke, A.: Focused Web Searching with PDAs. Computer Networks. International Journal of Computer and Telecommunications Networking 33(1-6), 213–230 (2000) 7. Chan, L.K.V., Wu, M.L.V.: Quality Function Deployment: A Comprehensive Review of Its Concepts and Methods. Quality Engineering 15(1), 23–36 (2003) 8. Chen, G., Kotz, D.: A Survey of Context-aware Mobile Computing Research. Dept. of Computer Science, Dartmouth College, Tech. Rep. TR2000-381 (2000), ftp://ftp.cs.dartmouth.edu/TR/TR2000-381.ps.Z

76

S. Ochoa, R. Alarcon, and L. Guerrero

9. Glew, P., Vavoula, G.N., Baber, C., Sharples, M.: A ‘learning space’ Model to Examine the Suitability for Learning of Mobile Technologies. In: Attewell, J., Savill-Smith, C. (eds.) Learning with Mobile Devices: Research and Development, London, pp. 21–25. Learning and Skills Development Agency (2004) 10. Grudin, J.: Groupware and social dynamics: Eight challenges for developers. Communications of the ACM 37(1), 92–105 (1994) 11. Guerrero, L., Ochoa, S., Pino, J., Collazos, C.: Selecting Devices to Support Mobile Collaboration. Group Decision and Negotiation 15(3), 243–271 (2006) 12. Haag, S., Raja, M.K., Schkade, L.L.: Quality Function Deployment: Usage in Software Development. Communications of the ACM 39(1), 41–49 (1996) 13. Hakkila, J., Mantyjarvi, J.: Collaboration in Context-Aware Mobile Phone Applications. In: Proc. of HICSS 2005. IEEE Computer Society Press, Los Alamitos (2005) 14. Kortuem, G., Schneider, J., Preuitt, D., Thompson, T., Fickas, S., Segall, Z.: When peerto-peer comes face-to-face: collaborative peer-to-peer computing in mobile ad-hoc networks. In: Proc. of First Int. Conf. on Peer-to-Peer Computing, pp. 75–91 (2001) 15. Kristoffersen, S., Ljungberg, F.: Mobility: From stationary to mobile work. In: Braa, K., Sorensen, C., Dahlbom, B. (eds.), Planet Internet, Lund, Sweden, pp. 137–156 (2000) 16. Kusiak, A.: Engineering Design: Products, Processes, and Systems. Academic Press, San Diego (1999) 17. Ljungberg, J., Holm, P.: Speech acts on trial. Scandinavian Journal of Information Systems 8(1), 29–52 (1996) 18. Luff, P., Hindmarsh, J., Heath, C.: Workplace studies: Recovering work practice and informing system design. Cambridge University Press, Cambridge (2000) 19. Malladi, R., Agrawal, D.: Current and future applications of mobile and wireless networks. Communications of the ACM 45(10), 144–146 (2002) 20. Martin, M.V., Kmenta, S., Ishii, K.: QFD and the Designer: Lessons from 200+ Houses of Quality. In: Proc. of World Innovation and Strategy Conference (WISC 1998), Sydney, Australia (1998) 21. Orlikowski, W.: Learning from notes: Organizational issues in groupware implementation. In: Proceedings of the ACM Conference on Computer-Supported Cooperative Work, CSCW 1992, pp. 362–369. ACM Press, New York (1992) 22. Roth, J., Claus Unger, C.: Using Handheld Devices in Synchronous Collaborative Scenarios. Personal and Ubiquitous Computing 5(4), 243–252 (2001) 23. Sarker, S., Wells, J.: Understanding Mobile Handheld Device Use and Adoption. Communications of the ACM 46(12), 35–40 (2003) 24. Suchman, L.A.: Office Procedures as Practical Action: Models of Work and System Design. ACM Transactions on Office Information Systems 1(4), 320–328 (1983) 25. Tschudin, C., Lundgren, H., Nordström, E.: Embedding MANETs in the Real World. In: Conti, M., Giordano, S., Gregori, E., Olariu, S. (eds.) PWC 2003. LNCS, vol. 2775, pp. 578–589. Springer, Heidelberg (2003)

Continuous User Interfaces for Seamless Task Migration Pardha S. Pyla, Manas Tungare, Jerome Holman, and Manuel A. P´erez-Qui˜nones Department of Computer Science Virginia Tech Blacksburg, VA, 24060, USA [email protected],[email protected],[email protected], [email protected]

Abstract. In this paper, we propose the Task Migration framework that provides a vocabulary and constructs to decompose a task into its components, and to examine issues that arise when it is performed using multiple devices. In a world of mobile devices and multiple computing devices, users are often forced to interrupt their tasks, move their data and information back and forth among the various devices manually, recreate the interaction context, and then resume the task on another device. We refer to this break from the task at hand as a task disconnect. Our objective is to study how software can bridge this task disconnect, enabling users to seamlessly transition a task across devices using continuous user interface. The framework is intended to help designers of interactive systems understand where breaks in task continuity may occur, and to proactively incorporate features and capabilities to mitigate their impact or avoid such Task Disconnects altogether.

1 Introduction and Motivation Today, with the advent of mobile devices and the deployment of computing in various form factors, the paradigm of a single user interacting with a single computer on a desk is losing its dominance. Even though the massive storage and computational power of a desktop computer has helped it to continue to be a central part of our daily work, most people interact with more than one device for their everyday tasks. From a recent survey of knowledge workers in a huge software development company and a major university [15], almost all participants reported that they use at least two computational devices for their day-to-day activities. The desktop computer and the notebook computer are still two primary devices that people use to accomplish their daily work. This proliferation of multiple computing devices is burdening the user with overheads for transferring information among different devices. Often, users are forced to copy files back and forth, open and close applications, and repeatedly recreate task context as they move from one device to another. In this paper we provide a theoretical foundation to describe the extraneous actions users need to perform as they switch from one device to another and propose the idea of Continuous User Interfaces (CUIs) to facilitate seamless task migration across multiple devices. We describe the development and evaluation of a proof-of-concept continuous user interface and observations from a preliminary usability study. J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 77–85, 2009. c Springer-Verlag Berlin Heidelberg 2009 

78

P.S. Pyla et al.

1.1 Tasks, Activities, Units and Cost Before formally defining a task disconnect, we first describe the vocabulary and terminology for tasks and the various associated parameters. Tasks

User Action

User Action

User Action

Procedure: Execution Sequence of Units

cost = x

cost = z

cost = y

Unit 1

Unit 2

Unit 3

Instructions

Instructions

Instructions

Fig. 1. Tasks, User actions, Units and Instructions

A task can be defined as a goal to be attained in given conditions [11]. Leplat expresses these conditions using three points of view: the states to be covered, the permitted operations, and the procedure [12]. At a lower level, we define tasks to be user actions. A user action is what the subject puts into operation (cognitive operations, behavior) in order to meet task demands. We also make use of Leplat’s definition of elementary units to be the elementary tasks, and elementary states or operations. Leplat uses these definitions to describe task complexity. However, we use the term units to further subdivide user actions to their lowest granularity. For non-trivial tasks (i.e., tasks that involve multiple activities), we define a procedure to be an operation execution sequence of multiple units. We also associate with each unit a parameter required for the successful execution: an instruction. Instructions are knowledge directions necessary to execute units, that can exist in the user’s understanding of the world or it can exist in the aids and artifacts in the task environment.The cost of a unit is a multidimensional attribute set that is incurred during the execution of a unit [12]. These dimensions could be cognitive, physical, memory-intensive, resource-intensive or a combination depending on the nature of the unit and the expertise of the user. Another important parameter of a task is time. In the words of Leplat, every task takes place in time and may be described by the temporal dimensions of its organization [12]. Out of the few temporal dimensions that Leplat describes, temporal ruptures is of particular importance to our work. We adapt and modify Leplat’s definition of temporal ruptures to mean interruptions by activities that do not directly contribute to the successful execution of the task at hand.

Continuous User Interfaces for Seamless Task Migration

79

1.2 Task Disconnects With the increasing proliferation of mobile and ubiquitous computing devices, people are forced to adapt their daily workflows to try and keep their work products and other data available to themselves at all times. E.g., Jones et al. [10] found that people often emailed to themselves the URLs of the websites they visited or their favorites list when they needed to access it from a different location. In scenarios such as this, where a user attempts to work with two devices, the need to transfer task and information back and forth between them is a burden. Users are forced to use workarounds like USB key drives, remote desktop software, e-mail, network file storage, and other means. These attempts to orchestrate a migration of data back and forth between the two devices create a gap in task execution. When a task is to be migrated from one device to another, the process involves more than just breaking it into two chunks along a selected boundary. It also involves the insertion of extra units into the task procedure that are entirely absent when the same task is executed on a single device. It is the inclusion of these extra units that hinders the seamlessness of task migration. Depending upon the exact nature of the task and the two devices in question, this process may involve simply adding more instructions to the procedure (low additional cost), or may involve an entirely new set of user actions (high additional cost). A task disconnect is a temporal task rupture arising due to extraneous user actions required when performing a task using multiple devices. Such extraneous user actions are required to accomplish the task across multiple devices, but do not directly aid in the completion of the task at hand. This raises issues such as how to help the user make a switch from one task condition to another in such a way that the need for a user’s attentional resources, cognitive workload, reaction time, and memory demands are minimized.

2 Related Work Bellotti and Bly [2] observed information workers to be mobile within the confines of their office; this local mobility existed mainly to enable the use of shared resources and for communication with other staff members. While mobile, users were seen to use secondary computational devices in addition to traditional devices such as desktops. A few strands of research have tried to address the problem of migrating tasks or applications over multiple devices. However, most of these studies have focused primarily on the technological aspects of this problem: Chu et al. [6] take the approach of migrating an entire application to support seamless task roaming, but with considerable latency (thus interrupting the user’s tasks sequence.) They do not discuss the implications on the user’s tasks and goals. Bandelloni et al. [1] describe user interaction with an application while moving from one device to another, in three levels of migration: total, partial and mixed. Chhatpar and P´erez-Qui˜nones [5] call this migration dialogue mobility and propose a requirement for the application data and logic to be separate from the user interface. Neither one of these projects take the task perspective we propose in this paper. Florins et al. [9] describe rules and transformations that attempt to provide graceful degradation of user interfaces while an application is migrated from one device to

80

P.S. Pyla et al.

another; even though their work is based on the same principle of continuity, their focus is on user interface generation and not on task migration. ARIS [4] is a window management framework that relocates running applications from one display to another; TERESA [14] helps design and develop model-based nomadic applications. Toolkits and tools such as TERESA have utility in rapidly deploying applications that can be migrated over multiple devices, but do not address the task semantics that users wrestle with while trying to interact with a multi-device interface. Denis and Karsenty [7] discuss a conceptual framework for the inter-usability of multiple devices. They provide an analysis of different cognitive processes in inter-device transitions and postulate two dimensions required for seamless interaction: knowledge continuity and task continuity. We base our work and the definition of Continuous User Interfaces on this requirement of seamlessness. We take this task-centered approach to solving the problem and we provide a definition, description, parameters, requirements, and a prototype to demonstrate a seamless interaction over multiple devices without task disconnects. Interruptions are events that break the user’s attention on a particular task to cater to another task that is in need of attention, and are the focus of a whole area of study by themselves [13]. Task disconnects can also be envisioned as analogous to interruptions, but occurring over multiple devices, locations, contexts, and most importantly, over a much longer time interval. Interruptions happen often in normal conversations [3,8]. However, in the field of linguistics [8,3], not all interruptions are disruptive or need repair. Even in cases where the interruptions are disruptive, the costs associated with repair are low, because humans have an inherent ability to repair, recover and proceed with most of the conversations using their ingrained social and cultural aids.

3 Study Design We targeted a specific application domain with sufficient complexity to allow us to observe clearly the different parameters responsible for task disconnects. Software development is a domain that involves work on several tasks of different affinities for devices. Software engineers perform requirements gathering and early design at client locations, away from their own offices, with laptop computers. They bring the artifacts of this stage to their office and continue to create a complete design using their desktop computers. The software that is designed must finally run on clients’ machines. We chose this application domain because of the need to use several tools such as text editors, drawing packages, scheduling programs, etc. when accomplishing a task, and because the nature of the task requires the use of multiple devices. We built a prototype to support the preliminary design phase of software engineering where developers collect client requirements and generate initial design prototypes, diagrams, and models. 3.1 Prototype We built a prototype that incorporated knowledge continuity and task continuity to provide a seamless multi-device user experience. The Task Explorer (Figure 2(a)) allowed users to create and demarcate a task, track the activities they performed in a to-do list tool (included), and provided constant visual feedback on the status of the connected devices in range. Opening a task in the Task Explorer launched the Task Viewer

Continuous User Interfaces for Seamless Task Migration

(a) Task Explorer

81

(b) Task Viewer

Fig. 2. Task Explorer and Viewer

(Figure 2(b)), a place to view documents and files related to a task. In our prototype application domain, these are requirements documents, diagrams and prototypes, e-mail addresses, and people related to the project in a unified view. Each task is uniquely color-coded to establish a visual identity with the task. Opening a document such as a requirements specification launched that file in an external editor. This was implemented as an application on a tablet interface. The interface leveraged spatial organization, shape, color, partitioning of data and function, recovery of state of data and recovery of activity context on its user interface. Tasks were migrated from the desktop computer to the tablet interface either automatically (because the task was open) or manually (by dragging and dropping). For each task, we displayed the task parameters in the same color gradient on the tablet and the desktop. The last drawing that was being accessed on the desktop computer was automatically loaded to maintain task continuity and activity context. If the drawing was cleared and replaced by another, the new diagram was synchronized automatically with the desktop. This obviated the need to open and save documents, making the interface more like paper. As artifacts were being generated, they were populated into the task tree on the right side of the screen. The task tree on the tablet brought together artifacts related to the task on the desktop computer, e.g. requirements documents, people, to-do list, and email messages. 3.2 Evaluation Interviews and user surveys were conducted to gather insights into the example task of prototyping and the existence of disconnects when using multiple devices to prototype. Six professional software developers were asked open-ended questions targeting the technologies and devices they used to prototype and any insights into disconnects arising due to the mediation by these technologies. In addition, we received N=32 responses to a survey that targeted software developers, graduate students with software development experience, and researchers in HCI who were familiar with computing and did prototyping tasks.

82

P.S. Pyla et al.

The prototype was evaluated with a group of graduate students with a software engineering background. A total of six participants participated in the evaluation. Three of the six participants constituted a control group where they were given tasks that required switching between a tablet and a desktop computer. The other three participants comprised our test group and were asked to perform the same tasks using our prototype. Each participant was assigned a total of seven tasks. Each task required drawing simple low-fidelity user interface prototypes using our custom drawing tool, updating requirements specifications using a text editor, or a combination of these two tasks. Participants were provided a background scenario explaining the context of a software development project for a fictitious client and the need to transfer documents between the tablet and the desktop. The participants were asked to use a tablet when meeting the client. Their interaction with the client was scripted in the scenario provided. The participants were asked to think aloud while they worked and the evaluator prompted the users when they stopped talking during a task. 3.3 Tasks The first task required the participant to make changes to an existing requirements document based on a fictitious client’s new insights into the project at the client’s location (i.e. using a tablet). The second task required the participant to prepare a low-fidelity prototype for the new requirements specification on the desktop. The third task asked the participant to visit the client to demo the prototype that was created on the desktop at the participant’s office. The fourth task required the participant to work on the desktop and to add more description to some requirements based on the client’s feedback. The participants were asked to assume they were at home for the fifth task (i.e. they were to use a tablet.) When they thought of a design feature, they were to create a new prototype with that insight to demo to the client the next day. The sixth task asked the participant to visit the client and demo the new prototype and get feedback. Based on the feedback, they were required to change the prototype and the requirement specification. The last task was set at the participant’s office where they were asked to update their desktop files with the latest prototype and requirements specifications. These tasks were designed with the goal of making the participants transfer information between the two devices as they progressed through the tasks. In the test group, this transfer was automatic because the participants used our prototype. In the control group, the participants had to move the files themselves using their choice of a USB pen drive, email, or other server-based technologies. The control group participants were provided the tablet and the desktop that were both connected to the Internet. They were given one task at a time with the associated scenario to provide the context of the interaction. At the end of the session, all participants were asked to fill out a questionnaire.

4 Results and Discussion In this study, we found several aspects of multi-device interaction that current systems do not adequately support. Specifically, several users reported dropping the use of multiple devices in favor of a single computer, to avoid the costs of task migration. They

Continuous User Interfaces for Seamless Task Migration

83

also reported that migrating files across devices taxes their short-term memory, is often frustrating, and likely to result in errors. We examine each of these in turn, based on our observations and responses from our study participants. 4.1 Use of a Single Computer Another interesting observation that one participant made was: “this [migrating data] almost makes me use the tablet alone for all the tasks and forget about my desktop if I had the choice”. When asked if she would do that even if the task at hand required more processing power (such as that available in a desktop), she responded affirmatively. Several survey respondents in another study [15] also confirmed that they chose to use only a single computer for convenience rather than have to copy/move files among several computers. This illustrates that the high costs of user actions associated with a task switch from one device to another prompt users to forgo computational power in favor of eliminating the switch entirely. 4.2 Consistency (or the Lack Thereof) of File Locations One common complaint from participants was that they needed to remember file locations and the state of a file on each device. As one participant put it, “this version control is getting irritating”. Remembering such extraneous information increases the short-term memory costs of this activity tremendously. Given that short-term memory is unreliable, it is difficult for the user to remember which device had the latest version of the data if temporal ruptures in a task take place over a longer period of time. This is another observation that directly supports our hypothesis that transferring activity context is important. The experimental group, who performed the same tasks using our prototype, were instantly able to locate documents and the information they needed. When switching from one device to the other, they reported being able to restart their task immediately and to be productive because the environment was already automatically loaded with the appropriate task context. Because information was re-displayed using less screen real estate, users were immediately able to focus on their work while keeping related information in their peripheral vision. The only limitation of the system was that users spent time moving and resizing the requirements window to enable them to easily see both and work between them. The act of copying files manually involves two steps: copying the data over, and placing it in the correct location on disk. Most current data copying tools and media (e.g. USB drives, email to self, etc.) assist in performing the first task, but not in the second. Thus, to perform the second step, users are forced to rely on their short-term memory, increasing cognitive workload and scope for error. Automatic system support for the second step therefore was viewed as a distinct advantage of our prototype. The related issues of version control and conflict management were also automatically handled.

84

P.S. Pyla et al.

4.3 Fear, Frustration In the questionnaire, all three control group participants reported a fear of making errors due to the overheads associated with the migration of information across devices. They also reported difficulty in keeping track of document version information. One participant commented that if a scenario such as the one in the evaluation were to occur in real life, the costs would be higher because of longer temporal ruptures. One of the participants forgot to copy the files from the desktop to the tablet before visiting the client. When she realized this, she remarked, “Wow! In real life, this would mean I’d have to go back to my office to get my updated files or redo the prototype that I did in the last task.”. On the questionnaire, members of the experimental group reported that they were less likely to make errors accomplishing the tasks. Also, because file state and application state were transferred automatically, the experimental group only had to worry about finding the appropriate location in the UI to begin work again. There were comments by some users that it would be nice to have a better view of all the files related to a project, but creating a new file system view was not the purpose of our prototype. Overall, participants of the experimental group responded that the application was more satisfying and easier to use (per Likert scale ratings.) This means that as the task procedure lengthens (in light of the extraneous actions required for task switching), and associated increase in costs, there is a corresponding rise in the likelihood of user error. In continuous user interfaces, such costs are reduced because of in-built system support for task migration. 4.4 Use of Mobile Computers as Secondary Displays For the second task, where the participants were required to create prototypes based on the requirements specification document, all the three participants in the control group preferred using the tablet as an information display. They opened the specification document on the tablet and referred to it as they sketched the prototype on the desktop. When asked about this, they said that having the information on a secondary display was good as it did not make them switch between different windows on one device. This might mean that CUIs should leverage the capabilities of the various devices even when they are co-located.

5 Discussion and Summary We explored the issues that arise when users use multiple devices to execute a single task. We found that current systems lend inadequate support for several user actions that must be performed during a task migration between/among devices. Among the problems reported were: dropping the use of multiple devices in favor of a single computer; increased short-term memory costs while migrating files across devices; higher frustration; and a higher likelihood of errors. We proposed and designed a prototype Continuous User Interface that ensured a seamless task migration for users attempting to perform a requirements specification

Continuous User Interfaces for Seamless Task Migration

85

and gathering task, using a tablet computer and a desktop computer. This system provided support for automatic migration of task context (e.g. the applications that were in use; pages and objects such as diagrams that were selected and active; etc.) between the two devices. In an evaluation conducted, participants reported that it helped mitigate the disruptive effects of task disconnects to a high degree. An interesting observation was that users expected to be able to annex existing collocated devices when performing their tasks (i.e., using the tablet computer at their desk along with their primary desktop computer.) They also reported that the automatic availability of necessary data on mobile computers directly contributed to higher perceived reliability and lower likelihood of error.

References 1. Bandelloni, R., Patern`o, F.: Flexible Interface Migration. In: IUI 2004: Proceedings of the 9th International Conference on Intelligent User Interface, pp. 148–155. ACM Press, New York (2004) 2. Bellotti, V., Bly, S.: Walking away from the desktop computer: distributed collaboration and mobility in a product design team. In: CSCW 1996: Proceedings of the 1996 ACM Conference on Computer Supported Cooperative Work, pp. 209–218. ACM Press, New York (1996) 3. Bennett, A.: Interruptions and the interpretation of conversation. Discourse Processes 4(2), 171–188 (1981) 4. Biehl, J., Bailey, B.: Aris: An interface for application relocation in an interactive space. In: Proc. 2004 Conference on Graphics Interface, pp. 107–116 (2004) 5. Chhatpar, C., P´erez-Qui˜nones, M.: Dialogue mobility across devices. In: ACM Southeast Conference (ACMSE), Savannah, Georgia (2003) 6. Chu, H.-h., Song, H., Wong, C., Kurakake, S., Katagiri, M.: Roam, a seamless application framework. Journal of Systems and Software 69(3), 209–226 (2004) 7. Denis, C., Karsenty, L.: Inter-usability of multi-device systems - a conceptual framework. In: Seffah, A., Javahery, H. (eds.) Multiple User Interfaces: Cross-Platform Applications and Context-Aware Interfaces, pp. 373–384. John Wiley and Sons, Chichester (2004) 8. Drummond, K.: A backward glance at interruptions. Western Journal of Speech Communication 53(2), 150–166 (1989) 9. Florins, M., Vanderdonckt, J.: Graceful degradation of user interfaces as a design method for multiplatform systems. In: IUI 2004: Proceedings of the 9th International Conference on Intelligent User Interface, pp. 140–147. ACM Press, New York (2004) 10. Jones, W., Bruce, H., Dumais, S.: Keeping found things found on the web. In: CIKM 2001: Proceedings of the tenth international Conference on Information and Knowledge Management, pp. 119–126. ACM Press, New York (2001) 11. Leontiev, A.: Le Developpement du Psychisme. Editions Sociales, Paris, France (1972) 12. Leplat, J.: Task complexity in work situations. In: Tasks, Errors and Mental Models, pp. 105–115. Taylor & Francis, Inc., Philadelphia (1988) 13. McFarlane, D.C.: Interruption of people in human-computer interaction. Doctoral dissertation, The George Washington University (1998) 14. Mori, G., Patern`o, F., Santoro, C.: Tool support for designing nomadic applications. In: IUI 2003: Proceedings of the 8th international Conference on Intelligent User Interfaces, pp. 141–148. ACM Press, New York (2003) 15. Tungare, M., P´erez-Qui˜nones, M.: It’s not what you have, but how you use it: Compromises in mobile device use. Technical report, Computing Research Repository, CoRR (2008)

A Study of Information Retrieval of En Route Display of Fire Information on PDA Weina Qu1, Xianghong Sun1, Thomas Plocher2, and Li Wang1 1

State Key Laboratory of Brain and Cognitive Science, Institute of Psychology, Chinese Academy of Sciences, Beijing 100101, China {quwn,sunxh,wangli}@psych.ac.cn 2 Honeywell ACS Labs, Minneapolis MN 55418, USA [email protected]

Abstract. This study was concentrated on which way is the most convenient for firefighter to get information, comparing among audio display, text display, and combined multi-modal display. Can fire commanders effectively obtain key fire information while they are en route to the fire, especially when they sitting in a moving and bumpy car? The task includes free-browse, free-recall and searching information. The result showed that: (1) Audio only always made firefighter taking the longest time to browse and search, but the introduction of audio display made the two combined displays more quickly to access information, and more easy to remember. (2) Searching in a moving environment took a little longer than searching in lab. (3) Comparing in the lab and in moving car, it was found that searching in a moving environment took a little longer than in lab. (4) It was proved that text display was still a necessary and indispensable way to display information. Keywords: Information retrieval, Display, PDA, Free-browse, Free-recall, Search.

1 Introduction The rapid growth of the IT industry during the last few decades has increased demands on mobile devices such as PDAs, cellular phones, and GPS navigation systems. With emerging concepts of context-aware computing, the mobile devices can provide mobile users with timely information by using not only common knowledge but also environmental context such as current time and location [1]. PDA has applied many systems. For example, alerts in healthcare applications [2], navigation system [3]. Auditory system is another important sensory system for getting information, which is the major complement to visual system. Moreover, human responds to auditory stimulus is faster than visual stimulus [4]. For Firefighting, time means life saving. En route display system is a kind of handheld device with mobile communication, which aimed to help fire commander to access the most current fire information as quickly as possible. Purpose of the experiment is to answer the following questions: can fire commanders effectively obtain J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 86–94, 2009. © Springer-Verlag Berlin Heidelberg 2009

A Study of Information Retrieval of En Route Display of Fire Information on PDA

87

key fire information while they are en route to the fire, especially when they sitting in a moving and bumpy car? Comparing among audio display, text display, and combined multi-modal display, which way is the most convenient for firefighter to get information?

2 Method 2.1 Experimental Environment Test equipment. An en route display prototype (installed in a handheld PDA device) has been developed as the experimental platform, which could show the current fire related information to fire commanders, such as the address of the building got a fire alarm, the location of the first alarm in the building, and so on. The size of words is based on previous experiment[5]. Based on the result from card sorting experiment, all the fire information has been clustered into three-level menu structure. The first level and the second level are total information (see Fig. 1). The third level is specific information.

Fig. 1. Content of the first and the second level menu

Fire scenarios. Totally 16 fire scenarios drawn from the previous 3D fire information display prototype[6] were pre-installed in the prototype, supposed to happen in 3 buildings: Camden building, a 45-floor simple structured tower building, and an 8-floor

88

W. Qu et al.

complex structured hospital building with unregulated shape. 2 of them were Camden building fires (single fire). 7 of them were tower building fires (including 2 SS fires (Single fire seed spread on Single floor), 3 SM fires (Single fire seed spread on Multiple floors)), and 2 MM fires (Multi fire seeds spread on Multiple floors). The other 7 of them were hospital fires (including 2 SS fires, 2 MM fires, and 3 SM fires) were prepared as the scenario pool. Experimental places. The experimental tasks were completed both in a lab (indoor test part) and in a moving car (outdoor test part). A set of standard usability test room was used as the indoor test place. A Volkswagen sagitar car was used as test car. Driving speed is about 45 kilos/hour. In the test room, firefighters were asked to fulfill two kinds of test tasks (free recall, information searching) by using en route display prototype. In the moving car, one firefighter sitting in co-driver’s seat was asked to do a dual-task: the main task is a counting how many street lamps passing by on right side along the road and speak out the number loudly; the secondary task is to finish information searching by using en route display prototype. 2.2 Participants 12 firefighters participated in our experiment. 11 of them aged from 21 to 29, and 1 subject is 35 years old. 6 of them have bachelor degree. 9 of them have more than 5-year firefighting experience. All participants had accomplished all the experimental tasks.

3 Procedure Each test was conducted by one firefighter and two experimenters. During the test, one experimenter worked as a moderator, the other one was in charge of video recording and note taking. The whole test consisted of three parts: training, test, and interview. Training experimenter took a couple of minutes to explain 5 key’s function on the PDA device so that subjects can know clearly how to use them (up: previous item, down: next item, left: previous level menu, right: next level menu and middle key: updated information) to browse the system and get information. Test. This part was divided into two stages: test in lab, test in moving car. Test in lab 1. Task 1 free browse and free recall: firefighter was asked to explore en route system for a few minutes (3-5 minutes) then stopped when he thought he was already familiar with the prototype, and was asked to recall whatever he can remember in his mind. Each firefighter could only try one of the four display ways to explore the en route system. 2. Task 2 Information searching: moderator show four items to firefighter one by one, and ask him using the en route prototype to find the items as quickly as possible. In order to assure firefighter had not only found the location of the item but also remembered the content, after finding the item, he was asked to repeat the content about the item without seeing the information.

A Study of Information Retrieval of En Route Display of Fire Information on PDA

89

Test in moving car 3. Task 2 Information searching: same as the test in lab. Interview. After the test part, firefighters were asked to evaluate the menu structure, to tell their preference to the four ways of fire information display. Then experimenter asked firefighter several questions. 3.1 Data Analysis of Free-Browse and Free-Recall Task Free-recall task was to test what firefighters can really get from the en-route display system and what they have memorized. Experimental design. One way between subjects design was used in this task. There were four ways of information display on en route system, which were audio only, text only, combined text + audio and text + the third level auditory display. 12 subjects were randomly divided into four groups. Each group has 3 firefighters. Each group tried only one kind of four information display ways. They were asked to browse and operate the PDA device freely till they felt familiar with the system and knew the fire related information. Then, they were asked to recall key information from they had browsed. Browsing time, the item number they browsed, and the items they recalled were recorded. All subjects used the same scenario (Camden building fire). Experimental result. Table 1 showed the numbers of browsed items and average time of free browsing. The first level menu had 6 items, so there are total 6 scores. If subject browsed only 1 item, he would get 1 score. The second level and third level menu had 24 items respectively. It’s found that “text only” and “combined text + audio” display are better than other two kinds of display. Subjects spent least time on “combined” display style. Table2 showed free-recall item numbers under the four different kinds of display conditions. If subject could recall one item correctly, he would get 1 score. It’s found that subjects could get more scores using “test only” and “combined text + audio” display than the other two display styles. Table 1. Free-browse item number and average time

audio only

1st level menu (6 scores including updated inf.) 5.3

2nd level menu (24 scores)

3rd level menu (24 scores)

average time for exploring

12.3

6

0:05:35

text only

5.3

17

13.3

0:04:40

combined text + audio

5.3

16

13.3

0:03:17

text + the 3rd level audio

5.3

12

10

0:03:41

5.3

14.3

10.67

0:04:18

mean

90

W. Qu et al. Table 2. Free-recall item numbers

audio only text only combined text + audio text + the 3rd level audio

1st level menu (6 scores including updated inf.) 0.7 1

2nd level menu (24 scores)

3rd level menu (24 scores)

total number

2.7 2.3

2 5

5.4 8.3

3

4.7

1

8.7

1

2.3

3

6.3

From the data on table 1 and table 2, we can say firefighters have browsed all the items at the first level, most items at the second level, and about half of the items at the third level. The more items they browsed, the more items they can correctly recalled. But the amount of their memory was still around the limitation of short memory: 7+/-2. 3.2 Data Analysis of Searching Task Experimental design. Searching task was to test the efficiency of the en route system for firefighter’s operations. 2*4 between subjects design was used in this task. Two factors were the places where the en route system was used, and the ways of fire information display. Lab and moving car were used as the two experimental places. Four different display ways (same as mentioned above) were tested to find which way is easier and more convenient to search information. 12 subjects were randomly divided into four groups. Each group has 3 firefighters. Each group used the same way of display as in the browse and recall task to search information. In the lab situation, each subject was asked to search four items: facility manager, hose, security passage and name and address of the building got a fire alarm. All subjects used the same scenario (hospital fire). In moving car situation, each subject searched four different items: Power Company, control room, road information and building structure. All subjects used the same scenario (Camden fire). In this task, subjects were asked to find the item as quickly as possible and repeat detailed content without watching back the PDA screen. The searching time and the percentage of correctly repeated content were recorded. Experimental result. Table 3 and table 4 showed percentage of correct repetition and searching time in the lab and in moving car. Searching time didn’t include subjects’ repetition time. Percentage of correct answers means accuracy of subject repeating detailed content. About the percentage of correct answer, 0 meant that subjects could not repeat the detailed content that they found; 1 meant that subjects could repeat part of detailed content; 2 meant that subjects could repeat all detailed content. It’s found that audio display took the longest time to search, and “text + 3rd level auditory display” took the shortest wherever in the lab and in moving car. The results

A Study of Information Retrieval of En Route Display of Fire Information on PDA

91

Table 3. Percentage of correct repetition and searching time in the lab Percentage of correct answers 0 1 2 Searching time

combined text + audio

Text + 3rd level audio

audio only

text only

16.70% 33.30% 50%

0 25% 75%

0 16.70% 83.30%

0 0 100%

0:00:55

0:00:17

0:00:25

0:00:13

Table 4. Percentage of correct repetition and searching time in moving car Percentage of correct answers 0 1 2 Searching time

combined text + audio

Text + 3rd level audio

audio only

text only

33.30% 33.30% 33.30%

8.30% 8.30% 83.30%

8.30% 0.00% 91.70%

8.30% 16.70% 75%

0:01:03

0:00:31

0:00:31

0:00:23

Fig. 2. Comparison of searching time between lab situation and car situation

showed that combined text + audio display were better repeated by firefighters than the other two ways. Among the four displays, there was no significant difference existed. Fig. 2 showed the comparison of searching time between the lab situation and the moving car situation. It’s found there was the same trend among the four ways of

92

W. Qu et al.

information display, and searching in moving car took about 8 seconds longer than searching in the lab.

4 Interview The short interview was trying to answer the following questions: 1. What do you think the menu structure? At each level of the menu, which item do you think the most important? Could you sort the items at the same level by their significance? Is there any suggestion to improve the current menu structure? 2. Which way of showing fire information do you prefer to choose when you are using the en route prototype on the way to the fire scene? Why? 3. Do you think the information shown in the system is easy to remember? How much information can you remember each time? 4. Do you think the fire information is easy to find? 5. For the button of updated information, what do you think it, useful, or not? 6. How do you think the en route information display, is it useful, or not? Why? 7. Is there any other suggestion to the prototype? 4.1 Subjective Ratings to the Four Display Styles From Table 5, we can say most of users prefer combined visual and audio display. (answer to the question 2) Table 5. Preference to the four display styles

audio only in the lab in moving car

combined text + audio

text only

0

16.70%

66.70%

8.33%

33.30%

41.70%

Text + the third level auditory display 16.70% 16.70%

4.2 Subjective Ratings to Menu Sequence 1 means the most important, 5 means the least important. Some firefighters said the most important item should be put on the first line, but at the first level menu the alarms was put on the bottom line although it was ranked the most important by everyone (see Table 6). Table 6. Preference to the first level menus sequence

mean

1 Facility phone contact 3.5

2 Firefighting equipment 3.2

3 Site information 3.8

4 General building information 2.6

5 Alarms 1.9

A Study of Information Retrieval of En Route Display of Fire Information on PDA

93

For the sorting of 2nd level and 3rd level menu items, it was found that at each category firefighters’ rating was consistent with the current sorting in Table 7. Combining the data in Table 6 and 7, to answer the first question about the menu structure, we think the current one is good enough except two things: 1) alarms should be put on the first line at the first level. 2) Security passage should be moved from “4 General building information” to “3 site information”. Table 7. Preference to the second level menus sequence

mean

mean

mean

mean

mean

1.3 1.2 hazard facility manager coordinator 2.6 3.2 2.2 2.1 fire equipment 2.3 hose equipment shutoff 1.9 3.3 3.4 3.1 secu3.2 road 3.3 road rity passage limit information 1.4 2.3 2.3 4.3inf.of 4.2gener surround4.1 ocal inf. of ing buildcupants building ings 2.8 2.4 4.6 5.1 1st 5.2 5.3 alarm alarm list hazard 1.4 2.2 2.4 1.1 building owner 1.8

1.4 power company 3.9 2.4 outdoor standpipe 3.4

4.4 building structures 3.5

1.6 water department 4.3 5.3 2.6 2.5contr power ol room room 3.9 5

1.5 gas company

4.5 keybox location 4.5

4.6name and address 2.4

5 Conclusion In order to answer how useful the en route information display system for firefighters’ information accessing, current situation understanding, and decision making, we did a series of tests to investigate the efficiency of the system, to compare different display ways including audio, text, and their combinations to find the most appropriate one. Based on the data, and the subjective ratings, we can summarize our findings as follows: 1. En route information display system was useful to help firefighter get the critical fire information and make decision more quickly and accurately. 2. Comparing the four information display ways: audio only, text only, audio + text, and text + the 3rd level audio, audio only always made firefighter taking the longest time to browse, search, but the introduction of audio display made the two combined displays (text + audio, and text + 3rd level audio) more quickly to access information, and more easy to remember.

94

W. Qu et al.

The reasons making audio display the worst could be: a. the voice message was not clear enough for firefighter to hear, especially in moving car; b. 3-level information structure is difficult for people to understand just by listening. But the data also showed that, after training, if firefighter has already got the information structure in mind, the convenience of operating with audio way will be the same as operating the system with other way. 1. Comparing the two situations of using the en route system: in lab, and in moving car, it was found that searching in a moving environment took a little longer than searching in lab. 2. It was proved that text display was still a necessary and indispensable way to show information. The reason was that, even there was a voice message playing, people still need to look at the text to make sure what they heard and understand was correct, especially for the building address, alarm location, and the contactors’ name, and so on. Because those information is really critical for firefighting and life saving.

References 1. Kim, N., Lee, H.S., Oh, K.J., Choi, J.Y.: Context-aware mobile service for routing the fastest subway path. Expert Systems with Applications 36, 3319–3326 (2009) 2. Chiu Dickson, K.W., Kwok Benny, W.-C., Kafeza, M., Cheung, S.C., Eleanna, K., Hung Patrick, C.K.: Alerts in healthcare applications: process and data integration. International Journal of Healthcare Information Systems and Informatics 2, 36–56 (2009) 3. Lee, W.C., Cheng, B.W.: Effects of using a portable navigation system and paper map in real derving. Accident analysis and Prevention 40, 303–308 (2008) 4. Quan, P.: Design of application Interface based on the human cognition. Computer Engineer and Application 19, 148–150 (2001) 5. Sun, X.H., Plocher, T., Qu, W.N.: An empirical study on the smallest comfortable button/icon size on touch screen. In: Aykin, N. (ed.) HCII 2007. LNCS, vol. 4559, pp. 446– 454. Springer, Heidelberg (2007) 6. Qu, W., Sun, X.H.: Interactive Style of 3D Display of Buildings on Touch Screen. In: Harris, D. (ed.) HCII 2007 and EPCE 2007. LNCS (LNAI), vol. 4562, pp. 157–163. Springer, Heidelberg (2007)

A Mobile and Desktop Application for Enhancing Group Awareness in Knowledge Work Teams Timo Saari1, Kari Kallinen2, Mikko Salminen2, Niklas Ravaja2, and Marco Rapino2 1

Temple University, 1801 N. Broad Street, Philadelphia, PA, USA, and Center for Knowledge and Innovation Research (CKIR), Helsinki School of Economics, Finland, and Helsinki Institute for Information Technology (HIIT), Finland [email protected] 2 Center for Knowledge and Innovation Research (CKIR), Helsinki School of Economics, Finland [email protected], [email protected], [email protected], [email protected]

Abstract. In this paper we present a first prototype for a mobile and desktop system and application for enhancing group awareness in knowledge work teams. The prototype gathers information from the interactions of the group within the application and analyses it. Results are displayed to members of the group as key indexes describing the activity of the group as a whole and the individual members of the group. The advantages of using the prototype are expected to be increased awareness within group possibly leading to positive effects on group performance. Keywords: Group awareness, emotional awareness, knowledge work, mobile application, desktop application.

1 Introduction We see knowledge work consisting of the capacity to act in intelligent ways in one’s context and environment. Senge [1] suggests that while information implies knowing “about” things, and is received and passed on, knowledge implies knowing “how”, thereby giving people the capacity for effective action. Davenport et al. [2] define knowledge work as “the acquisition, creation, packaging, or application of knowledge. Characterized by variety and exception rather than routine, it is performed by professional or technical workers with a high level of skill and expertise.” Consequently, knowledge work includes the creation of knowledge, the application of knowledge, the transmission of knowledge, and the acquisition of knowledge. McGrath and Hollingshead [3] have proposed that technologies, as they have been applied to groups, can be placed along a dimension of increasing and decreasing richness of social cues. Face-to-face groups have access to a rich variety of social cues that they can then use to determine the preferences and positions of other group members. On the other hand, computer-mediated groups, do not have access to nonverbal cues, and must rely simply on the written word. That is, computer mediated groups and hence computer mediated group work are low in the richness of social cues. J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 95–104, 2009. © Springer-Verlag Berlin Heidelberg 2009

96

T. Saari et al.

There are different types of group awareness some of which are relevant to worklike tasks. According to Greenberg [4], there are several types of group awareness needed for effective collaboration: • Workspace awareness is ``the up-to-the minute knowledge a person requires about another group member's interaction with a shared workspace if they are to collaborate effectively''. • ``Group-structural awareness involves knowledge about such things as people's roles and responsibilities, their positions on an issue, their status, and group processes.'' • ``Informal awareness of a work community is basic knowledge about who is around in general (but perhaps out of site), who is physically in a room with you, and where people are located relative to you.'' • ``Social awareness is the information that a person maintains about others in a social or conversational context: things like whether another person is paying attention, their emotional state, or their level of interest.'' Other information can be the special skills a co-worker has. Also, emotional awareness within a group has been discussed in relation to knowledge work groups [see 5]. Within this article we see emotional awareness, i.e. awareness of emotion and mood states of the members of the group as part of social awareness. Often in knowledge work situations, awareness of others provides information that is essential for frictionless and effective collaboration. Even though group awareness is taken for granted in face-to-face work, it is rather difficult to maintain in distributed settings. Hence, there is a considerable challenge in designing technology to support the types of group awareness that actually may lead to increased performance or other beneficial effects at work. We propose an application to increase awareness in knowledge work groups. Our application enhances group awareness by making explicit the implicit interaction patterns in a group. We expect our application to have beneficial effects on performance in knowledge processes and tasks.

2 Use Scenario and System Design Supporting general knowledge work processes Knowledge work tasks can roughly be classified to job specific tasks and general processes [see 6]. Job-specific tasks differ greatly as a function of the type of work. Examples are preparing a budget, analyzing results in terms of estimated and actual costs, planning and scheduling a project, eliciting and documenting system requirements, and writing applications software. There are also many general processes when working. General processes can be such as goal setting, communication, updates, group cohesion and synchrony maintenance and informal group communication and coordination. “Mobile” knowledge work differs mostly from “normal” knowledge work in that it takes place in distributed settings with the use of mobile technologies. Mobile knowledge work can be mostly mobile such as when a person is communicating with others

A Mobile and Desktop Application for Enhancing Group Awareness

97

and accessing files on the field while conducting work tasks. Mobile knowledge work is naturally intertwined with non-mobile knowledge work as people move in and out of their offices. Mobile technologies for knowledge work also mix with desktop computing environments as workers also carry their mobile phones to the office. Our use scenario is enhancing general communication processes of knowledge work teams. We feel that general communication and coordination processes of knowledge work vs. task-focused processes are not well supported by current technologies. In short we propose a system in which various data is collected from the status of single users and transmitted to other users to an easy-to-use mobile and desktop application. The application supports better group awareness in terms of workspace awareness, group-structural awareness, informal awareness and social awareness. Group performance and effectiveness We hypothesize that our system and applications will influence group performance. However, there is more to groups than performance on task only. For instance Andriessen [7] has divided the key aspects of group interactions as i) performance related to activities and specific tasks aimed at reaching a common goal for the group and ii) group maintenance, i.e. activities aimed at enhancing and building cohesion and trust in the group or to gain status and power in the group. Regards group awareness discussed above we expect that “better” group awareness (workspace awareness, group-structural awareness, informal awareness and social awareness) will then lead to positive outcomes in group interactions, in both performance and group maintenance. In addition to group performance another concept, group effectiveness has been proposed to consist of three components: 1. production function: effectiveness is here defined as “the degree to which the productive output meets or exceeds the performance standards set by the clients”. Criteria used to define such dimension are product quality, product quantity, efficiency and innovativeness; 2. group well-being function: is defined as “the degree to which the attractiveness and vitality of the group is strengthened”; 3. member support function: “the degree to which participating in the group results in rewards for the individual group members” [6, pp 100]. We propose that different types of group awareness are linked to group performance and group maintenance and are prerequisites for group effectiveness. Previous studies have shown that there are several possibly central concepts that are related to social interaction in our focus area of group performance. We use the examples of group cohesion, cooperativeness or reciprocity and convergence below. Group cohesion has referred to as “the extent to which group members perceive and feel attracted to the group” [6, pp 118]. Cohesion in a group has also been described as “a tendency to stick together or to be united either physically or logically” [8, pp 130]. Another important dimension in defining the quality of a social interaction is cooperativeness, a kind of reciprocity of communication. It is described as the behavior of people towards others in the group with which they share common interests and tasks. The behavior is also characterized by the fact that each person strives towards their goals within the group and this progress is facilitated by the other person’s actions, leading each to expect reciprocation [9].

98

T. Saari et al.

There is also another interesting approach to seeing how groups communicate: conveyance and convergence. Conveyance can be seen as the exchange of information among participants of the group in which the interpretation of the messages is done by the receiving individual [10]. This refers to the individual act of receiving and interpreting messages. Convergence, on the other hand, is about producing and facilitating shared social meanings among participants, rather than individual interpretations. In previous studies it has been suggested that these two different dimensions of meaning-making in a group may emerge differently based on the different mediating technologies used. People in face-to-face interactions tend to reach consensus (high level of convergence) faster than groups using online chatrooms [11]. It has been proposed that synchronous media are better in facilitating convergence whereas asynchronous media are better in supporting conveyance [12]. It then seems that it may be sensible to think of solutions to enhancing cohesion, cooperativeness, the efficient emergence of shared meanings and convergence as part of group awareness in the group in the context of technological support for group work. We have tried to address these issues in our prototypes with several ways of creating and sharing group awareness. Activity indexes for a working group For our prototypes we have preliminarily selected a number of key activity indexes that describe the state of the group or its activities. The indexes are formed by analyzing the user’s activities through our system. The indexes are displayed to users via our applications in visual representations. We have preliminarily chosen among many potential indexes which would be useful in group work. It should be noted that we have not yet tested our chosen indexes in field tests within the context of our applications and that the indexes after our first field tests could be altered, rejected or changed. Our chosen indexes for group activity are: group reciprocity, group centralization, my participation, my reciprocity, my popularity and my influence power. These indexes are hypothesized to be related to different types of group awareness. We expect that making such indexes available in a group work situation therefore enhances certain types of group awareness. In more detail the indexes and their meanings are as follows divided into group and individual levels:

A. Group level indexes: 1. Cooperation level Description: Describes the collaboration level in the group. Based on the tendency to contact others within the group in work tasks. A group could at certain times be very highly collaborative and at some times very little. This could be used to give an understanding the activity level and “tightness” of the group. Types of group awareness supported: Workspace awareness: what is the activity level of the group, what is the context into which I am sending my messages or requests? Social awareness: What is the level of “attention” of the group, how intensively the group works together?

A Mobile and Desktop Application for Enhancing Group Awareness

99

2. Communication hierarchy Description: Describes whether the group activity reflects a tendency to be hierarchical (top-down) around some individuals or whether it is democratic (many-tomany). In other words it is like the “power structure” of the group. Types of group awareness supported: Group-structural awareness: Who is the intellectual leader of the group? Who provide the best ideas? Is the group really collaborative and democratic or really top-down led by a single individual, perhaps corporate supervisor of the group? What are the different roles of people in the group?

B. Individual indexes: 1. My influence power Description: Describes how much a person is involved in the actions of the group. Can be based on how many contacts or messages the user receives. This reflects the importance of the user in the group. Types of group awareness supported: Group-structural awareness: How central am I as a person in the group? How central are others in the group? Do I have a lot of influence over the group? Who has most and least influence over the group? 2. My popularity Description: Describes the user’s popularity based on the idea that a user is more popular the more messages he/she receives. Types of group awareness supported: Group-structural awareness and social awareness: Am I popular? Who is the most popular person in the group? How popular is the person I am sending my message to? How should I craft my negative feedback message to this person as he is so popular in the group? 3. My participation Description: Describes how actively the user participates in the activity of the group. Is assessed on how much activity the user inputs into the system. Types of group awareness supported: Group-structural awareness: How actively do I participate in the groups activities? How active is my boss or co-worker?Social awareness: What is the level of interest of a person in the task we are doing based on his participation index? 4. My reciprocity Description: Describes how much the user is mutually contacted when the user contacts others. It reflects a kind of symmetry or mutuality of communication as seen from the point of view of a single user. For instance, a user could be communicating a lot but not receiving a lot of feedback, or the user’s sent and received messages reflect a balance or communication activities. Types of group awareness supported: Social awareness: What is the attention level of other users I am contacting regards my messages? Do they reply to my messages actively or not at all? Is there someone I am ignoring? Am I isolated in this group and if so, why? Should I talk more to this person as I have not really communicated with him?

100

T. Saari et al.

3 Prototype System and Application Mobile Application We constructed two prototypes based on our approach to designing the system: a mobile application running on mobile phones and a desktop application running on top of Microsoft Outlook. These two applications are the first ones constructed and reflect the status of the project at the time they were built. First we will discuss the mobile application. Technical description: client The client is based on Flash and Python technologies. The Flash part of the application is basically for handling vector graphics and showing results that come from the server. Python handles the hardware devices like Bluetooth and camera. We used Py60 for the Symbian platform and PyCE for WM. The Flash/Python application is able to run in many platforms such as Symbian phones (S60 2nd 3rd editions), WM, desktops (Linux/Windows/Mac OSX) and PocketPC. Technical description: server Server is based on Python/PHP. The main computations for social network analysis (SNA) indexes are done by the PSE, a special software that analyses and generates the social indexes discussed earlier. The interaction with PSE happens through a special protocol - SOAP - and the SOAP client is PHP based. Interface description When starting, the application updates data and this may take some time. The application opens to an orbit view (see Figure 1, column A). In this view the members of the group are placed to a three concentric circles with the user in the middle. Other group members are placed to the orbits around you either by “My Participation” or by “My Reciprocity” indexes. A person on the innermost orbit has similar index values with the user, whereas a person on the outermost orbit has very dissimilar values with the user. The user is given explanations of all the icons shown in Figure 1. For example MyParticipation is explained as ollows My participation refers to how extensively involved a person is in the communication among the groups members. Similar explanations and icons are used for other indexes. Left and right buttons in the mobile phone are used for rolling the circles to select a user, for the selected user the user name is highlighted. A firing button press opens a details window for the group member that has been selected. In the details window following data are shown (Figure 1, panel D): user name, availability and time tag, activity: a free text field for the current activity, last used: a time tag for when the user last time launched the mobile application, nearby: a list of nearby group members, based on mobile phone’s Bluetooth proximity, also, a user taken photo, with a time tag, is shown. The details window can be shut by pressing again the firing button. The Actions menu under the right soft key (Figure 1, panel E) offers sending a private message to the selected user, calling to the selected user, or sending him/her a sound message. From the actions menu the application can also be put to the background or closed.

A Mobile and Desktop Application for Enhancing Group Awareness

101

For some phone models it is not possible to put applications to the background, in these cases the only way is to shut the application, if other phone functionality is needed, and then again restart it.

A - Orbit

D - Details pop-up

B - Main

E - Details pop-up

C - List

F - My Details

G - Chat

Fig. 1. Different views of the first version of the mobile application. A) The orbit view. B) Main menu. C) List view. D) Details pop-up window for a single user data. E) Details pop-up window with the actions menu opened. F) My details view for inputting availability information etc. G) Chat view.

Pressing the Main with the left soft key opens the Main menu (Figure 1, panel B). Here the user can choose between the previously described orbit view, the List view, Input menu, and the Chat area. In the list view the list of group members is sorted with the same criteria as in the orbit view. Pressing the firing button opens the details pop-up window for a single user. In the My Details the user can input his/her status, mood, reason, action and launch the camera for taking pictures (Figure 1, panel F). On some phone models it is not possible to take pictures through the application. In the chat area the messages are preceded with a time tag (for example 1h = the message was sent 1 hour ago) and a user name (Figure 1, panel G). Desktop Application The desktop application works through an Outlook plug-in with Microsoft Outlook 2003 (and Outlook 2007 to a limited degree) and provides augmentation to the mail client on the basis of emails sent within the application. Additionally, Internet Explorer 7 or Mozilla Firefox 2.x is required for the Outlook tab to work properly.

102

T. Saari et al.

After the installation the user should be able to see the Pasion button (i.e. the button for our application to be launched) below the standard command bar (where there are icons to create a new document, open, save etc.) in Outlook. The Outlook plug-in collects data about e-mail traffic between trial group members and sends it to the Pasion server (i.e. our application server). The collected data are sender, recipient, time and date. It should be noted that no information on the content of e-mails is collected The data collected by tour server is used in calculation of the chosen indexes. These indexes are visualized both in the Outlook plug-in and in mobile application. The indexes are calculated from pooled e-mail and chat messages. In the upper left panel of the Outlook tab is a list of group members (Figure 2). Pressing Sort list in the upper left panel opens the sorting criteria menu. The user list can be sorted by Username, My Participation (automatically calculated index, described above), My Reciprocity, My Popularity, My Influence Power, Mood (manually set variable), and Availability (manually set variable).

Fig. 2. First version of desktop application interface in Outlook

Placing the mouse pointer over the user icon in the list view opens a pop-up window. From this window it is possible to send the user a private instant message, or view the user’s detailed information. Below the group panel is the MyStatus panel. Where the user can set his/her mood, reason, activity, or download a picture which is then shown in the pop-up details window. It is also possible to ask other group members to update their mood, reason, activity, or picture by placing the cursor over the Ask button. The request to update will be sent to the chat area. After editing the values the Set button must be pressed for the values to update. A logout button is also located there. Pressing the details button opens a view where the user can change password, language (English, Finnish, Italian, and Spanish), phone number, Bluetooth address, and e-mail.

A Mobile and Desktop Application for Enhancing Group Awareness

103

The orbit view operates similarly as with the mobile application, described above. Bringing a cursor over a user shows the details window on the right. A private instant message can be sent to user by clicking him/her. A separate tab is opened for private messaging to each group member. These can be closed from the chat areas title bar. In the details pop-up window there are shown: the user name, user taken picture, reason for mood, activity, last used time tag and nearby users determined by the Bluetooth proximity. The shown indexes are: My Participation, My Popularity, My Influence Power, and My Reciprocity.

4 Discussion One of the key future challenges for our work is to identify tasks that most benefit from the use of our system. After identifying the tasks we can redesign our system to support them and optimize the use of increased group awareness to fit the purpose. We hope that our initial emphasis on general communication processes across several types of tasks in knowledge work teams will produce results that help to focus our work further. There may also be challenges in the area of privacy when using our system. For instance, we hypothesize that no single user would wish to publicly transmit their social information. Similarly, problems of social comparison may arise. This has at least two sides. First, if a user is always in the periphery of the group as indexed by our visualizations in different knowledge work tasks, it may indicate unpopularity, not being liked, or not being valued. Second, as this is a system for work, there is always the question of one’s value to the employer. If the social network analysis visualizations show to one’s boss that one is constantly not producing much input and is at the periphery of interactions at a task in which one should perhaps not be at the periphery, this may create doubts about the role and value of such a person to the employer. On the other hand, both problems described above regards social comparison also cut the other way. For instance, if someone is very central and active as indexed by our system in some knowledge work task, it informs the person as well as the boss of his popularity and value to employer. In this way one could perhaps have “hardcore data” of one’s performance within the group and in the eyes of the employer. At the group level, it could be hypothesized that people tend to pump-up their performance relative to others and this would have positive outcomes regards group performance in total. However, the main problem may be the social mirror- effect. The indexes used create an individual mirror image in terms of: “Who am I in the context of this group? What is my worth? Am I popular and liked? Am I effective at my tasks?” Our system can also create an image of the others in the system, such as: “Who is this popular person? Why is this person the center of our communication? Why is this person at the periphery of our discussions when I expected more?” Our system creates a mirror image on some dimensions of the group using the system. This image can be either flattering or not. Despite the obvious challenges we feel that users will benefit from the use of our system by gaining a more holistic view of the group they are working with in addition to gaining insights about themselves in

104

T. Saari et al.

relation to the group. Our rationale is to enable users to transmit and receive enriched social cues to enhance their communication processes while working. The intrusiveness, resolution and accuracy of gathering the information as well as the understandability of visual representations of various indexes are naturally critical issues. The next stage of our work is to run field tests of our application in real-life working environments. Based on this research a new, refined version of the system will be built with tested and selected functionalities and improved visualization schemes.

References 1. Senge, P.: Sharing knowledge. Executive Excellence 14(11), 17–18 (1997) 2. Davenport, T., Jarvenpaa, S., Beers, M.: Improving knowledge work processes. Sloan Management Review 37(4), 53–65 (1996) 3. McGrath, J.E., Hollingshead, A.B.: Putting the “group” back in group support systems: Some theoretical issues about dynamic processes in groups with technological enhancements. In: Jessup, L.M., Valacich, J.S. (eds.) Group Support Systems: New Perspectives, pp. 78–96. Macmillan, New York (1993) 4. Greenberg, S., Gutwin, C., Cockburn, A.: A using distortion-oriented displays to support workspace awareness. Technical report, Dept. of Comp. Science, Univ. of Galgary, Canada (January 1996) 5. Saari, T., Kallinen, K., Salminen, M., Ravaja, N.: A System for Facilitating Emotional Awareness in Mobile Knowledge Work Teams. In: 41st Hawaii International International Conference on Systems Science (HICSS-41 2008), Proceedings, Waikoloa, Big Island, HI, USA, January 7-10, 2008. IEEE Computer Society, Los Alamitos (2008) 6. Woodman, R.W., Sawyer, J.E., Griffin, R.W.: Towards a theory of organizational creativity. Academy of Management Review 18(1), 293–321 (1993) 7. Andriessen, J.H.E.: Working with groupware. Understanding and Evaluating Collaboration Technology. Springer, London (2003) 8. Reber, A.S.: The penguin dictionary of psychology. Penguin, London (1985) 9. Raven, B.H., Rubin, J.Z.: Social psychology: People in groups. Wiley, New York (1976) 10. Wheeler, B.C., Dennis, A.R., Press, L.I.: Groupware comes to the Internet: charting a new world. ACM Sigmis Database 30(3-4), 8–21 (1999) 11. Dennis, A.R., Valacich, J.S.: Beyond media richness: an empirical test of media synchronicity theory. In: Proceedings of the Thirty-Second Hawaii International Conference on System Sciences, vol. 1 (1999) 12. Hung, Y.T.C., Kong, W.C., Chua, A.L., Hull, C.E.: Reexamining media capacity theories using workplace instant messaging. In: Proceedings of the 39th Annual Hawaiii International Conference on System Sciences, vol. 1, 19.2 (2006)

A Study of Fire Information Detection on PDA Device Xianghong Sun1, Weina Qu1, Thomas Plocher2, and Li Wang1 1

State Key Laboratory of Brain and Cognitive Science, Institute of Psychology, Chinese Academy of Sciences, Beijing 100101, China {quwn,sunxh,wangli}@psych.ac.cn 2 Honeywell ACS Labs, Minneapolis MN 55418, USA [email protected]

Abstract. This study was concentrated on how useful the en route information display system for firefighters’ information accessing, current situation understanding, and decision making, we did a series of tests to investigate the efficiency of the system, to compare different display ways including audio, text, and their combinations to find the most appropriate one. The result showed that: (1) Audio only always made firefighter taking the longest time to information detection, but the introduction of audio display made the two combined displays (text + audio, and text + 3rd level audio) more quickly to access information, and more easy to remember. (2) It should be clarified that en route system could be used very well either in quiet and static environment, or in a moving and a little bumping environment if user could get some training before using it. Keywords: Information detection, PDA, fire.

1 Introduction Fire incidents came into being with the discovery and utilization of fire and are closely linked to the advancement of human civilization [1]. Firefighter will know about details after arriving firing spots today. For saving time, a handheld PDA device has been developed, which could show the current fire related information to fire commanders. Fire alarm system is an essential part of high buildings in modern times, which helps firefighters’ detection more efficiently and reduces the casualty [2]. Ko’s research focused on proposed robust fire-detection algorithm that is installed in home network server [3]. How useful the en route information display system for firefighters’ information accessing, current situation understanding, and decision making, we did a series of tests to investigate the efficiency of the system, to compare different display ways including audio, text, and their combinations to find the most appropriate one. Purpose of the experiment is to answer the following questions: is the information they can get from the en route display system really helpful for firefighting decision making? In other words, comparing with the situation without en route display system, is the en route information really help the incident commander assess the situation and make decisions any faster or more accurate? Could firefighters be “primed” by en route information? J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 105–113, 2009. © Springer-Verlag Berlin Heidelberg 2009

106

X. Sun et al.

2 Method 2.1 Experimental Environment A handheld PDA device has been developed as the experimental platform, which could show the current fire related information to fire commanders. The size of words is based on previous experiment [4] .Totally 16 fire scenarios drawn from the previous 3D fire information display prototype [5] were pre-installed in the prototype. The experimental tasks were completed both in a lab and in a moving car. Since the PDA device was not dedicated for the en route display system, all the keys and buttons used in the experiment were four arrow keys for going up/down to the upper/lower level menu, and one middle key for getting updated information, the other keys became to be distracters. In order to avoid interrupting the application of en route prototype during the test, all the other functions were closed, and all the other keys and buttons were masked by plaster (see Fig. 1).

Fig. 1. PDA device as the experimental platform

Experimental places divided two parts: in a lab and in a moving car. In the test room, firefighters were asked to finish fire information detection task by using PDA prototype and 3D fire information display prototype. In the moving car, firefighters were asked to do a dual-task: the main task is to counting how many street lamps passing by on right side the road and speak out the number of lamps loudly; the secondary task is to finish fire information detection task using PDA prototype. 2.2 Participants 12 firefighters aged from 21 to 35 years participated in our experiment. 9 of them have more than 5-year firefighting experience. 6 of them have bachelor degree.

3 Procedure Test part was divided into three stages: test in lab, test in moving car, and test in lab with 3D prototype.

A Study of Fire Information Detection on PDA Device

107

Test in Lab 1. First alarm and fire spread finding (first on PDA then on PC touch screen): firefighter was exposed to four different scenarios (using hospital fires: SS fire, and SM fire) one by one. Each scenario was corresponding to one kind of displays: audio only, text only, combined text +audio, and partly combined text + audio. The display order was counterbalanced. For each scenario, firefighter was asked to find the location of the first fire alarm, and the fire spread by using en-route display as quickly as possible, task completion time was recorded as the performance of en route system. Then he was asked to go to the 3D fire information display system, which was used to compare the efficiency in the experiment of 2006[5], to find the location of the first alarm and the fire spread again. Task completion time was also recorded as the performance of 3D fire information display prototype. Test in Moving Car 2. First alarm and fire spread finding on PDA: procedure and the task requirement is the same as the task 3 in the lab. Also four scenarios (tower building fires: SS fire, and SM fire) was exposed to firefighter one by one. But firefighter was only asked to find the first alarm and the fire spread on PDA device. Task completion time was recorded as the performance. Test in Lab again: 3. First alarm and fire spread finding on touch screen: Just after the test in car and going back to the lab, only one scenario (the last one of the four scenarios used at the fifth step) was exposed to firefighter. The time of finding the location of first alarm and the fire spread was recorded as the performance. 4. First alarm and fire spread finding: A MM fire (multi fire seeds spreading on multiple floors) was exposed to firefighter, and record the task completion time both with en route system and the 3D information display prototype. 3.1 Data Analysis of Fire Detection Task This task was to test whether en route system help the incident commander assess the situation and make decisions any faster or more accurately and which way of information display was more helpful. Experimental design. 2*4 within subjects design was used in this task. Two factors were the places where the en route system was used, and the ways of fire information display. Lab and moving car were used as the two experimental places. Four different display ways were the same as mentioned above. Each firefighter was asked to complete 8 scenario fire detections (including locating the first fire alarm and judge the fire spread). 4 of them (hospital fires: SS or SM) were detected in the lab, and the other 4 (tower building fires: SS or SM) were detected in a moving car. Each scenario was corresponding to one kind of display style. The display order of the four scenarios and the matching pair between scenarios and the ways of information display were counterbalanced. For each subject, the test in the lab went first, and then followed by the test in moving car.

108

X. Sun et al.

During the test in car, as described in the experimental environment part, firefighter needs to fulfill a dual-task: counting as a distracted task was used as main task, and fire detection is the secondary task. Experimental Result Test in Lab Fire detection with en route system. Table 1 showed mean and std. deviation of task completion time in finding first alarm when using en route prototype. Task completion time was recorded from choosing the scenario by moderator to find the first alarm item. It’s found that “audio only” display was the worst way. It took the longest time. (F (3) =25, p=0.00). Among the other three display styles, there was no significant difference existed. Table 1. Task completion time in finding first alarm task with PDA

Mean of task completion time Std. Dev. N

combined text + audio

text + 3rd level audio

audio only

text only

0:00:50

0:00:11

0:00:09

0:00:06

0:00:26 12

0:00:11 12

0:00:07 12

0:00:02 12

Table 2 showed mean and std. deviation of task completion time in finding fire spread when using en route prototype. Task completion time was recorded from just finding the first alarm to finding the fire spread item. Among the four display styles, Both table 1 and table 2 showed the audio display took the longest to fulfill the fire detection task, but there was no significant difference existed. Table 2. Task completion time in finding fire spread task with PDA

Mean of task completion time Std. Dev. N

combined text + audio

text + 3rd level audio

audio only

text only

0:00:15

0:00:03

0:00:04

0:00:09

0:00:24 12

0:00:01 12

0:00:04 12

0:00:18 12

Fire detection with 3D information display prototype. Table 3 showed mean and std. deviation of task completion time in finding first alarm when using 3D information display prototype. Among the four display styles, there was no significant difference existed. Table 4 showed mean and std. deviation of task completion time in finding fire spread when using 3D information display prototype. Among the four display styles, there was no significant difference existed.

A Study of Fire Information Detection on PDA Device

109

Usefulness of the en route information display system. In our previous experiment conducted in 2006 [5], it’s found that, without the help of en route system participants didn’t finish fire detection task within 30s. In this experiment, combining the data in table 3 and table 4, we found all subjects could finish all the tasks no more than 30s when using “Text + the third level auditory” display for SS or SM fires. Table 3. Task completion time in finding first alarm task using 3D prototype

Mean of task completion time Std. Dev. N

combined text + audio

text + 3rd level audio

audio only

text only

0:00:24

0:00:17

0:00:24

0:00:11

0:00:25 12

0:00:18 12

0:00:25 12

0:00:05 12

Table 4. Task completion time in finding fire spread task using 3D prototype

Mean of task completion time Std. Dev. N

combined text + audio

text + 3rd level audio

audio only

text only

0:00:45

0:00:22

0:00:24

0:00:19

0:00:50 12

0:00:14 12

0:00:21 12

0:00:16 12

Test in Moving Car Sitting in a moving car, firefighter was asked to do a dual-task: counting street lamps loudly + fire detection. In this situation, moving and bumping made text on screen difficult to read, and dual-task made attention resource very limited to fire detection task. One hypothesis was that audio display would help firefighter to access information and understand it easily and quickly. Especially for the situation that user’s hands were not available for operating the en route system, which happened rarely in the lab situation. Therefore, except for the four kinds of displays we used in lab, an auto-play way was added as the fifth level of the information display ways. Here the auto-play meant that the fire information could be automatically play by voice message and any key/button operation was not necessary, which ensure firefighter could get all the information even if they had no hand, or had no time to activate the en route system. So, for each firefighter in this test part, he would do 5 fire scenarios and each scenario was corresponding to one kind of display styles. There are total 5 display styles, including auto-play audio way. The order of the 5scnarios was counterbalanced. Table 5 showed mean and std. deviation of task completion time in finding first alarm when using en route system. The time was recorded from choosing the scenario by moderator to the time point of having found the first alarm item. It’s found that finding first alarm using “audio only” display was the worst way. It took the longest time (F (3) =4.4, p=0.01). The bottom half in table 5 showed mean and std. deviation

110

X. Sun et al. Table 5. Task completion time in fire detection using en route system in car

task completion time Mean the first fire alarm

Std. Dev. N Mean

the fire spread

combined text + audio

text + 3rd level audio

audio only

text only

0:00:40 **a

0:00:13

0:00:05

0:00:05

0:00:54

0:00:10

0:00:02

0:00:03

12 0:00:08 **b

12

12

12

0:00:03

0:00:03

0:00:03

0:00:02

0:00:01

0:00:01

12

12

12

Std. 0:00:05 Dev. N 12 a: F (3)=4.4 p=.01; b: F (3)=9.2 p=.00.

of task completion time in finding fire spread when using en route system. It’s found that using “audio only” display to find fire spread was the worst way too. It took the longest time also (F (3) =9.2, p=0.00). Counting Task Performance This task aims to simulate an attention distracter, which can interrupt how the subject finished detection task. All the firefighters were required to count how many street lamps were passing by on right side along the road and to speak out the number loudly. Participants began to counting task when the recorder asked the participants to do it. After the experiment, the experimenter and the recorder count the number of street lamps in all together. From the table 6, we can see that S1 and S10 have the worst counting task performance, 67% and 69%, respectively. Most of the participants have good counting task performance. Someone even counted the number absolutely correct! The mean of correct percentage is 89%. Table 6. Percentage of correct counting street lamps in moving car Subjects

S1

S2

S3

S4

S5

S6

S7

S8

Percentage of correct 67% 90% 97% 96% 89% 85% 92% 100 answer

S9 S10

S11

S12

99% 69% 88% 91%

Comparison of the two test situations: in test room vs. in moving car. Fig 2 and Fig 3 showed the comparison of task completion time of finding first alarm and fire spread task between lab situation and car situation. They showed the same trend wherever in the lab or in moving car. It seemed that subjects could take shorter time in car to fulfill the fire detection task than to do it in the lab. There could be two reasons for the result: 1) The en route system was not so complicated to operate and understand, so even in a moving and bumping situation, it was

A Study of Fire Information Detection on PDA Device

111

0:01:00 0:00:52 audio only 0:00:43

text only Text + the third level auditory display combined text + audio

0:00:35 0:00:26 0:00:17 0:00:09 0:00:00 in the lab

in driving car

Fig. 2. Task completion time of finding first alarm

0:00:17 0:00:16

audio only text only

0:00:14 0:00:12

text + the third level auditory display

0:00:10 0:00:09

combined text + audio

0:00:07 0:00:05 0:00:03 0:00:02 0:00:00 in the lab

in driving car

Fig. 3. Task completion time of finding fire spread

still easy for firefighter to use. 2) There perhaps existed a training effect. In our experiment firefighter always completed the tasks in the lab first and then went through the rest tasks in car, which probably made them more and more familiar with the PDA device.

112

X. Sun et al.

Comparison of the two auditory displays: auto-play vs. play by manual control. For the auto-play display, mean of the task completion time is 2 minutes 52 second. 8 subjects thought automatic style wasted much time to find useful information. Test in Lab Again (Complicated Fire Test) Most scenarios used in lab test and in car test were SS fire, or SM fire, which were easy for firefighters to understand the current fire situation. In order to find the advantages and disadvantages of en route system, and to find the appropriate conditions that en route system could be made the best use of, after completing the 8 fire scenarios and firefighter’s going back to the lab firefighter was asked to fulfill fire detection task to a complicated fire (hospital or Camden fires: MM fire). The task completion time was recorded. 12 subjects were divided into three groups, and each group did one scenario using one of the four kinds of display styles. After they finished finding first alarm and fire spread by en route system, they would be asked to use the 3D FirstVision graphical display prototype to do the fire detection task. Table 7 showed mean and std. deviation of task completion time of fire location when using en route system. Result was similar as above. Audio display took the longest time (F (3) =71.03, p=0.00). Among the other three display styles, there was no significant difference existed. For the performance of fire spread, among the four display styles, there was no significant difference existed. Table 8 showed the task completion time of fire detection when using 3D prototype. Among the four display styles, there was no significant difference existed. Table 7. Task completion time in fire detection using en route system

task completion time

the first fire alarm

the fire spread

Mean Std. Dev. N Mean Std. Dev. N

audio only 0:00:12 ** 0:00:1 3 0:00:4 0:00:4 3

text + 3rd level audio

text only

combined text + audio

0:00:03

0:00:04

0:00:02

0:00:0 3 0:00:2 0:00:1 3

0:00:01 3 0:00:2 0:00:0 3

0:00:0 3 0:00:3 0:00:1 3

**F (3) =71.03, p = .00. Table 8. Task completion time in fire detection using 3D prototype

task completion time the first fire alarm the fire spread

Mean Std. Dev. N Mean Std. Dev. N

audio only

text only

combined text + audio

0:00:16 0:00:12 3 0:00:19 0:00:07 3

0:00:12 0:00:10 3 0:00:53 0:00:09 3

0:00:26 0:00:16 3 0:01:01 0:00:43 3

text + 3rd level audio 0:00:14 0:00:4 3 0:00:31 0:00:16 3

A Study of Fire Information Detection on PDA Device

113

For complicated fires, subjects couldn’t finish all the tasks within 30s. And few of the 12 firefighters could find there were 2 fire seeds in the test scenario.

4 Conclusion In order to answer how useful the en route information display system for firefighters’ decision making, we did a series of tests to investigate the efficiency of the system, to compare different display ways including audio, text, and their combinations to find the most appropriate one. Based on the data, we can summarize our findings as follows: 1. En route information display system was useful to help firefighter get the critical fire information and make decision more quickly and accurately. 2. Comparing the four information display ways: audio only, text only, audio + text, and text + the 3rd level audio, audio only always made firefighter taking the longest time to understand the information, but the introduction of audio display made the two combined displays (text + audio, and text + 3rd level audio) more quickly to access information, and more easy to remember. 3. Comparing the two situations of using the en route system: in lab, and in moving car, it was found that for the task of fire information detection, it took shorter when it was in moving car than in lab, but there was no statistical difference. This result didn’t mean people could do a better job in car than in lab. It should be clarified that en route system could be used very well either in quiet and static environment, or in a moving and a little bumping environment if user could get some training before using it.

References 1. Guo, T.N., Fu, Z.M.: The fire situation and progress in fire safety science and technology in China. Fire Safety Journal 42, 171–182 (2007) 2. Fang, Z.J.: Development of human-machine interaction: multimedia and multisensory. Human Factors 2, 34–38 (1998) 3. Ko, B.C., Cheong, K.H., Nam, J.Y.: Fire detection based on vision sensor and support vector machines. Fire Safety Journal 44, 322–329 (2009) 4. Sun, X.H., Plocher, T., Qu, W.N.: An empirical study on the smallest comfortable button/icon size on touch screen. In: Aykin, N. (ed.) HCII 2007. LNCS, vol. 4559, pp. 446– 454. Springer, Heidelberg (2007) 5. Qu, W., Sun, X.H.: Interactive Style of 3D Display of Buildings on Touch Screen. In: Harris, D. (ed.) HCII 2007 and EPCE 2007. LNCS (LNAI), vol. 4562, pp. 157–163. Springer, Heidelberg (2007)

Empirical Comparison of Task Completion Time between Mobile Phone Models with Matched Interaction Sequences Shunsuke Suzuki1, Yusuke Nakao2, Toshiyuki Asahi1, Victoria Bellotti3, Nick Yee3, and Shin'ichi Fukuzumi2 1

NEC Corporation, Common Platform Software Research Laboratories, 8916-47, Takayama-Cho, Ikoma, Nara 630-0101, Japan {s-suzuki@cb,t-asahi@bx}.jp.nec.com 2 NEC Corporation, Common Platform Software Research Laboratories, 2-11-5, Shibaura, Minato-ku, Tokyo 108-8557, Japan {y-nakao@bp,s-fukuzumi@aj}.jp.nec.com 3 Palo Alto Research Center, 3333 Coyote Hill Road, Palo Alto, CA 94304, USA {bellotti,nyee}@parc.com

Abstract. CogTool is a predictive evaluation tool for user interfaces. We wanted to apply CogTool to an evaluation of two mobile phones, but, at the time of writing, CogTool lacks the necessary (modeling baseline) observed human performance data to allow it to make accurate predictions about mobile phone use. To address this problem, we needed to collect performance data from both novice users’ and expert users’ interactions to plug into CogTool. Whilst novice users for a phone are easy to recruit, in order to obtain observed data on expert users’ performance, we had to recruit owners of our two target mobile phone models as participants. Unfortunately, it proved to be hard to find enough owners of each target phone model. Therefore we asked if multiple similar models that had matched interaction sequences could be treated as the same model from the point of view of expert performance characteristics. In this paper, we report an empirical experimental exercise to answer this question. We compared identical target task completion time for experts across two groups of similar models. Because we found significant differences in some of the task completion times within one group of models, we would argue that it is not generally advisable to consider multiple phone models as equivalent for the purpose of obtaining observed data for predictive modeling. Keywords: Cognitive Model, CogTool, Evaluation, Human Centered Design, Human Interface, Mobile Phone, Systematization, Usability Test.

1 Introduction Usability evaluation should be performed in the early phase of a product development process [1]. In addition, commercial enterprises demand that evaluations do not incur high costs. To satisfy these requirements, some of the authors of this paper have been J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 114–122, 2009. © Springer-Verlag Berlin Heidelberg 2009

Empirical Comparison of Task Completion Time between Mobile Phone Models

115

developing systematized evaluation methods and tools that can be applied early and economically [2]. CogTool [3] is a user interface evaluation tool for predicting task execution process and task completion time, using a given interface. In CogTool, a user model based on ACT-R cognitive architecture [4] mimics execution of a task by using graphical specification data extracted from frames of a storyboard for the task, which is input into CogTool in advance. CogTool offers a low-cost evaluation approach for the early part of a product development process. Just a sketch as the storyboard, which need not be functionally implemented, is enough for evaluation with CogTool. This small requirement allows us to evaluate the user interface early, cutting the cost of developing the system in which the user interface works. As a computational user model, not an actual human, executes tasks in CogTool, costs such as recruiting, organizing and paying participants and use of a usability lab are avoided. To apply CogTool to an evaluation for a new system, it is necessary to refine the user model to improve the accuracy of its predictions, using observed data of actual experts’ and novice’s task execution (observed performance data). In this refinement, we planned to incorporate observed performance data into CogTool’s user model and then compare its predictions with additional observed data [5]. The user model in CogTool can represent a novice who explores how they should interact with the target system, or an expert who can quickly execute the most efficient interaction sequences. In order to collect enough observed data to both incorporate in CogTool and to compare with its predictions, we needed to recruit a considerable number (approximately twenty) of experts, who had owned the specific model of the product to be evaluated for longer than two months. This research was part of an effort that also included comparing CogTool’s predictions in mobile phone evaluation to subjective user impressions in order to see if there were any correlations between these two different evaluation approaches.

2 Challenge of Recruiting Owners of Specific Models Recruiting owners of specific mobile phone models is very hard because the numbers of owners of a given model are low, due to the fact that new models are released frequently. Also, recruiting is expensive because “Owner of a specific mobile phone model” is a stricter qualifying condition for a recruiter than general conditions such as age and gender. Even if cost is not an obstacle, the recruiting may take a long time. Of course, an alternative way to make the recruiting easier would have been to reduce the number of participants. However we wanted to keep the required number (twenty) because we would like to modify the user model in high accuracy and to analyze the correlation between completion time and subjective impression statistically. Thus, it was clear that it would be quicker and cheaper to find owners of several similar related models that have matched interaction sequences for target tasks than only owners of one specific model. We defined a “matched interaction sequence” as the same sequence of key presses required to complete a given task.

116

S. Suzuki et al.

As mentioned above, our main objective in this research was a planned comparison between CogTool model predictions and observed user performance data. Our planned method was to capture the duration of each interaction event on the mobile phone by analyzing video frames and recording real user key presses [5] as they perform task execution steps in the same order as is specified for the CogTool model (thus excluding idiosyncratic user performance). This protocol was what drove the demand that all mobile phone models for the observed data have to have matched interaction sequences. In this paper, we report on a preliminary procedure that was conducted prior to our main observed user performance data collection effort. This was an empirical validation to clarify whether we could treat multiple mobile phone models with matched interaction sequences as equivalent for the purposes of predictive modeling.

3 Experiment This section describes our experiment in which we collected user task completion times (not individual key presses as is planned for our future study) across phone models in two groups (A and B), each defined by its members being a target phone that we wished to evaluate with CogTool or having matched interaction sequences to the target phone. This meant that within each group, the user interfaces were similar and tasks could be executed using exactly the same steps across models. The main difference between the models was simply their physical form factor (they had equivalent but differently sized and spaced keys). Participants executed a set of the same tasks with the same key press sequences across all the models in the group. After collecting data on participant performance, we compared the mean of the completion time for each task between the phone models. We explain this method in more detail below. 3.1 Mobile Phone Models We defined 2 mobile phone model groups. In Group A, there was the N905i, which was a target model for CogTool, and the N905iμ, which has the same interaction sequences as the N905i (Fig.1). In Group B, there was W61CA, another target model for CogTool, and the W61H and W53H, which had same interaction sequences as the W61CA (Fig.2). The models in each group have a matched key layout. For example, both N905i and N905iμ have a mail key above a main menu key. However, size, form, depth to press, and distance of keys varies by model. Because Fitts’ law [6] [7], which was used in CogTool, was the logarithm of key distance / key size, we supposed that a small gap of the distance or the size between the models would not affect the time to reach from a key to another. In this experiment, we selected only tasks with matched interaction sequences on the phone model within the group. For instance, in the case where a user has to select a target item to go to the next frame in a task, if the number of items above the target is different between the models, the number of key presses is also different with them. This difference means that their interaction sequences are not matched. Therefore, we

Empirical Comparison of Task Completion Time between Mobile Phone Models

117

Fig. 1. N905i (left) and N905iμ (right) in Group A

Fig. 2. W61CA (left), W61H (center), and W53H (right) in Group B

did not use either these models or tasks in the experiment. If the difference in number of items did not affect the interaction sequence (e.g., the items which are below the target item), the models and the task were usable in this experiment. Also, in cases where displayed labels of the items except the target one differed between the models, we used the models and the tasks in this experiment, because the participants trained to be experts, who already knew the interaction sequences for the task, could find which item they should select without comparing its label with other items’. 3.2 Participants 20 participants (16 males and 4 females, age: 20-40s) for Group A and 24 participants (18 males and 6 females, age: 20-40s) for Group B took part in this experiment. We did not select participants based on prior experience with specific mobile phone models. Instead, we provided all participants with extensive time to learn specific target tasks on specific phone models as described in 3.6 Learning.

118

S. Suzuki et al. Table 1. Task list for Group A

Content

Task 1

Task 2 Task 3

After inputting 000-3454-1111, store the number in a phonebook. Enter the name “Nihondenki” in Kana letter. Set a schedule named “Nomikai” in Kana letter from 19:00 to 23:00 tomorrow. Turn on/off “the auto keypad lock after folded”

The number of key presses 39

43 24

Task 4

Check the newest mail sent to Aoki-san in the “Friend” folder.

8

Task 5

Take a picture soon after launching a camera. Then save the picture in the “Camera” folder in the “My Picture” folder.

11

Table 2. Task list for Group B

Content

Task 1

Task 2 Task 3 Task 4

After inputting 000-3454-1111, store the number in a phonebook. Enter the name “Nihondenki” in Kana letter. Set a schedule named “Nomikai” in Kana letter from 19:00 to 23:00 tomorrow. Check the newest mail sent to Aoki-san in the “Friend” folder. Take a picture soon after launching a camera. Then save the picture as an idle screen, with a sub menu.

The number of key presses 39

40 6 13

3.3 Tasks We used 5 tasks for Group A and 4 tasks for Group B. The tasks are listed in Table 1 and Table 2. They are common mobile phone functions. At the same time, we selected tasks with various numbers of key presses. Although there were alternative interaction sequences for each task, we instructed all participants to use the same sequence for each task in this experiment. 3.4 Task Assignment to the Participants For Group A, we assigned two of the five tasks (see Table 1) to each participant. Thus, for each task in Group A, we had data from eight participants. For Group B, we

Empirical Comparison of Task Completion Time between Mobile Phone Models

119

again assigned two of the four tasks (see Table 2) to each participant. Thus, for each task in Group B, we had data from twelve participants. Each participant completed these two tasks across the phone models within the group. 3.5 The Number of Trials for Data Collection During the study, each participant repeated each task five times. We will refer to this portion of each task as the “main trial”. In addition, when participants switched from one phone model to the next, they performed two practice trials before the main trials for each task. These practice trials are not included in the data analysis. 3.6 Learning We set aside time for participants to learn assigned interaction sequences. In the learning phase, they executed the tasks assigned to each of them, with the assigned interaction sequences. In this experiment, we had to compare data generated by experts, because the observed data required for refinement of CogTool also needed to come from experts (as well as novice users). Another purpose of the practice was to reduce variance in completion time, because the more trials a person does, the smaller the gap of the completion time between the previous trial and the next one, along a general learning curve [8]. In these purposes, we set as many practice trials as possible so that participants were able to learn the interaction and develop as much expertise as possible. The practice for both models took place before the main trial part, making the participant’s learning level for each model more similar since we would expect transfer effects. If the order of the experiment had been “1. practice with N905i, 2. main trial with N905i, 3. practice with N905iμ, 4. main trial with N905iμ”, the participant’s learning levels for N905iμ could have been higher than for N905i because the participants benefit from experience with the phone they use first so that they would have far more experience during 4. main trial with N905iμ than when they did 2. main trial with N905i. Therefore, we set the order as, “1. practice with N905i, 2. practice with N905iμ, 3. main trial with N905i, 4. main trial with N905iμ”. By setting the order alternately for each task, we avoided a large gap between the learning levels for each model. The numbers of practice for each task were 46 times (23 times per a phone model) in Group A and 36 times (13 times per a phone model) in Group B. The numbers were dictated by a practical concern that the entire session for each participant should be completed in 90 minutes to avoid participant’s fatigue. For example, in Group A, there are all 120 trials; 20 main trials for 2 tasks (5 trials × 2 models × 2 tasks), 8 trials to get used to the model when the model switching (2 trials × 2 models × 2 tasks), and 92 practice trials (46 trials × 2 models × 2 tasks). If it takes maximum 45 seconds to execute 1 task, it takes 90 minutes to execute all 120 trials (45 seconds × 120 trials = 5600 seconds).

4 Results We conducted a one-way repeated-measures ANOVA for each task with phone model as the factor. In Group A, there was no effect of phone model in any of the tasks (p’s > .16), see Figure 3. In Group B, we found two significant differences. There was

120

S. Suzuki et al.

a significant effect of phone model in task 2 (F[2,22] = 5.86, p < .01) and task 4 (F[2,22] = 18.72, p < .01), see Figure 4. The other two tasks in Group B were not significant (p’s > .11). Post-hoc comparisons showed that in task 2, W53H was significantly different from the other two models (p’s < .05). And in task 4, W61CA was significantly different from the other two models (p’s < .05).

Fig. 3. Average completion time for each task with each model in Group A

Fig. 4. Average completion time for each task with each model in Group B

Empirical Comparison of Task Completion Time between Mobile Phone Models

121

5 Discussion Based on the results discussed above, for our planned data collection exercise to gather mobile phone interaction performance data to incorporate into CogTool, we will use only one model as the target model. Thus, even though it is likely to be more time consuming and difficult, we should recruit only owners of, and do our evaluations on, only the specific target phone model that we plan to model with CogTool, even though the recruiting cost is more expensive. One possible concern with the study design was that we had 12 participants for each task in Group B, but only 8 participants for each task in Group A. Thus, it may be the case that we only found significant differences in Group B because we had more statistical power from the larger sample size. To examine this concern, we reanalyzed the data from Group B with 4 participants randomly removed from each task. We found that both tasks were still significant at p < .05. This suggests that the difference in sample size alone is not why we found significant differences in Group B but not Group A. One of possible reasons of the significant difference between phones is the physical characteristics of the keys because many of the participants commented that these characteristics had affected their subjective performance. For example, some of the participants commented that flat keys had been difficult to distinguish from adjacent keys because of the lack of tactile cues. Others commented that keys with deeper key press feel made it easier to distinguish multiple repeated key presses using the tactile sense. Actually, W61H and W53H have flatter and shallower keys than W61CA has.

6 Conclusion The study suggests that we should not consider multiple mobile phone models with matched interaction sequences as equivalent to the same model, because we found significant differences in the mean task completion time between the models in Group B. Even though we found no significant differences between the two models in Group A, the findings from Group B suggest that a more conservative approach overall in using only one model may be warranted for developing cognitive models to minimize potential noise from usage variations across phone models. In Group B, there were one or two tasks with a significant difference in completion time between the models even though only four tasks out of a total of 10 target tasks were executed in this experiment. Based on the differences found in this preliminary study, we expected that it would be hard to find 10 tasks that did have matched interaction sequences but did not exhibit significant differences in completion time that would be suitable for our planned main objective to collect valid observed data on which to base modeling of mobile phone interaction. With more tasks, more participants and more trials in the main study, we would expect the number of significant differences between models to increase and make our observed data less reliable. As mentioned in the Discussion section based on the participants’ comments, we expect one of possible reasons of the significant difference between phones is the

122

S. Suzuki et al.

difference in tactile key press sensation due to hardware differences between different phone models.

References 1. Nielsen, J.: The Usability Engineering Life Cycle. Computer 25(3), 12–22 (1992) 2. Bellotti, V., Fukuzumi, S., Asahi, T., Suzuki, S.: User-Centered Design and Evaluation The Big Picture. In: Proceedings of Human Computer Interaction International. Springer, Heidelberg (to appear, 2009) 3. John, B.E., Prevas, K., Salvucci, D.D., Koedinger, K.: Predictive Human Performance Modeling Made Easy. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2004, pp. 455–462. ACM, New York (2004) 4. Anderson, J.R., Bothell, D., Byrne, M.D., Douglass, S., Lebiere, C., Qin, Y.: An integrated theory of the mind. Psychological Review 111(4), 1036–1060 (2004) 5. Teo, L., John, B.E.: Comparisons of Keystroke-Level Model predictions to observed data. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2006, pp. 1421–1426. ACM, New York (2006) 6. Fitts, P.M.: The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology 47, 381–391 (1954) 7. Fitts, P.M., Peterson, J.R.: Information capacity of discrete motor responses. Journal of Experimental Psychology 67, 103–112 (1964) 8. Newell, A., Rosenbloom, P.S.: Mechanisms of skill acquisition and the law of practice. In: Rosenbloom, P.S., Laird, J.E., Newell, A. (eds.) The Soar papers. Research on integrated intelligence, vol. 1, pp. 81–135. MIT Press, Cambridge (1993)

Nine Assistant Guiding Methods in Subway Design – A Research of Shanghai Subway Users Linong Dai School of Media and Design, No. 500 Dongchuan Road, Shanghai Jiaotong University, Shanghai, China [email protected]

Abstract. In big cities, it often occurs that passengers (users) have great difficulties to recognize subway stations. Except improving the signs of subway stations, based on large amounts of field researches, we find 9 practical and effective methods to help passengers to identify subway stations. These 9 methods include visual design, aural design, and tactual design etc. This paper also tries to apply some theories of cognitive psychology about human memory in the research of subways. These methods are also applicable to other space design in subway and even general underground space design. Keywords: User Research, Subway Station, Quick-Identification.

1 Introduction The following photos of the 4 stations were taken in Shanghai Subway Line 1 from the same point of view from the train.

Fig. 1. 4 different stations in Shanghai subway

From both photos taken on the site and large amount to interview, we find that most stations in Shanghai Subway are similar. If we omit the signs which reads name J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 125–132, 2009. © Springer-Verlag Berlin Heidelberg 2009

126

L. Dai

of a station, users can barely identify stations quickly through watching the platforms. In fact, the crowd condition of the train and the different user perspective all result in missing the sight of signs. In addition, from the research on familiarity of subway users and their behaviors, we find something very interesting: the more familiar a user with subways, the less he reliant on signs. Furthermore, the flood of advertisements also disturbed users’ sight. Thus we propose to develop some assistant methods to help users obtaining guiding information form the environment. Biological cognitive theories emphasize on the offers of the environment, and the offers about human surviving can be obtained instinctively, or can be learned without much effort. This enlightened us to make good use of human instinct of identifying environment. Thus, through proper design, we can make it more energy saving and more convenient for users in guiding themselves in underground environment. The way people guiding themselves on streets enlightened us that people seldom depend on signs in identifying familiar environment, such as the way home. People tend to build their cognition right from information provided by the environment. In order to transfer the good experience on the ground to underground environment, we need to add more characteristic information for users to memorize. This may be a new way in subway guiding field. The researches on behavior and way of cognition of subway users will enlighten designers to build humane subway environment. The techniques of research include abundant observation and interviews, eyetrack experiences and psychological experiences, questionnaires, and some literature reviews. We tried to find out new fields of guiding methods in subways through developing all kinds of senses of users, such as vision, hearing, and touching, and combine them with researches about human memories and human experiences.

2 9 Assistant Guiding Methods for Subways 2.1 Visual Series – Spatial Design Aids Guiding in Subways Voice from a Beijing subway user: I absolutely won’t miss Yonghe Palace Station! Even the passengers only have been there once, they also feel the same. Compared to those featureless stations, Yonghe Palace station is easy for users to recognize and memorize.

Fig. 2. Yonghe Palace Station, Beijing (left) VS the featureless Gulou Street Station (right)

Nine Assistant Guiding Methods in Subway Design – A Research

127

In subway construction, we can add some features to subway space, thus effectively help passengers to recognize stations and guide themselves. Common method such as, setting several layers of ground or ceiling, making inter-junction of horizontal position, comparing different spatial areas, making different arrangement of the columns along corridors, etc. Even adding features in a part of the space is also effective, e.g. a clearstory in the ceiling or a raised plant container on the ground. Furthermore, we can imitate the way people recognizing surroundings above the ground, i.e. to build characteristic landmarks in underground spaces, which is also a good guiding method for users. Spatial design can effectively leave strong memories in users’ minds, because the ability of recognizing three-dimension spaces is people’s instinct, and is continually developing in the process of human evolution. One person can store huge amounts of spatial memories in his mind. If we make good use of human brains, the results will be remarkable. 2.2 Visual Series – Color Design Aids Guiding in Subways Applying au unique color for each station works well in the design of Hongkong subway lines. As long as the passenger recognizes the color of the destination station, he will take off at the right station. The result of our research shows that we should pay attention to three aspects in color design for subways: 1. Use colors with high saturation, avoid using compound colors. Make it easy for users to identify the hue of the color. 2. The area of the color should be large enough for users to identify through the window in every corner of the train. 3. Do not use similar color in two neighboring stations.

Fig. 3. Hongkong Subway: Zhonghuan Station is red (see the left picture), while Jiulongtang Station is blue

128

L. Dai

Chicago subways are named by color. Stations of the same line all continuously bear the same colored ribbon in and out of the stations. It is easy for a user to know which line he is taking, thus strengthen the guiding effect of color. Surely enough, when we use colors to mark different subway lines, large areas of high saturated color is needless; otherwise, with all the stations of one similar color, memories of the users will be disturbed. 2.3 Visual Series – Decoration Design Aids Guiding in Subways The result of eye-track experiments shows that testees are able to identify a station by decoration. Another interesting finding is that memories of decoration is not accumulating with time, but appears to be a salutatory process, i.e. once or twice a user is attracted by some decorations, he will soon form a vivid memory of it. If he sees the same decorations again, he will recognize them immediately. Considering the limited space for users to pass, when we apply this method for guiding, we’d better use big and global decoration in the station, or set the decorations along the only path for users. Furthermore, as users need to identify a station from a train window quickly, the decorations in the platform should be distinctive and eye-catching; otherwise it will reduce the resolving power of it. As for the big decorations in other places, even not so eye-catching, users will identify them when passing by. 2.4 Visual Series – Lighting Design Aids Guiding in Subways Lighting is inevitable in underground spaces. Use lighting system to guide users is a good idea. Especially in the transfer stations, sometimes, it is too crowded for users to find the features of ground and wall. Then, using lights on the air for guiding is another effective way. There are two kinds of lighting design to aid guiding in subways: the shape of the lights and the color of the lights. As most lights in subways are fluorescent lamps, we advice that we should make different arrangement of these lamps, thus take advantage of the shape of the lamps. Fig. 4 shows with low cost, the guiding effect will rise. Condition permitted, the lighting system design in Japan is better in effect. As for using the color of lights for guiding, colorful lights such as LED is needed.

Fig. 4. The different arrangement of fluorescent lamps in Beijing subways

Nine Assistant Guiding Methods in Subway Design – A Research

129

Fig. 5. Lidabashi Station, Japan, designed by Makata Sei

2.5 Visual Series – Product Design Aids Guiding in Subways In our interviews, some sensitive passengers said that they use seats, billboards, newspaper boards, lamps, and advertisements to identify destination stations. Products which can aid guiding in subways are not only unitary but also diverse. They can help the users who pay great attention to details identifying stations quickly. Fig. 6 shows two subway stations with different multi-functional billboards in Vienna, which also serve as a symbol for station identification.

Fig. 6. Different multi-functional billboards in Vienna subways

2.6 Visual Series – Material Design Aids Guiding in Subways In the interview to Hongkong subway users, the effect of material in guiding is confirmed. Different materials are used in different lines of Hongkong subways. Thus the inter-changeable station Lijing Station between Quanwan Line and Dongyong Line use mosaic on one wall and plastic-aluminum board on another.

130

L. Dai

Fig. 7. Lijing Station of Hongkong subway

In the research of Shanghai subway users, some users said: Shanghai South Railway Station is relatively new. Thus proves that some passengers do use material to help identifying stations. But most passengers haven’t perceived the different materials used in Shanghai subways. Experiment shows that the effect of material in guiding is relatively poor. Only when the feature of material is enlarged and form a global impression, the effect will be obvious. 2.7 Aural Design Aids Guiding in Subways Some Shanghai subway users identify stations by sound. We got following comments in our interviews: I know which station is People’s Square even with my eyes closed, because almost all the people on the train will take off at this station with a big noise. Although I miss the broadcast of the name of a station, it will be fine, because the following broadcast will introduce some tourist sites for passengers. Once I heard a familiar name, I know it’s time for taking off. What’s more interesting is the musical broadcast in Pusan subway in Korea. If the train will soon stop in a station with sea nearby, you will hear the sound of ocean wave and sea-gulls. If the station is near mountains, you will hear birds twittering in the woods. Commuters will easily identify stations with those sounds of nature. Subway users take advantage of all kinds of sound to help identify stations. Based on memory theory, music is easier for people to memorize than other sounds. Using background music to guide passengers is feasible. Especially for those people who with poor eye-sight, a familiar tune is more friendly than any other guiding method. 2.8 Scent Design Aids Guiding in Subways We have not find any example about how scent help Shanghai subway users identifying stations in our research, but some passengers did mentioned that sometimes a

Nine Assistant Guiding Methods in Subway Design – A Research

131

special scent will remind them of a place. For example, a user who frequently takes off at Xujiahui Station mentioned that he can even smell the path to Pacific Bazzar because there is a W.C. on the way. These interesting reports excite us to explore how scent memories aid guiding in subways. As we all know, the nose of human being can help to store large amounts of information in human memory. Compared to other sense organs, nose is more reliant on intuition. Once a memory of scent is formed, it is hard to forget. 2.9 Experimental Memory Aids Guiding in Subways From some interesting interviews we find that some passengers’ guide themselves by a special experience in a certain station in the past, i.e. they memorize the station because they did something special in such place. Yes, I can identify Shanghai Railway Station because there is a W.C. on the platform. That’s true. I have been there, too. Haha~ Our interview proved that if we provide some additional functions in subways properly, e.g. commerce, entertainment, and exhibition, those colorful experiences will also help users memorizing and recognizing subway space. Subways in Paris and Japan, which built many years before, also have lots of commercial areas. Underground commerce can not only bring great profits, but also brings more stories about subways, and makes the dull trip a colorful experience. 2.10 Other Aspects of Design Aids Guiding in Subways There are many more interesting methods about how users guide themselves in subways, which is beyond our list. Only when we merge ourselves into real sites, can we hear them, see them, and feel them. For example, one passenger said he identify stations by watching the side of the opening door. Another passenger said that, we only need to remember one featured station, and then count the stops for destination. Other passengers remember the inter-change station by noticing other passengers’ behavior. We also meet a passenger who marks the number of the door to get on the train in the morning. Some passengers are so smart that even they have fallen asleep on the train; they will wake up automatically on the destination station.

3 Conclusion The 9 methods mentioned above are rooted in clues from different human senses. We hope that through proper design, underground spaces can provide adequate methods for users to memorize, thus let users identify and get to their destination quickly and conveniently. Make it easy for passengers forming cognizing-map in their minds, improve the efficiency of underground traffic. But we should notice that many users have their unique methods to identify underground environment, the construction conditions in different cities are also various, it is not possible for designers to consider every details. So how these methods make effect depends on the development of subway design and the purpose of constructors.

132

L. Dai

What’s more, the reorganization to environment of human beings is a global system. Based on theories of Gestalt psychology, perception of human beings is an integrated Gestalt, which is inseparable. Thus the 9 methods mentioned above are guide lines for designers in their exploration. It was the comprehensive effect of all the methods that make users identify underground environment quickly. A really good designer is who can guide users to use facilities conveniently and efficiently, thus bring pleasant and humane experiences.

Reference Li, J., Tang, Y., Qu, L., Zhang, D., Chen, R., Yao, Z., Xi, W., Zhou, X., Ren, H., Yu, L., Zhao, Z.: All the data in this paper is from the PRP Creative Project of Shanghai Jiaotong University (Serial Number: PRP-C10067) and National Creative Experience Project of College Students (Serial Number: ITP040). Project team members include the graduate and undergraduate students as follows

Pull and Push: Proximity-Aware User Interface for Navigating in 3D Space Using a Handheld Camera Mingming Fan and Yuanchun Shi Department of Computer Science &Technology, Tsinghua University, P.R. China [email protected], [email protected]

Abstract. In the 3D object controlling or virtual space wandering tasks, it is necessary to provide the efficient zoom operation. The common method is using the combination of the mouse and keyboard. This method requires users familiar with the operation which needs much time to practice. This paper presents two methods to recognize the zoom operation by sensing users’ pull and push movement. People only need to hold a camera in hand and when they pull or push hands, our approach will sense the proximity and translate it into the zoom operation in the tasks. By user studies, we have compared different methods’ correct rate and analyzed the factors which will affect the approach’s performance. The results show that our methods are real-time and high accurate.

1 Introduction Many 3D interaction tasks need the zoom operation. Suppose that if we want to wander in the 3D campus, we may need to go ahead to watch the landscapes. In order to satisfy this requirement, we can use the mouse to control the moving direction and the up arrow key to move ahead. The disadvantages of the method are as following. First, the operation needs relatively complex combinations of keyboard shortcuts with mouse movement and clicks. This usually operates with two hands. Second, it gives a low level of naturalness and is not a good choice for the children or people who are not familiar with the keyboard and mouse operations. In order to crack the above two disadvantages, we propose a method that users could simply pull or push their hands to move in or out by holding the camera. When they want to go ahead, they just need push their hands forward. When they would like to go back, they just need pull their hands back. Our approach needs only people’s natural movement and almost need no study. Besides the naturalness, the operation only needs one hand and people may use the other hand for other operations. Some researches [2, 5, 6] have done the familiar studies, such as the Harrison and Dey [2] try to recognize the people proximity by the camera in the computer. However, during their mode, the camera is still and the approach is not proper for the 2D or 3D interaction tasks such as the object control or the virtual space navigation. IsseU [5] is similar to our approach. IseeU tries to calculate the change in the standard deviation of the positions of feature points, which are selected in the image captured by the camera, and transform it into a zooming message. However, we analyze that it is not enough to give a high accuracy. J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 133–140, 2009. © Springer-Verlag Berlin Heidelberg 2009

134

M. Fan and Y. Shi

After having studied the previous works, we first give two methods to recognize the zooming message. After that we test the accuracy rates of them and analyze the factors which have an effect on the accuracy. Then by taking more factors into consideration, such as how to support large distance zoom, we modify the methods to make them more efficient.

2 Framework of the Algorithm The handheld camera is just a tool for interaction. Because the camera is hold steadily in user’s hand, the camera’s movement will reflect the hand’s movement. In order to detect the camera’s movement, first, we detect some corner points in the image frames captured by the handheld camera, then by analyzing the geometric characters of the corner points’ positions, we try to decide whether the movement is zooming or not. During the following part, we propose two methods to detect the zoom and then compare them with each other to see which one is better.

Corner points detecting

Corner points tracking

Zooming detected

Controlling Application

Fig. 1. The whole framework of processing

3 Corner Points Detecting and Tracking Corner-like points [4] which are corners, with big eigenvalues in the image, are easy to find on incoming frames and are relatively stable while being tracked. Tracking the points means finding the new positions of the corner points, which appeared in the last frame, in the frames. Our approach tracks the feature points by implementing sparse iterative version of Lucas-Kanade optical flow in pyramids [1] .

Fig. 2. The green points are the corner points

Pull and Push: Proximity-Aware User Interface for Navigating

135

4 Zoom Detecting Algorithms In this part, we will discuss two algorithms to detect the zoom in detail and then compare their performance. 4.1 Algorithm One: Sensing the Distance As the figure3 shows, A, B, C, D are positions of the corner points in the last frame, the A’, B’, C’ D’ are the positions in the frame. The average distance between the corner points and their centers becomes farther when the camera zooms in, since the distance between the camera and background are shortened. According the above analysis, first, we calculate the positions of the corner points in the last and now frames. Then, calculate the average distances among them and their centers. Finally, calculate the rate of the new distance and old distance. If the rate is over 1.0, it means that the camera pulls back. If the rate is less 1.0, it means that the camera pushes forward. In the real experiment, due to the hand’s jitter, the camera may have the slight movement. In order to reduce the jitter’s interference, we set a threshold to instead of the above number 1.0.

Fig. 3. Corner points’ positions before and after the camera zoom in. o and o’ are the centers of the old corner points in the last frame and new corner points in now frame.

4.2 Algorithm Two: Sensing the Change of the Area As the figure 3 shows, the corner points form a polygon ABCDE. After the zoom in

，

operation, the polygon becomes the A' B 'C ' D ' E ' the area of the polygon ABCDE changes to be larger. Through sensing the change of the area, we can decide whether a zoom in or out happens. 4.3 The Accuracy of the Two Algorithms Participants Seven participants, six male and one female, take part in the test. They use a webcamera with frame rate 30 fps(frames per second) and a pentium 4 PC with the

136

M. Fan and Y. Shi

main frequency 3.2GHz. Each of them takes the experiment for about five minutes. Before the test, they are given no more than five minutes to be familiar with the camera. Experiment We have rendered a 3D virtual space with DirectX 3D(see figure4). People are asked to use the camera to go ahead or back in the virtual space. We count the total decisions and the right decisions, then calculate the accuracy rate. (We ask the testers to do zoom in operation, then we count the total judgment and the actural zoom in times).

Fig. 4. The left one is the last image. When users push the camera forward, our view goes forward and the house becomes bigger than ever.

Accuracy Rate Participants are asked to do the zoom in and out movements to test two algorithms’ accuracy rates. According to the Fitts law [3], the distance between the camera and positions of the corner points in the real world will have an effect on our algorithms. So in order to test how the distance factor affects our algorithms’ performances, we do the experiments at different distances, such as 0.6m, 2~2.5m, 5m. We calculate the seven participants’ results and give the average accuracy rates at different distances in figure 5. Discussion From the figure5, we can conclude that: • The average accuracy rate of the algorithm2, which senses the proximity by calculating the changes of areas, is higher than the algorithm1, which detects the zoom by calculating the changes of distances. • As the distance between the camera and the corner points’ real world positions increase, the accuracy rate declines rapidly. At the distance about five meters, the accuracy rate of algorithm one has been below 50% and the accuracy rate of algorithm two is almost equal 50%. • The zoom detecting algorithms are totally sensitive about the distance. Within the five meters, the algorithm can keep the accuracy rate above 50%. Within the

Pull and Push: Proximity-Aware User Interface for Navigating

137

Fig. 5. The average accuracy rates of seven participants’ results

distance 1~2m, the two algorithms can keep the accuracy rate over 80%. This results guide us that our hands had better push or pull the camera in a direction in which there are some objects with in 1~2m. • The experiments are taken by seven participants who only use the camera to operate less than five minutes. The results show that they can operate the zooming easily and need less time to practice. 4.4 Large Distance Zooming Support Using a Finite State Machine From the questionnaire, they reflect that the approach is not suitable for moving a long distance at one time. If they want to go ahead in the virtual space for a long time, they must keep pushing or pulling the camera for a long time. This is impossible due to users’ moving space is limited. In order to crack this hard nut, we give the strategy that we first detect the movement. If we continuous detect the zoom in movement for two times, then we simply think that users want to zoom in. In this situation, we output the zoom in. if users want to stop zooming in, they can pull back the camera. If our approach detects the zoom out movement for two times, we think that users want to pull back. The whole procedure can be described as a finite state machine (FSM)(Figure 6). During the zoom in / out state, our approach will output the “zoom in/out” decision. Suppose the current state is “zoom in”, and current judgment of the algorithm is “zoom out”, then the counter Count1 will add by one, then we examine whether Count1 is two or not. If the count1 is two, then the state will change to “zoom out” and the output is “zoom out”. But if the count1 is not two, then the state will still be “zoom in” and the output is “zoom in”. if the current state is “zoom in” and the current judgment is “zoom in”, our approach will output “zoom in” and set the count1 as zero. The reason why we use the counters when the state is changing is to make our algorithm stable. Since the camera is held in people’s hand and the hand will be

138

M. Fan and Y. Shi

Zoom in Count1 = 0

Zoom out Count2 = 0 Count2= 2

Zoom in Count2 = 0 Zoom out Count1 ++ && Count1 !=2

Count1= 2

Zoom out Count1 = 0 Zoom in Count2 ++ && Count2 != 2

Fig. 6. The finite state machine of zoom in and zoom out

naturally jitter while suspending in the air. And this jitter maybe causes some zoom in or out motion. If the approach does not use the state machine or two counters, then only a slight noise will cause false decisions. Experiment for Testing the Effect of Finite State Machine Participants and the hardware conditions are the same as the above experiment. We have done two zoom detecting algorithms, one uses the finite state machine and the other does not. All participants have required to do zoom in and out movements alternatively for about five minutes. Each have done the experiments twice, one time is without the finite machine and the other time with the finite machine. The average distance between the corner points’ positions and users’ hands’ positions is about one meter which is good for our algorithm to work. The average accuracy rates are calculated.

Fig. 7. The accuracy rates of the algorithms. One is 0.93, the other is 0.95. The result shows that the FSM improves the algorithm’s performance.

After the experiments, participants give us some valuable feedbacks, based on which we conclude the following • With the finite state machine, they can continuously zoom in or out. While they use for the object controlling, they can magnify or reduce the size of objects. While

Pull and Push: Proximity-Aware User Interface for Navigating

139

using in the virtual space wandering, they can continuously go ahead or back in the scene. • They can switch between the zoom in and zoom out movement with higher accuracy rate. • Before the hand’s motion state changes to the other one, the counter must count to two. Since the frame rate is 30fps, then the delay time is 6.6 milliseconds which is almost real-time to our eyes. By using the finite state machine and the counter, the accuracy rate is improved.

5 Applications As we have claimed that the given proximity-aware algorithm can be used in the object controlling and the virtual space navigation. In the object controlling task, people can magnify or reduce the virtual object by pushing or pulling the camera. The application is shown in Figure8. In the virtual space navigating tasks, the algorithm can be used for going forward or back in the scene which is especially useful for the games(Figure4).

Fig. 8. The left one is the former image of a cube, when the user pushes the camera forward, the cube’s size will increase as is shown in the right picture

6 Conclusions In this paper, we have proposed and compared two proximity-aware algorithms. From the experiment’s results, we conclude that the algorithm two has a better performance. In order to support the large distance zoom and improve the accuracy rate, we take in a finite state machine. Comparing to the traditional mouse and keyboard operation, our methods are much more natural and easier to learn. Our approaches are real-time and have high accuracy rate. Our methods can be used in the object control and virtual space navigating tasks to fulfill the zoom function.

Acknowledgements Specialized Research Fund for the Doctorial Program of Higher Education, China, No.20050003048 and this research is also supported by the Nokia Research Center.

140

M. Fan and Y. Shi

References 1. Bouguet, J.V.: Pyramidal implementation of the Lucas Kanade Feature Tracker Description of the algorithm. Intel. Corporation Microprocessor Research Labs (1999) 2. Harrison, C., Anind, K.D.: Lean and Zoom: Proximity-Aware UserInterface and Content Magnification. In: Proc. CHI, pp. 507–510 (2008) 3. ISO. Ergonomic requirements for office work with visual display terminals (VDTs) Requirements for nonkeyboard input devices. ISO 9241-9 (2000) 4. Shi, J., Tomasi, C.: Good features to track. In: Proc. IEEE Comput. Soc. Conf. Comput. Vision and Pattern Recogn., pp. 593–600 (1994) 5. Sohn, M., Lee, G.: ISeeU: camera-based User interface for a handheld computer. In: ACM MobilCHI 2005, pp. 299–302 (2005) 6. Wang, J., Canny, J.: TinyMotion: Camera Phone Based Interaction Methods. In: Proc. CHI 2006, pp. 339–344 (2006)

A Study on the Design of Voice Navigation of Car Navigation System Chih-Fu Wu, Wan-Fu Huang, and Tung-Chen Wu Graduate School of Industrial Design, Tatung University 40 JhongShan North Road, 3rd Section Taipei 10452, Taiwan [email protected]

Abstract. This study tries to find the designing blind spots of the voice prompt function in the current car navigation systems and make improvement suggestions. The experimental plan was implemented through videotape analysis of the voice-prompt mode, referring to Urban Road Classification Regulations and the questionnaire survey results. Driving simulation tests were conducted with 15 experimental subjects, 13 road combinations, and 3 running speeds, and different prompt modes which were run synchronously were also included. Compared with the present mode (prompt time is determined by distance.), the newly-designed mode (prompt time is determined by running speed.) significantly improved driving performance and reduced mental workload. When driving on a main artery with fast lanes and slow lanes, adding a changing-lane prompt with a clear sound to the system can help increasing the driving accuracy rate. Keywords: navigation systems, voice prompt function, driving accuracy rate.

1 Introduction Following the development of science and technology, car navigation systems have increasingly been used by a lot of drivers. Car navigation systems provides drivers with information about how to get from one place to another in a turn-by-turn format such as distance to the turn, the name of the street to turn onto and turn direction [1]. However, whether a car navigation system is pre-installed on the dashboard or setup on the windshield of the driver, the driver inevitably needs to move the eyesight from the road ahead to a 3.5 to 8-inch LCD display of the automobile navigation system. Such distraction from the road is one of the main causes of traffic danger [2,3]. Nowadays, car navigation systems provide information through not only monitor displays but also voice prompt messages, so as to reduce the time drivers spend on monitors and reduce the danger of driving. Moreover, certain drawbacks remain in voice prompt function of car navigation systems, which prevent drivers from using them as the only way to receive navigation information. However, noises on the road, chattering voices of the passengers or sounds made by other vehicles, as well as music, radio and other factors may all interfere with navigation information. If voice playbacks are overlapping, it may confuse drivers and cause accidents. Unclear J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 141–150, 2009. © Springer-Verlag Berlin Heidelberg 2009

142

C.-F. Wu, W.-F. Huang, and T.-C. Wu

navigation information may cause accidents, especially when drivers need to switch lanes from one multi-lane road section to another, or to enter special terrains such as tunnels or overpasses. Due to the limitation of information processing, attention needs to be allocated when a person is multitasking, and his/her mental workload is increased as a result. Attention resources of drivers can be categorized into visual resources, operative resources, mental-workload resources, acoustic resources and information ranking for analytical purposes [4]. Drivers depend largely on the visual modality for drivingrelated information [5]. While mental workload varies along with changes in acoustics and operation, several studies claim that if a message is instant, simple and to be reacted immediately after being received, vocal prompt is suggested; if a message is complicated, lengthy and not to be reacted immediately after being received, visual prompt is preferable [6]. Some studies suggest that names of road are not necessary to be provided by voice prompt in navigation system, because these are not easily comprehended by drivers immediately [7]. Given that driving is metal-requisite, the timing of information to be provided should be limited: if the information is provided too early, drivers may forget such information; if the information is provided too late, drivers may not have enough time to execute relevant tasks. Although mobile navigation systems have been increasingly popular, issues with regard to navigation have appeared as well. Foreign or domestic studies in this area, however, are rather scarce. This study, as a result, aims first at pointing out the issues regarding comprehension of displayed information and voice prompt information from currently available car navigation systems through questionnaires to drivers. Further, with driving simulation tests, driving patterns of drivers passing intersections of different kinds of roads are recorded and compared with the results of questionnaires. Finally, through the combination of variables such as monitor displays, driving reminders and timing of voice prompt, further designs and verification are conducted for the study to be applied in the design of voice prompt systems.

2 A Survey of Currently Car Navigation Systems The study started from investigating products on the market. Three types of car navigation system from four large manufacturers were selected, and installed to proper positions above the wind outlet of air-conditioner in a real car. After setting the route, the car was driven on the same way in Taipei city and the playing mode was recorded in video taping for three runs. From the content of the video, the syntax of driving reminders and broadcasting timing with matching road classification principle were analyzed for designing questionnaire. Questionnaires are used in the early analysis so as to understand issues regarding the provision of information to drivers from currently available automobile navigation systems. They are described below. • Sample Setting: Three most representational car navigation systems (Mio CT720,

PaPaGo R12, TomTom ONE ) were selected from the products available in the market. Two CCDs were used to simultaneously record the status on the road ahead and the graphic displayed and voice played by the navigation systems. In total, three rounds of tests were conducted. Grammar analysis is employed based on

A Study on the Design of Voice Navigation of Car Navigation System

143

the abovementioned samples. After the analysis, Mio-Tech’s product is the most complicated one, with driving information provided before a decision point in a pattern of a intersection in front + b please drive close to X + c please turn + d enter Y + e leave Y + f drive Z, in which “a” and “f” are broadcasted before every intersection, “b” is broadcasted when driving straight is required, “c” is broadcasted when a turn needs to be made at an intersection, “d” and “e” mean entering or leaving special roads, such as overpasses or tunnels, are required, “X” means turning left or right, “Y” means special terrains, such as overpasses and tunnels, and “Z” means the name of the name to drive on. The information provided by PaPaGo is less complicated than that by Mio-Tech but in a similar pattern. The difference is that it does not provide names of the roads to drive on (namely, without the item, f drive Z). TomTom uses the simplest way to broadcast, which only presents one simple sentence such as “PLEASE TURN LEFT” or “PLEASE DRIVE STRAIGHT”. • Categorization of and Regulation on Downtown Roads: Based on the analysis of voice prompt functions of the abovementioned mobile navigation systems, navigation scripts do not adapt in accordance with different navigation contents for different roads. Though additional contents are made for special roads such as overpasses and tunnels, they are too limited. In order to fully understand the issues regarding the provision of driving information with mobile navigation systems, roads need to be fully understood. In accordance with Urban Road Classification Regulations relevant regulations in Taiwan [8], according to the property of the road service such as the speed limit, traffic lane number, lane width, traffic control devices, etc. condition, downtown roads may be categorized into expressway, major road, sub road and service road. Combination of roads and questionnaires are used to understand issues regarding comprehensibility of voice prompt of mobile navigation systems. • A survey of questionnaires interview: The information from the navigation system was presented on paper in the questionnaires with voice prompts being played. As for the graphic information, 13 representative graphic images were captured during the actual uses of the current navigation systems. Through cluster analysis with Urban Road Classification Regulations, with road names changed to prevent the effect from subjects’ memories, they were then re-made into the testing images for the questionnaires with CoreDraw. There are 12 questions in 3 categories including awareness, UI preference, and habit. Only those who had used a navigation system before need to answer the habit-related questions. The investigation was carried out for forty-three subjects (20-55 years old, M=28.4) who owned driving licenses and had driving experiences. Fifteen of forty-three subjects had driven an automobile with a car navigation system. From the above analysis, we can summarize the finding as follows: (1) Products of Mio and PaPaGo have better comprehensibility compare to that of TomTom for those who haven’t used car navigation systems to get more detail the drive information. (2)On the roads with fast lanes and slow lanes, starts at about 20~30 meters from intersections, there might be double solid white line as “keep in lane” lane marking to

144

C.-F. Wu, W.-F. Huang, and T.-C. Wu

divide the car flows toward the same direction for the sake of safety. If a right-turn needs to be made at this kind of intersections, drivers must switch to the slow lane on the outside before passing where the double solid white line starts. (3) The voice prompts in the current products are usually played 500 meters before critical intersections. The same content is played four times. The last time begins about 20~30 meters before intersections are also present graphically in Fig. 1. But drivers usually care only about the last prompt and ignore the first three. Therefore there would be actions like braking, slowing down, etc. right after the last prompt.

Fig. 1. Some voice prompt models of the currently available products (on expressway)

3 Driving Simulation Test The key point in the design of human-machine interface is to discuss the interaction relationship among practical users, objects and environment. As to this study, the real experience in car driving was the best in the sense of reality. However, due to lots of factors interfering the road driving and experimental variables not easy to control, simulation driving was carried out to assure the consistency of test conditions in the experiment. Virtual reality scenes were constructed by thirteen combinations of turns in the road that created from the roads classification and regulation on downtown roads. 3.1 Participants A total of 15 graduate students of Tatung University at the age of 20 or more (20-46 years, M=26, SD=3.2) and were paid to participate in the experiment. All participants

A Study on the Design of Voice Navigation of Car Navigation System

145

have valid driving licenses and driving experience. All participants have normal or corrected visual acuity. 3.2 Apparatus In the present study, the simulated vehicle cab, a 1990 FORD TELSTAR car, included all the normal automotive displays and controls found in an automatic vehicle. The simulator used two Pentium IV PCs to control the simulator, scenario respectively and one Notebook to show visual in-car navigation system. The scenario graphics were projected onto a projector screen located about 2 m in front of the driver’s seat to produce 60 *40 degree field of view. The steering, throttle and brake inputs were connected to the PC that controlled the simulator software. The car navigation system is set up on in right-front of the driver, the height and location of which is around the ventilator of the air-conditioning system as the visual display. For a driver of normal height, the in-vehicle display was 15degrees below his/her straight-ahead plane and 20 degrees to the right. A speaker in front of the passenger seat provided auditory information in the form of a digitized human female voice with a speech rate of ~150 words/min and sound effects.

Fig. 2. One scene of driving simulation ware/hardware

&

Structure of virtualization control soft-

GW InstruNet Data Retrieving System is used for transformation and retrieval of data. Data is retrieved and input into Excel and SPSS for further determination. For analyses of the broadcasting timing of navigation systems, speakers are connected with the GW Instruments data retrieving box at Vin+ and Vin-, and hence voltage values vary as sounds are played. 3DSTATE World Editors developed by Morfit was used to generate the virtual environment. Scene data can be recalled by using Visual Basic together with the animation engine of 3D Developer Studio for Visual. Structure of Virtualization control software/hardware is depicted in Fig. 2. 3.3 Experimental Designs Two different tests were assessed. One was driving simulation tests of currently available car navigation system with modifying switch-lane reminder (hereafter referred to

146

C.-F. Wu, W.-F. Huang, and T.-C. Wu

as Test1), and the other was driving simulation tests for newly-design evaluation car navigation system (hereafter referred to as Test2). Test1 was a 3 *2*2*2 mixed-factors model that compared results by speed on the driven road (three levels: 70 km/hr in expressway , 50 km/hr in major road, as well as 20 km/hr in service road ), way of voice prompt (two levels: complicated broadcasting patterns: Mio-Tech, simple broadcasting patterns: TomTom), monitor display(two levels: on, off), reminders of switching lanes (two levels: on, off).When the last factor is set to be “off”, the mode being tested is the currently available voiceprompt mode; when it’s set to be “on”, the mode being tested is a new trial. The way it works is that a clear and short sound will be played before a voice prompt to remind divers paying attention to the voice content which is about to be played. The effect is like that of the broadcasts to look for somebody in the railroad stations. The evaluation on the newly-designed mode (prompt time is determined by running speed instead of prompt time is determined by distance.) was performed in Test2. In this voice-prompt mode, the reaction distance should be [speed (km/h)*2.5second+30] meters. The number of tests was reduced because of the experience from Test1. Only the factors “speed on the driven road” was kept the same as planned. The numbers of levels were reduced to 1 for the two factors “monitor display (one level: on)” and “way of voice prompt (one level: simple broadcasting patterns: TomTom)”. But the number of levels for the factor “reminders of switching lanes” was increased to 3 (three levels: display, sound, voice). The display mode shows a red arrow icon as a guide to switching lanes (to the left or right lane) on the navigation screen. The sound mode is the same as the “on” mode in Test1. The voice mode has voice prompts for switching lanes (to the left or right lane). Dependent variables were based on both objective and subjective measures. Whether it would be successful or not while turning into scheduled road would be regarded as the object evaluation criterion. Subjective measures were obtained using a modified five-point scale NASA TLX workload assessment. 3.4 Procedure First, a ten-minute explanation is provided with regard to the test purpose and procedure and to instruct the subjects on how to manipulate in the driving simulation environments with the wheel, accelerator and brake pedal (those were connected to the serial port of the personal computers). Then the subject entered the test phase where he will receive the 36 navigation scenarios in pre-planned different sequences but were counterbalanced to prevent any learning/order effect. A short break was taken if necessary after each scenario trial. For each navigation test, all subjects were required to drive through the designated routes to reach an identical destination. All participants were told to arrive the destination in accordance with the driving information provided by available mobile navigation models, to drive along the route as accurate as possible, and to maintain the car moving in a controllable speed. Chart of changes in voltage are drawn with Excel based on data retrieved, which are analyzed to determine whether subjects have driven in accordance with the rules of the test. Finally, subject filled out a NASA TLX subjective evaluation questionnaire to describe his/her subjective feelings toward different experimental conditions. It took ~120 min for participants to complete the present study.

A Study on the Design of Voice Navigation of Car Navigation System

147

4 Results Data collect for present study were analyzed by inferential statistics of ANOVAs. The LSD treatment contrast tests were used for post-hoc comparison. Table 1. Test1 Resuls -1 ( Currently Available Products )

Driving Speeds 70km/h Expressway 50km/h Major Road 20km/h Service Road

Broadcasting Patterns Mio TomTom Mio TomTom Mio TomTom

Driving Correctness(%) Monitor Display On Monitor Display Off 13.3% 0.0% 26.7% 0% 40.0% 0.0% 93.3% 26.7% 96.7% 36.7% 100% 46.7% 100% 93.3% 100% 90% 100% 86.7%

Table 1 shows that only 26.7% of drivers can correctly leave the expressway by only listening to the final navigation information 20~30 meters before the event decision point. On major roads, 96.7% of drivers can make turns correctly with the monitor on, and 36.7% with it off. On service roads, whether monitors are on or off, the correctness rates are more than 90%. As TomTom’s navigation contents are shorter and require less time for broadcasting within the same distance, they tend to be fully listened to by drivers compared with those of Mio-Tech’s. Table 2. Test1 Resuls -2 ( Switch-Lane Reminding with sound effect)

Driving Speeds 70km/h Expressway

50km/h Major Road

20km/h Service Road

Switch-Lane Reminding Sound On Off On Off On Off

Driving Correctness 78.3% 68.3% 83.3% 58.3% 80.0% 85.0%

Chi-Square Inspection Value DOF Sig. 1.534

1

0.215

9.076

1

0.003**

0.519

1

0.472

From Table 2, it shows that when driving on major roads with median strips, driving correctness rate is 83.3% with switch-lane reminders and 58.3% without such, while “p” is less than 0.05 under the chi-square test. Table 2 shows descriptive statistics and a chi-square inspection sheet of driving with different broadcasting patterns. There is neither significant difference between these two broadcasting patterns, nor between whether the monitor display is on or not. Two major disadvantage of currently available car navigation systems based on the conclusion from early analysis: (1) The timing of final voice prompt on function

148

C.-F. Wu, W.-F. Huang, and T.-C. Wu

cannot be changed based on the car speed. As a result, when the driving speed is rather high, navigation information may not be fully broadcasted, and driving action may not be made in time by drivers. (2) It would result in the same problem if drivers don’t fully understand the switch-lane prompt from the navigation system when driving. But when using reminding sounds on some sections of the roads, as they may be misunderstood as “PLEASE MARK A TURN”, when no switch-lane reminder is provided, driving correctness is higher, nevertheless. Table 3. Test2 Results (Switch-Lane Reminding with Different Ways)

Driving Speeds 70km/h Expressway 50km/h Major Road 20km/h Service Road

Switch-Lane Reminding Sound Voice Display Sound Voice Display Sound Voice Display

Driving Correctness 93.3% 100% 73.3% 86.7% 100% 53.3% 73.3% 80.0% 93.3%

Chi-Square Inspection Value DOF Sig. 5.850

2

0.05*

10.833

2

0.004**

2.218

2

0.345

Table 3 shows descriptive statistic data of correctness on three kinds of tested roads. Correctness rates on the three kinds of roads are generally the same, while different driving reminders contribute to changes in correctness rate. Based on the significance derived with chi-square tests, on expressways and major roads, three different kinds of reminders affect driving correctness significantly, while there is no significant difference on service roads. Subjective measure Subjective grading items include quality of information provision, suitability of broadcasting timing, necessity of monitor display and comprehensibility of information. Through single-variable analyses of variance, it shows that broadcasting patterns designed by this study are better than those of the currently available products and can reduce drivers’ mental workload.

5 Discussion and Conclusions During driving and navigating, drivers have to monitor the car by searching the environmental information and shifting attention from one information source to another [10,11], and drivers depend largely on the visual modality for driving-related information [5]. According to the multiple resource theory, in a heavily loaded visual display environment, an auditory display will improve time-sharing performance [12]. When driving conditions and information are complicated, drivers may have more difficulty in filtering and remembering useful information presented by an auditory display because of the memory interference problem [13]. Similar results were found in the present study.

A Study on the Design of Voice Navigation of Car Navigation System

149

Although voice information aims at reducing the occasions in which drivers move their eyesight from the road, deficiencies in voice navigation functions may nevertheless become a safety concern. However, drivers may have difficulty in paying attention to the auditory display all the time. The timing of the appearance of prompt messages is a keypoint. Four of these findings are worth summarizing: • Prompting Issues of Voice Navigation Systems: Insufficient prompted information and unsuitable timing of prompting result in misunderstanding of voice navigation information. Unclear information, such as not reminding drivers to switch lanes, leads to driving actions not being accomplished when event decision points, such as turning and switching lanes, are passed. In terms of prompting timing, drivers usually neglect the first three prompted contents before the decision, while the final information is not prompted based on driving speeds and may not be heard by drivers in time. • Switch-Lane Reminder : With regard to reminding sounds for switching lanes: (1) On major roads, such sounds can effectively improve driving correctness (pi>1 (1)

Places are under this condition selected when they maximize the distance to the previous and the subsequent place. Figure 4 illustrates the algorithm: illustration I is the initial situation - a FP and the elements E1 , E2 and the places P1 , ..., P8 . P2 , P6 are at significant locations and considered as fixed places. In illustration II we can see the segmentation of the FP into three parts. The algorithm now selects the fixed places P2 , P6 as representatives for the first and the third segment, only the middle segment has a choice of optimizable places. The algorithm maximizes the distance between

Enhancing the Accessibility of Maps with Personal Frames of Reference

205

P2 and P3 , P4 , P5 and betweenP6 . In this case P4 is selected (see illustration III in Figure 4).

3

Visualization


Maps are visual representations of the environment, i.e. we need to visualize the personal frame of reference defined by the selected places.
Maps are intended to support the wayfinding process dynamically, i.e. they have to cover typical requirements of wayfinding assistance during all phases: the communication of survey knowledge and the support during navigation. To support cognitively adequate, we require specialized representations reflecting the task with matching visualizations ([13, e.g.]). This does not only hold for principle configurational issues, but also for the incorporated symbols. Entities on maps should either follow a cartographic convention or in case of non-existence new cartographic symbols have to be created ([14, e.g.]). Up to the knowledge of the author, there are no available symbols for personally meaningful places and pointers to them. It is beyond the scope of this work, to analyze the requirements of these new kind of visual elements. We decided to use a straightforward visualization: in our examples and illustrations we will depict places as circles (illustrations) and solid dots (generated maps) and the pointers to them as lines. 3.1

Visualization of Places on the Route

The course of the FP of the route is schematized by means of the discrete curve evolution (DCE), see [15]. DCE simplifies the geometry by successively removing geometric control points from the shape information. Applying the DCE without explicitly considering the places, the coordinates of the places are no longer guaranteed to be located on the course of the route. I.e., we have to compute the schematization of FP differently; the schematization has to consider and preserve the position of places as the route is described in relation to them. In the following algorithm we sketch the positioning of places (and branches to remote places) on a schematized path: 1. In the first step we segment the route at the points where the selected places (or the branching points) are located. Illustration I in Figure 6 shows the initial situation. Illustration II depicts the segmentation of the route at the places P1 , P2 , P3 into self-contained units. 2. In the second step, we schematize each segment by means of the DCE (see [15]). This will transform the places into fixed points of the curve and are not removed by the DCE. This step is important as we do not consider any other constraints, required by the DCE to declare fixed points. 3. In the third step we compile all segments again to one coherent FP. This can be done straightforwardly, as the positions of the contact points (places) are not altered in each segment (see Illustration III in Figure 6).

206

F. Schmid

a)

b)

c)

d)

Fig. 5. Selecting places. a) The map of Figure 1 with the places 1, 2, 3 (bold black dots). Note the different schematization of the FP in b), c), d) due to the integration of places. b) the FP is only one single segment: place 1 is selected, as it is on-route. c) FP consists of two segments: place 1 (first segment), and place 2 (second segment) is selected. Place 2 branches off at a significant location. d, FP consists of three segments: all places are selected (each is within one of the three segments). Note the different compression rates: b) is the most compact map as it utilizes the on-route place 1. c, requires more space as it points to place 2 (although FP is compressed with the same ratio as b)). d) is significantly larger, because place 3 would intersect the unfamiliar part of the map on the bottom if we would apply the same minimization as in b) and c). This illustrates the effect of local rendering constraints on map compression (see Section 3.2).

3.2

Visualization of Branch-Off Places

The question now is how we can visualize places which are not located on the route. In this case we need the differentiation between the two basic assistance types: communication of veridical survey knowledge and navigation support. In the following, we will differentiate between the two scenarios and show some examples for respective
Maps. Furthermore we have to propagate new local visualization constraints to the global map rendering.

Enhancing the Accessibility of Maps with Personal Frames of Reference

207

Fig. 6. Schematization with places as fixed points: illustration I shows the initial situation, II the segmentation with the places as start and endpoints of the segments, III the result of the schematization and compilation

Reference Frame Visualization for Survey Maps. Survey maps are means to visualize the embedding of the route within the environment in a geographic veridical manner. I.e., the real geographic relations amongst the elements of the route, and between the route and the surrounding environment have to be represented according to a allocentric (geographic) frame of reference. Survey maps are intended to communicate overview information for a certain route. However, in
Maps, the familiar part of the route is always schematized and minimized (as otherwise no compression could be achieved), but the configuration of all elements is not altered. The schematization of the known paths works as described in Section 3.1: the places (and the branches to remote places) serve as constrained supporting points of the familiar part of the route. The crucial step for the veridical visualization of remote places are the paths to them: we depict the path within the familiar environment with the same degree of schematization and minimization as the route starting at the branching point at the route and ending at the configurable street network depth k, which is the number of expanded vertices from the branching point towards the place (see place 2 in Figure 5). Reference Frame Visualization for Navigation Maps. Navigation maps are intended to support the wayfinder during the wayfinding process. As discussed in [9], the maps follow the egocentric, bottom-up approach of mobile wayfinding maps: the part of the route which is ”in-front” of the wayfinder (in terms of travel direction), is at the top of the display, the remaining parts at the bottom. A number of studies showed that people encode turning actions usually as 90 degree angles ([16, 17, e.g.]). The mental representation of turning actions are termed wayfinding choremes (see [17] and Figure 7 for an illustration). Branchings to remote places are, due to the egocentric and direct experience in the real environment mentally encoded as wayfinding choremes [17]. For this reason we depict the branch to the remote place by means of a choreme. We replace the real angle α with the angle α of the respective choreme. However, as the spatial configuration at the particular point can be complex, the choreme holds between the segment of the route before the branch and the branch in travel direction (see Figure 7). This reflects the perception and the expectation of the wayfinder in the FP.

208

F. Schmid

a)

b)

Fig. 7. The chorematization of places for the navigation perspective: a) depicts the set of wayfinding choremes. b) depicts a turn at a place within the FP (left is the initial configuration), on the right we see the navigation perspective of the intersection. The intersection is rotated in travel direction and the angle α is replaced by the angle α of the corresponding wayfinding choreme.

a)

b)

c)

Fig. 8. Communication of local rendering constraints to the global minimization procedure: a) depicts the global minimization distance h. b) illustrates the minimization constraints of the visual elements of the FP, it is not possible to apply the global minimization factor to the FP. In c) we see the global minimization based on the local minimal distance k. See also d) in Figure 5 for an example.

Communicating Local Rendering Constraints for Global Rendering.
Maps minimize the familiar part of the route by moving the closest points of the convex hulls of the unfamiliar environment Ui , Ui+1 towards each other; so far the distance-to-keep was determined by a threshold h (see Figure 8). Now, with the integration of places, we have additional visualization constraints: a visual intersection of the used symbols has to be avoided, thus a distance threshold k between all elements has to be preserved. We can resolve the constraints by following procedure: 1. In the first step we determine the global minimization factor min(h) for the FP between Ui , Ui+1 , such that dist(Ui , Ui+1 ) = h. 2. In the second step, we determine the closest pair of elements by means of the euclidean distance (in Figure 8 it is E1 , P1 ). 3. We then compute the minimization factor min(k) for the familiar part, such that dist(E1 , P1 ) = k. 4. If min(k) ≥ min(h), we apply min(h) to the familiar part, min(k) otherwise.

Enhancing the Accessibility of Maps with Personal Frames of Reference

4

209

Conclusions


Maps are personalized wayfinding maps for devices with small displays like mobile phones. By means of relating a route to familiar parts of the environment,
Maps can achieve significant visual compression rates by at the same time preserving the individual accessibility. The clarification of the embedding in the environment is based on the integration of a personal frame of reference, the places and paths a users usually visits and travels. However, due to the schematization of the familiar parts of a route, the integration of personally meaningful places require basic considerations about the selection of places, as well about their visualization within
Maps. The selection process for places is based on three considerations: structural significance, segmentation and distribution, and minimalistic visual appearance. The visualization considers the support of two basic requirements for wayfinding maps: the communication of geographic veridical survey knowledge and navigation support. We introduced the selection algorithm, as well as the visualization primitives for both map use conditions. Additionally we discussed the requirements to communicate the additional rendering constraints for integrated places and how we can resolve the conflict between local and global minimization attempts.

Acknowledgments This work has been supported by the Transregional Collaborative Research Center SFB/TR 8 Spatial Cognition, which is funded by the Deutsche Forschungsgemeinschaft (DFG).

References [1] Dillemuth, J.: Spatial cognition with small-display maps: Does the sum of the parts equal the whole? In: Association of American Geographers Annual Meeting, Boston (April 2008) [2] Reichenbacher, T.: Mobile Cartography Adaptive Visualization of Geographic Information on Mobile Devices. PhD thesis, University of Munich, Institute of Photogrammetry and Cartography, Munich, Germany (2004) [3] Rist, T., Brandmeier, P.: Customizing graphics for tiny displays of mobile devices. Personal Ubiquitous Computation 6(4), 260–268 (2002) [4] Baudisch, P., Rosenholtz, R.: Halo: a technique for visualizing off-screen objects. In: CHI 2003: Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 481–488. ACM, New York (2003) [5] Harrie, L., Sarjakoski, L.T., Lehto, L.: A variable-scale map for small-display cartography. In: Proceedings of the Joint International Symposium on GeoSpatial Theory: Processing and Applications, Ottawa, Canada, July 8-12 (2002) [6] Agrawala, M., Stolte, C.: Rendering effective route maps: improving usability through generalization. In: SIGGRAPH, pp. 241–249 (2001) [7] Parush, A., Ahuvia, S., Erev, I.: Degradation in spatial knowledge acquisition when using automatic navigation systems. In: Winter, S., Duckham, M., Kulik, L., Kuipers, B. (eds.) COSIT 2007. LNCS, vol. 4736, pp. 238–254. Springer, Heidelberg (2007)

210

F. Schmid

[8] Ishikawa, T., Fujiwara, H., Imai, O., Okabe, A.: Wayfinding with a gps-based mobile navigation system: A comparison with maps and direct experience. Journal of Environmental Psychology 28(1), 74–82 (2008) [9] Schmid, F.: Knowledge based wayfinding maps for small display cartography. Journal of Location Based Systems 2(1), 57–83 (2008) [10] Schmid, F., Richter, K.F.: Extracting places from location data streams. In: UbiGIS 2006 - Second International Workshop on Ubiquitous Geographical Information Services (2006) [11] Waller, D., Loomis, J.M., Golledge, R.G., Beall, A.C.: Place learning in humans: The role of distance and direction information. Spatial Cognition and Computation 2(4), 333–354 (2001) [12] Couclelis, H., Golledge, R.G., Gale, N., Tobler, W.: Exploring the anchor-point hypothesis of spatial cognition. Journal of Environmental Psychology 7(2), 99–122 (1987) [13] Klippel, A., Richter, K.F., Barkowsky, T., Freksa, C.: The cognitive reality of schematic maps. In: Meng, L., Zipf, A., Reichenbacher, T. (eds.) Map-based Mobile Services - Theories, Methods and Implementations, pp. 57–74. Springer, Berlin (2005) [14] MacEachren, A.M.: How maps work: representation, visualization, and design. Guilford Press, New York (1995) [15] Barkowsky, T., Latecki, L.J., Richter, K.F.: Schematizing maps: Simplification of geographic shape by discrete curve evolution. In: Freksa, C., Brauer, W., Habel, C., Wender, K.F. (eds.) Spatial Cognition II - Integrating abstract theories, empirical studies, formal models, and practical applications, pp. 41–53. Springer, Berlin (2000) [16] Tversky, B., Lee, P.U.: Pictorial and verbal tools for conveying routes. In: Freksa, C., Mark, D.M. (eds.) COSIT 1999. LNCS, vol. 1661, pp. 51–64. Springer, Heidelberg (1999) [17] Klippel, A.: Wayfinding choremes. In: Kuhn, W., Worboys, M.F., Timpf, S. (eds.) COSIT 2003. LNCS, vol. 2825, pp. 320–334. Springer, Heidelberg (2003)

Augmented Interaction and Visualization in the Automotive Domain Roland Spies1, Markus Ablaßmeier1, Heiner Bubb1, and Werner Hamberger2 1

Institute of Ergonomics, Technical University of Munich, Boltzmannstraße 15, 85747 Garching {spies,bubb}@lfe.mw.tum.de, [email protected] 2 AUDI AG, Development HMI, 85045 Ingolstadt [email protected]

Abstract. This paper focuses on innovative interaction and visualization strategies for the automotive domain. To keep the increasing amount of information in vehicles easily accessible and also to minimize the mental workload for the driver, sophisticated presentation and interaction techniques are essential. In this contribution a new approach for interaction the so-called augmented interaction is presented. The new idea is an intelligent combination of innovative visualization and interaction technologies to reduce the driver’s mental transfer effort that is necessary between displayed information, control movement and reality. Using contact-analog head-up displays relevant information can be presented exactly where it is needed. For control, an absolute natural and direct way of interaction is delivered by touch technologies. However, to leave the eyes on the road, the driver needs haptic feedback to handle a touchpad blindly. Therefore, the touchpad presented in this contribution, is equipped with a haptic adjustable surface. Combining both technologies delivers an absolutely innovative way for in-vehicle interaction. It enables the driver to interact in a very direct way by sensing the corresponding environment on the touchpad.

Keywords: head-up display, touch, haptic feedback, interaction, automotive, augmented reality.

1 Introduction To keep the increasing amount of information in modern vehicles easily accessible and controllable for the driver and also to minimize his mental workload, sophisticated presentation and interaction techniques are of major importance. In the car domain, error-prone situations often occur regarding the human-machine interaction with different in-car applications, as the driver often has a certain mental workload [1] by combining displayed information, interacting with different input devices and transferring it to the reality. J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 211–220, 2009. © Springer-Verlag Berlin Heidelberg 2009

212

R. Spies et al.

Automobile manufacturers have already introduced a couple of innovative visualization and interaction techniques. The head-up display for example provides information according to the situation in reality like e.g. navigation information or the current vehicle speed. This visualization method enables the driver to keep his glance on the road and reduces accommodation problems. In the recent past a lot of scientific work has been done on presenting the information in a contact-analog way in the HUD [2,3,4,5,6]. A further challenge of car manufactures today is the increasing amount of comfort functions in modern vehicles e.g. navigation, media and communication systems and soon internet services. To keep all these systems controllable while driving, car producers have integrated these functions in menu-based central infotainment systems which are mostly controlled by one multifunctional input device. Currently, many different solutions of such control devices are available on the market. These solutions can be divided into two different groups: first, an integrating approach represented by touchscreens and second an approach separating display and control element, e.g. turning knobs or joysticks in the center-console. These systems are often extended by voice control and a lot of recent research publications deal with multimodal in-vehicle infotainment interaction [7,8,9]. Further research activities handle with innovative ideas for control elements [10,11], e.g. flexible touch-sensitive surfaces for in-vehicle interaction [12,13]. This contribution delivers a highly new approach for intuitive and workload-reduced in-vehicle interaction by combining innovative technologies for visualization and control. In the following chapter, the theoretical background concerning controlling and displaying menu systems in dual task situations is discussed and the required technical background is given. Afterwards, the new approach of augmented interaction is explained and a couple of demonstrative examples for future realization are given.

2 Background The following chapter reflects the relevant theoretical background to analyze the parallel tasks of driving and menu operating. Therefore, the driver as well as the vehicle are considered as parts of the human-machine control loop. This consideration should uncover the need for action. 2.1 Ergonomic Reflection of the Driving Task The driver-vehicle interaction can be described in a conventional control loop as it is shown in Fig. 1 [14]. The left side of the picture shows the task as the input of the loop. The system components are the driver and the vehicle equipped with an infotainment system. The output of the whole system is permanently verified by the driver and adjusted if necessary. In case of menu control while driving it is about a dual task situation which can cause interferences between the two parallel tasks. Inattention, distraction, and irritation occur beside mental treating internal problems

Augmented Interaction and Visualization in the Automotive Domain

213

as a consequence of the high workload resulting from a superposition of the tasks, which will become manifest in an increased error potential and in erroneous operations of the systems [15]. According to Bubb the driving task can be classified in primary, secondary and tertiary task [16]. The primary task only consists of real required driving operations. These are segmented into navigation, steering, and stabilization. Choosing the route from departure to destination corresponds to the navigation task. Steering includes, for example, lane changes due to the current traffic situation. User interaction with the car to navigate and steer is called stabilization. These tasks are essential for a safe control of the car, and therefore have highest priority while driving. Secondary tasks are operations that are not essential to keep the vehicle on the track. Examples are the turn signal, honking, and turning the headlights up and down. Tasks not concerning the actual driving itself are categorized as tertiary tasks, e.g. convenience tasks like adjusting the temperature of the air condition or communication and entertainment features.

Fig. 1. Control loop of driver vehicle interaction

While working on a desktop PC, the user can predominantly execute her or his operations in a concentrated way, as there is no dual task competition. However in the car domain, often error-prone situations occur regarding human-machine interaction with different in-car applications, as the driver often has a certain mental workload. This basic stress level is due to the execution of so-called primary and secondary tasks, and may be increased by environmental impacts, like the conversation with a co-driver. If the driver interacts, e.g. with a communication and infotainment system in such a stress phase (tertiary task), he probably will be distracted from driving. A reason for this is founded in the human information detection. The fact that the primary as well as the tertiary task share mainly the visual channel [17] (Fig. 1) leads to gaze movements away from the road and consequently traffic blindness occurs. Here it is necessary to either provide additional channels (e.g. haptic) to transfer information or to merge it with the reality to avoid gaze movements away from the driving scene.

214

R. Spies et al.

Another reason for the interferences between driving and tertiary tasks is the mental overload effected by compatibility problems between displayed information, control movements and reality [18]. The following part explains a theory of the human information processing and presents ergonomic rules to avoid such incompatibilities. 2.2 Human Information Processing During system design it is important to identify workload bottlenecks and overload. As the human operator is a central part of a human-machine system, the correction of these workload problems is necessary for the safe and efficient operation. The Multiple Resource Theory of Wickens [19] allows predicting when tasks will interfere with each other or can be performed in parallel. If the difficulty of one task increases a loss of performance of another task will be the result. Wickens describes several processing resources. All tasks can be divided into the following components (see Fig. 2): there are the encoding, central processing and responding stages. The visual and auditory components are input modalities. The central processing component describes the level of information processing required. The responding component consists of manual or vocal actions. Stages Encoding

Central processing Responding manual

vocal

es ns

verbal

po es R

spatial visual

auditory

spatial

od C es

verbal

Fig. 2. Wickens’ Multiple Resource Theory (MRT) Model [20]

Wickens postulates that using multiple information channels increases the mental capacity and reduces interference effects [19]. To avoid additional mental effort a compatible design between information visualization, control element and reality is important [21]. To adjust the user interface according to the task, the task itself has to be analyzed. The ergonomic analyses according to Bubb provide a method to identify the content of the task according to space (dimensionality) and time [22]. Menu control is a two-dimensional task, what means that in principle, movements on the surface are possible in two directions (e.g. selecting items on a display). To guarantee

Augmented Interaction and Visualization in the Automotive Domain

215

a compatible interaction, ergonomic solutions require a two-dimensional input device for such a task. Turning knobs in some current automotive solutions are onedimensional, what means that e.g. moving maps on a display is very inconvenient and requires a mental transfer effort from the user. Concerning the information presentation, ergonomic solutions also require compatibility between information content, the user’s mental model and reality. Summing up the mentioned theoretic facts above, it is possible to derive visualizing as well as control concepts. 2.3 Conclusion To come up to the claim of a two-dimensional input device which can be controlled blindly, provide additional information via another channel and can be mounted in the center console for ergonomic reachability, the following concept proposes a touchpad with a haptic adjustable surface. The idea is that every kind of structure which is displayed at a screen can be felt on the touchpad surface for orientation. Elevated elements on the touchpad (e.g. buttons) can be sensed and pressed. This input device enables a direct intuitive interaction with a menu-based system. For a direct, non-distracting information presentation this concept suggests a contact-analog head up display. This technology enables to project information in reality. The technical realization of both concepts is described bellow.

3 Technical Solutions The following chapter describes the technologies and current application fields of the preliminary considerations and consequences mentioned above. 3.1 The Contact-Analog Head Up Display As described, a very innovative display technology for cars is the head-up display (HUD). The HUD projects information directly into the driver’s visual field. HUDs were pioneered for fighter jets in the early 70s and later for low-flying military helicopter pilots, for whom information overload was a significant issue, and for whom alter their gaze to look at the aircraft's instruments could prove to be a fatal distraction. In future, HUDs are likely to become more common in vehicles. Further recent developments are done to give the driver contact-analog information. A couple of recent research work shows the potential of giving the driver contact-analog information. The spectrum consists of giving speed and distance information [23], night vision information [3] as well as contact-analog navigation information [24]. There exist several technical approaches for contact-analog HUDs. An early solution is delivered by Bubb [4], where a spatial impression is obtained by bending the display in a certain way according to the first optical illustration law, which produces a virtual display lying on the ground. Further developments given by

216

R. Spies et al.

Schneid and Bergmeier bring this principle closer to automotive capability [2,3,6]. A completely different approach is delivered e.g. by DENSO [5]. Their suggested realization is based on the effect of stereoscopy via two monocular head-up displays covering an image distance of 2m. The drawback of such a solution is that either the head has to be in a fixed position or a highly accurate eye-tracking-system has to be used, what makes this solution extremely cost-intensive. Moreover, if there is a delay in the system caused by the data transferring computing system, sickness will be the consequence. Fig. 3 shows an example of contact-analog driver information.

Fig. 3. Example for contact-analog visualization

3.2 The Haptic Touchpad Touchpads are currently used exclusively in the notebook domain to move a cursor over a display and select items. To make it useable for automotive applications the idea is to give an additional haptic structure on the surface for orientation. Some similar approaches with just a few mechanical, moveable elements or simulated vibration feedback are already published [11,12,13]. Another possibility to realize such an adjustable haptic touchpad surface is to use the so called braille technology. Fig. 4 shows a few examples of haptic displays realized in the braille sector for blind people, using piezo-activated pins.

Fig. 4. Examples for Braille displays [25,26]

Augmented Interaction and Visualization in the Automotive Domain

217

4 Augmented Interaction – A New Approach for In-Vehicle Interaction This chapter explains a new approach for in-vehicle interaction and presents demonstrative use-cases for the combination of the presented display and control techniques. 4.1 Definition of Augmented Interaction A new way of interaction in the automotive domain can be reached combing both introduced innovative technologies from chapter 3. The structured surface of the touchpad (see section 3.2) enables direct mapping of the displayed information with virtual objects represented by the contact-analog HUD (see section 3.1). The driver interacts with the touchpad by sensing the corresponding environment, activates and manipulates functions directly by pressing and moving on the sensed elevated objects. As a consequence, the mental workload can be reduced through the simple and direct cognitive mapping (see chapter 2). Real and virtual objects are fused together. This kind of interaction will be called augmented interaction. 4.2 Illustrative Use-Cases for Augmented Interaction In the following, two examples for direct augmented interaction are given to illustrate the potential benefit of the suggested concept. 4.2.1 POIs along the Route An application of navigation systems is that the driver wants to get further information about a points-of-interest (POI) in his direct surrounding area while driving. With state-of-the-art interfaces, the driver realizes an interesting building outside his vehicle then searches the corresponding POI on the digital map on his central display inside the car and finally selects and activates the information with the central control element depending on the infotainment system. The concept of augmented interaction in this contribution enables the driver to feel his surrounded environment including relevant points-of-interest on the haptic touchpad surface. The driver places his finger on the interested objects and afterwards he gets the real object highlighted contact-analog by the HUD. To avoid an information overflow just the currently touched objects are highlighted. After the relevant object is selected the driver can directly activate further information by pressing the sensed elevated element on the touchpad (Fig. 5). 4.2.2 Adaptive Cruise Control The second example described in this context stands for interaction with an adaptivecruise-control (ACC). The potential of contact-analog information concerning distance and speed control is already shown in several recent research contributions (e.g. [24]).

218

R. Spies et al.

i

i

Fig. 5. Examples for highlighting POIs along the route

Fig. 6. Examples for adjusting the distance bar

Combined with ACC the contact analog HUD can give a direct system feedback merged with the reality. The drawback of current systems is to adjust speed and distance. Currently, there are a lot of different HMI-variants available on the market. Some control elements are mounted at a drop arm; some are integrated in the steering wheel. All these solutions require a certain mental transfer effort for control and are hard to handle while driving. The augmented interaction solution here projects the environment in front of the vehicle in bird view on the touchpad surface, so that the driver can feel the distance bar with his finger and directly adjust the distance to the front vehicle by moving this bar. A direct visual feedback is given by the

Augmented Interaction and Visualization in the Automotive Domain

219

contact-analog HUD by a green break bar (Fig. 6). As a consequence the vehicle adjusts the distance to the front vehicle according to the new position of the break bar. If the user wants to chose an illegal distance the system can give information via the HUD by giving the bar a red color for example.

5 Summary and Conclusion In this contribution a challenging new way of in-vehicle interaction is presented. The so-called augmented interaction is expected to reduce the mental effort for the driver while interacting. This is reached by the intelligent combination of innovative display methods together with new control technologies. For the driver-adjusted display of information a contact-analog HUD is used to present the information directly where it is needed. For an adequate control of this information a haptic adjustable touchpad is used that maps the reality to the surface. As a result the driver can now interact directly by touching the corresponding haptic surface. To realize the presented approach of augmented interaction for vehicles a lot of further aspects have to be considered. Contact-analog HUDs are still not automotive capable respecting packaging size, sensor technologies and field-of-view. Also the haptic configurable touchpad is a very complex element and still very space and costintensive. Additionally, a lot of effort has to be invested to map the reality to the haptic surface and to realize the presented use-cases. Therefore, elaborate computer processing is affordable. After the prototype is finished studies are necessary to evaluate this approach according to driving scenarios and prove the presented theoretical benefits of the combination of these new technologies.

References 1. Praxenthaler, M.: Experimentelle Untersuchung zur Ablenkungswirkung von Sekundäraufgaben während zeitkritischer Fahrsituationen, Dissertation, Universität Regensburg (2003) 2. Bergmeier, U.: Methode zur kontaktanalogen Visualisierung von Fahrerassistenzinformationen unter automotive-tauglichen Gesichtspunkten. In: Produktund Produktions- Ergonomie – Aufgabe für Entwickler und Planer. Kongress der Gesellschaft für Arbeitswissenschaft, vol. 54, pp. 125–128. GfA Press, München (2008) 3. Bergmeier, U., Bubb, H.: Augmented Reality in vehicle – technical realization of a contact analogue head-up display under automotive capable aspects; usefulness exemplified through night vision systems. FISITA World Automotive Congress (F2008-02-043), Munich (2008) 4. Bubb, H.: Untersuchung über die Anzeige des Bremsweges im Kraftfahrzeug, BMVg – FBWT 76-7, pp. 198–202 (1979) 5. Koji, N., Hiroshi, A., Nobuaki, K.: Denso Corporation, Windshield display for active safety, FISITA World Automotive Congress (F2006D105), Yokohama (2006) 6. Schneid, M.: Entwicklung und Erprobung eines kontaktanalogen Head-up-Displays im Fahrzeug, Dissertation, TU München (2009)

220

R. Spies et al.

7. Geiger, M.: Berührungslose Bedienung von Infotainment-Systemen im Fahrzeug, Dissertation, TU München (2003) 8. Mischke, M., Hamberger, W.: Multimodalität im Dualtask - eine Lösung für die Probleme der Sprachbedienung. In: Prospektive Gestaltung von Mensch-Technik-Interaktion, vol. 7. Berliner Werkstatt Mensch-Maschine-Systeme, Berlin (2007) 9. Hummel, S.: Akzeptanzentwicklung bei multimedialen Bedienkonzepten, Dissertation, TU München (2008) 10. Sendler, J.: Entwicklung und Gestaltung variabler Bedienelemente für ein Bedien- und Anzeigesystem im Fahrzeug, Dissertation, TU Dresden (2008) 11. Vilimek, R.: Gestaltungsaspekte multimodaler Interaktion im Fahrzeug Ein Beitrag aus ingenieurspsychologischer Perspektive, Dissertation, Universität Regensburg (2007) 12. Doerrer, C.: Entwurf eines elektromechanischen Systems für flexible konfigurierbare Eingabefelder mit haptischer Rückmeldung, Dissertation, TU Darmstadt (2003) 13. Hayward, V.: Change of Height: An Approach to the Haptic Display of Shape and Texture Without Surface Normal. In: Experimental Robotics III. Springer Tracts in Advanced Robotics, pp. 570–579. Springer, New York (2003) 14. Bubb, H., Seiffert, R.: Struktur des MMS. In: Bubb, H. (ed.) Menschliche Zuverlässigkeit, pp. 18–20. ecomed – Fachverlag, Landsberg (1992) 15. McGlaun, G., Lang, M., Rigoll, G., Althoff, F.: Kontextsensitives Fehlermanagement bei multimodaler Interaktion mit Infotainment- und Kommunikationseinrichtungen im Fahrzeug. In: Nutzergerechte Gestaltung technischer Systeme, Tagungsband VDIFachtagung USEWARE, VDI-Bericht 1837, pp. 57–65. VDI-Verlag Düsseldorf, Darmstadt (2004) 16. Bubb, H.: Fahrerassistenz primär ein Beitrag zum Komfort oder für die Sicherheit? VDI Nr. 1768, pp. 25–44. VDI-Verlag (2003) 17. Rockwell, T.H.: Eye Movement analyses of visual information acquisition in driving: an overview. Paper presented at the North Carolina State University, Raleigh (1971) 18. Bullinger, H.J.: Ergonomie Produkt und Arbeitsplatzgestaltung. B.G. Teubner Verlag, Stuttgart (1994) 19. Wickens, C.D.: Engineering Psychology and Human Performance. Columbus, Merrill (1984) 20. Wickens, C.D.: Attention and Situation Awareness, Ph.d. thesis, Univ. Illinois (1996) 21. DIN EN ISO 10075-2, Ergonomische Grundlagen bezüglich psychischer Arbeitsbelastung, Teil 2: Gestaltungsgrundsätze (2000) 22. Bubb, H., Schmidtke, H.: Systemstruktur. In: Schmidtke, H. (ed.) Ergonomie, vol. 3. Auflage, Hanser Verlag, München (1993) 23. Assmann, E.: Untersuchung über den Einfluss einer Bremsweganzeige auf das Fahrverhalten, Dissertation, TU München (1985) 24. Tönnis, M., Lange, C., Klinker, G., Bubb, H.: Transfer von Flugschlauchanzeigen in das HUD von Kraftfahrzeugen. In: Proceedings 22. Internationale VDI/VW Gemeinschaftstagung Integrierte Sicherheit und Fahrerassistenzsysteme, Wolfsburg (2006) 25. See by Touch, http://see-by-touch.sourceforge.net/index.html 26. TIM - Der Blindenmonitor, http://www.blindenmonitor.de

Proposal of a Direction Guidance System for Evacuation Chikamune Wada, Yu Yoneda, and Yukinobu Sugimura Graduate School of life Science and Systems Engineering, Kyushu Institute of Technology Hibikino 2-4 Wakamatsu-ku, Kitakyushu, Fukuoka, 808-0196, Japan [email protected]

Abstract. In this paper, we propose a device that indicates the direction to evacuate. Our proposed system, which would present the direction through the tactile sensation of the head, could be used in no visibility environment such as filled smoke. This paper describes a feasibility of our proposed system and indicates problems to be solved. Keywords: Evacuation, Smoke, Direction, Guidance, Tactile sensation.

1 Introduction Generally speaking, there is an electrical sign to escape in case of fire at a hotel. However, if there is a massive poll of smoke we will not able to see the sign and we will feel fearful while evacuation. Moreover, if the sign can not be seen or is turned off because of flat battery during night time we will not be able to know which way we have to go in the dark. In such this environmental condition, the sighted person will become similar to a blind person because visual information can not be available. Incidentally, we went about to develop a new device which presented the visually impaired person with the obstacles direction and distance. As for direction, we proposed a new method to present the direction of obstacle by tactile stimulation and revealed its effectiveness [1]. If using our experimental results, we hypothesize a person would know easily an escape direction by tactile stimulation. Also, we hypothesize a person will be able to escape without using visual information while smoky dark environment. However, our previous experimental results [1] were obtained under the condition in which subjects were fixed on a chair and they were not allowed to walk. Then, in this paper, we investigated whether or not subjects were guided by tactile stimulation to a designated direction while walking and we reported feasibility of our system to help evacuation.

2 Direction Displaying Method Figure 1 shows one of experimental results when our direction displaying method was used. In this experiment, first tactile stimulation was presented, next the subjects were J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 221–227, 2009. © Springer-Verlag Berlin Heidelberg 2009

222

C. Wada, Y. Yoneda, and Y. Sugimura

Horizontal plane

Stimulation

＝

Head

Response (βh)

Stimulation (αh)

≒

Vertical plane

Stimulation (αv)

Response (βv)

Fig. 1. Direction displaying method [1]

asked to imagine the direction of stimulated point, lastly the subjects were asked to point the direction with their fingers. In this experiment, we used an air stimulation as tactile stimulation because that the air stimulation did not cause uncomfortableness. Left figure shows the angle between tactile stimulation and face forward direction (alfa), and right shows the angle between pointed direction and face forward direction (beta). From the upper figures, alfa is equal to beta. That is, if there is an obstacle at 30 degrees rightward for example, tactile stimulation should be presented 30 degrees right on the head. Therefore, the blind person will be able to imagine where the obstacle is. Or, if you intend to guide the blind person to the 30 degrees rightward, tactile stimulation should be presented 30 degrees right on the head. However, these results were obtained when the head of subjects were fixed, that is to say, it was not known whether or not these results applied to the condition in which the head moved while walking. Then, we investigated a feasibility.

3 Feasibility Study In order to present a designated direction, tactile stimulation point should be changed according to the movement/rotation of head while walking. Firstly, we made a head movement measuring unit by combining gyro sensor and digital magnetic compass. The gyro sensor and digital magnetic compass was small and light weight, so the unit does not become obstructive to evacuate. On the basis of the head movement speed, optimal gyro sensor and compass were selected.

Proposal of a Direction Guidance System for Evacuation

＋

223

Z-axis Gyro sensor Digital magnetic compass

X-axis

Y-axis #7

Vibratory motor

#6

#5 #4

#3 #2 #1

#1 left 90 degrees #2 left 60 #3 left 30 #4 0 #5 right 30 #6 right 60 #7 right 90

Face forward direction (= 0 degree)

Fig. 2. Our guidance system

Next, air stimulation was used as tactile stimulation in previous experiment. However, it is impossible to carry the experimental setup which includes an air compressor and electrical valves because of its heaviness and big size. And we think the experimental results are applicable to any tactile stimulation such as vibration. Then, we used vibration caused by vibratory motor because a vibratory motor was light weight and easy to control. From the preliminary psychophysiological experiment, the arrangement of vibratory motors on the head was decided. Then, we have made a system by combining vibratory motors, gyro sensors and digital magnetic compass. The outline of our system was shown in figure 2. In this system, seven vibratory motors were arranged in every 30 degrees on the head and three gyro sensors and a compass were put on the head. Guided walking experiment was done in order to investigate whether or not the subjects could be guided to a designated direction. In this experiment, the four Z X

Transmitter

Y

Z Y

Z Y X

Measurement area Indication

Start

Fig. 3. Experimental setup

224

C. Wada, Y. Yoneda, and Y. Sugimura Measurement area Stand 0.96 m

(3)

Theta3 D3 End

Transmitter D2

Theta2 (2)

2-3 m

D1

Y Theta1

(1)

Start

X Z

Fig. 4. Experimental protocol

subjects who wore blindfold were asked to walk to the direction which was pointed out by vibration of a vibratory motor. Figure 3 shows experimental setup. The head position while walking was measured by a magnetic three-dimensional positioning sensor(Fastrak). A transmitter of Fastrak was put on a wooden frame and two receivers were put on the front and back of the head so that head center was calculated. A subject was asked to walk following a vibration on the head. Figure 4 shows the experimental protocol. First the subject started to walk at the start position(indicated by (1) in figure 4 and indicated by start in figure 3). At this time, #4-vibratory motor was vibrating, that is to say, vibration was added to the forehead center and the subject walked to the forward direction. After walking for about 2 or 3 meters, another vibratory motor started to vibrate((2) in figure 4 and indication in figure 3). The subject was asked to turn to the direction which the vibration indicated. In other words, the subject was asked to turn until vibration moved to the forehead center. Before executing experiment, the subjects were not trained at all. Figure 5 shows one result. These graphs show overhead view of track of head position. Vertical axis shows the head movement in backward and forward direction, whereas horizontal axis shows the movement in leftward and rightward direction. After starting the experiment, the center motor(#4) vibrated so the subjects went straight ahead and the track moved from the bottom to the center of the graphs. Next, in graph (a), #5-motor which was set at right 30 degrees vibrated. After the motor vibrated, the subject turned to the rightward. The graph shows that the direction of track changed and the angle change was about 30 degrees. Similar results were obtained for 60 and 90 degrees and were obtained for other subjects. From these results, we thought that our system could guide the subject to the designated direction in this experimental condition.

Proposal of a Direction Guidance System for Evacuation

120 0

30 degrees

100 0 80 0 60 0

]m [m 後 前

40 0

Vibration starts

20 0 0 0

-20 0

20 0

400

600

800

1000

-40 0 -60 0

Walking starts

-80 0 -100 0 [m m ] Movement [mm] 左右 (Leftward-Rightward)

(a) 30 degrees rotation 1200 1000

60 degrees

800 600

]m 400 [m 200 後前 0 -200

Vibration starts 0

200

40 0

600

8 00

1000

-400

Walking starts

-600 -800

右 [mm] Movement [mm] 左 (Leftward-Rightward)

(b) 60 degrees rotation 800 600

Vib ration starts

400

m]m [ 後前

90 degrees

200 0

0

200

400

600

800

-200 -400 -600

Walk ing starts

-800 [mm] Movement [mm]左右 (Leftward-Rightward)

(c) 90 degrees rotation Fig. 5. Track of guided walking

1000

225

226

C. Wada, Y. Yoneda, and Y. Sugimura

4 Problems to Be Solved The experimental results showed a possibility that our system could guide a person to the designated direction without training. However, an integral error would become problem if the head moved complicatedly. For example, figure 6 shows the angle difference between an angle measured by our head movement measuring unit and an angle obtained by two Fastrak receivers which were put on the front and back of head. Figure 6 shows one result when a subject walked with his/her intention for 30 seconds. The value changed for positive or negative but maximum value was about 50 degrees. Nevertheless to say, this angle difference depended on the walking condition, but it was difficult to imagine there was no angle difference. Then, we have to make a method which will decrease the angle difference.

Angle difference[deg]

20 10 0 -10 0

10

20

30

-20 -30 -40 -50 -60 Time[sec]

Fig. 6. Change of angle difference while walking 100

2 Right rotation

80

40

Forward movement

20 0 3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

-20 -40 -60

Value of acceleration [m/s^2]

Shaking head

60

Head angle[deg]

Turning

-80 -100

0 Time [s]

Fig. 7. Head movement and change of acceleration value

Proposal of a Direction Guidance System for Evacuation

227

The reason why the angle difference was occurred is thought to be accumulation of integral calculation error of gyro sensor. If small change of head movement such as head shaking which is not related to walking is avoided, the angle difference may become small. Then, we investigated whether or not small head movement could be detected by using two acceleration sensors which were put on both head sides. A subject executed the following four actions; shaking head, walking forward, rotating rightward and turning right. Figure 7 shows a result. Dots indicate head direction while two lines indicate values of two acceleration sensors which were put on left/right of head. This graph indicates that neither acceleration values changed much for the forward movement, though they did change in a similar way for the rightward rotation and in a similar way periodically for the right turn, and there was no relationship between them for the head shaking. Then, we thought small head movement during the forward movement might be detected, but more researches are necessary.

5 Conclusion In order to evacuate in no visibility environment, we proposed a method which present direction by tactile sensation. Our results showed the possibility of our proposition but showed problems to be solved. We would like to solve the problems and make a useful evacuation system in the near future.

Reference 1. Asonuma, M., Matsumoto, M., Wada, C.: Study on the use Air Stimulation as the Indicator in an Obstacle Avoidance System for the Visually Impaired. In: SICE 2005, MA2-14-2 (CD-ROM) (2005)

A Virtual Environment for Learning Aiport Emergency Management Protocols Telmo Zarraonandia, Mario Rafael Ruiz Vargas, Paloma Díaz, and Ignacio Aedo Universidad Carlos III de Madrid {tzarraon, mrrvarga, pdp}@inf.uc3m.es, [email protected]

Abstract. This paper presents a virtual environment designed to enhance the learning of airport emergency management protocols. The learning is performed in an informal manner, with each learner playing a different role in a particular emergency simulation. Learners interact within the virtual environment, managing the available information and following the steps prescribed for each type of emergency in the Airport Emergency Plan of the Spanish Civil Defence Organization. The simulation can be run in different modes of difficulty, and can be used as a learning tool as well as an evaluation tool to measure the accuracy of the learner's actuation within the protocol. It can also support standalone training having some of the emergency roles played out by the computer. The virtual environment has been built using DimensioneX, an open source multi-player online game engine. Keywords: Virtual environment, emergency, game engine, simulation.

1 Introduction Airports should always guarantee a fast and effective response to any kind of emergency. All the efforts and decisions should be perfectly coordinated to minimize the consequences whenever an airplane accident, natural disaster or any other emergency interferes in the normal progress of the aeronautical operations. Following this objective, airport emergency plans are specified in order to compile all the norms, measures and procedures that should rule all the actions taken by each of the actors involved in the emergency management, before, during and after the emergency. Learning such protocols and plans is then crucial. The use of games and simulation in the field of training has been widely explored [1, 2] due to the facilities they provide to recreate virtual environments that could support situated learning. Situated learning happens when knowledge and skills are applied in a realistic context [3], in our case a real emergency. Situation games or simulations bring a great level of realistic immersion and promote situated learning, which, according to the literature presents the following advantages: • Apprentices are aware or the actual conditions in which they should apply their knowledge and abilities. If the situation is a simulation of real incident children can dive into the real problems that are going to have to face. J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 228–235, 2009. © Springer-Verlag Berlin Heidelberg 2009

A Virtual Environment for Learning Aiport Emergency Management Protocols

229

• Real situations create an atmosphere that produces a greater motivation and engagement of users. • Apprentices understand better the implications of their knowledge or ignorance. • There is a better understanding of the knowledge structure that facilitates its application in real situations. For instance, in the military area an adapted version of the commercial game DOOM has been used to train US Marine fire teams [4], while the first person shooter game Unreal Tournament serves to implement a simulation of a first-responder to a mass casualty airline accident scene [5]. Simulation and virtual environments are also common tools for training and education in the aeronautical area, and they have also been applied to the area of emergency response training. [6, 7]. This paper presents a virtual environment designed to enhance the learning of airport emergency management protocols. An open source multiplayer game engine has been used to implement a virtual world where different types of airport emergencies can be simulated. Simulation participants play the role associated to their position, interact within each others and manage the emergency information as it becomes available. The virtual environment can be used as a learning tool as well as an evaluation tool to measure the accuracy of the learner's actuation according to the protocol. From the wide range of aspects of an airport emergency, our application will focus in the management of the emergency information and the communication between the different roles involved. The rest of the paper is organized as follows: first, the objectives and scope of the airport emergency plans is outlined. Second, the Airport Emergency Management simulator (AEM-Simulator) is presented, its interface is described and a example of use is provided. Next, the use of the simulator with different interaction devices is analyzed, and the characteristics of the game engine used for implementing the virtual environment are also detailed. Finally, some conclusions and future work lines are exposed.

2 Airports Emergency Plans The Spanish Directorate of Civil Defense and Emergencies (DGPCE - Dirección General de Protección Civil y Emergencias) of the Ministry of Interior defines an Airport Emergency Plan as “a set of pre-defined norms, measures and coordinated procedures which aim to minimize the impact of an emergency situation that could take place in the airport or in the response areas defined in the emergency plan” [8]. The emergency plan predetermines the degree of participation of all of the dependencies and services involved in an emergency by clearly stating their functions and the procedures to follow before, during and after the emergency. The operability of the plan is guaranteed by defining each of their responsibilities, the chain of command, the communication procedures, the coordination system and the specific actuation procedures. An emergency plan defines the set of methods and procedures to follow for a particular number of emergency types that are classified depending on whether they involve aeroplanes or not, if the aeroplane is flying or not, or by using the airport zone where the emergency takes place. Any other type of emergency which differs from

230

T. Zarraonandia et al.

the ones considered in the plan will be treated using the procedures of the closest typified emergency. The plan defines the actuations directives before (phase 1), during (phase 2) and after (phase 3) an emergency takes place. Phase 2 is the very essence of the plan and for each of the emergency types and each of the services involved in it a directive record is defined. The directive record defines the hierarchy, the person in charge, the pseudonym to be used in radio communications, the means of communication and an ordered explanation of the task to be performed until the emergency situation is under control. In order to guarantee the efficacy of the plan the staff involved are regularly trained in their specific functions, and the reliability and effectiveness of the plan is evaluated through periodic emergency exercises and practices. As a result of those experiences the plan is constantly reviewed and new norms and improvements are introduced whenever it is consider necessary. The aim of our work is the development of a tool to facilitate the training and learning of the different plans of actuations of an airport emergency plan, reducing the number of real simulations in order to decrease costs. Moreover, the simulation can also be used to test the own plan efficacy and detect flaws or inconsistencies since the behaviour during the emergency can be recorder and analysed in the aftermath to learn from errors and help to build and institutional memory [9].

3 AEM Simulator Currently, the training on the actuations plans is performed through tabletop exercises and full-scale practices. During the first ones, each of the participants play the role associated with their position, following the procedures established in the plan of actuation of a particular emergency. Participants use phones to communicate to each other the decisions adopted, ask for information, confirm data, etc. This scenario can be improved by making use of a graphic and interactive multimedia environment. The virtual environment can be used to support the communication between the different participants/players while keeping track of all the action adopted during the emergency procedure. The actions performed by one actor at a particular stage in the emergency can be compared to the one established by the specific plan of actuations. This can serve both as a training tool, providing the participants with suggestions or feedback on the appropriateness of her actions, and as an evaluation tool, providing a measure of how well each of the participants has followed the procedure. Moreover, the whole procedure can be recorded to be studied afterwards and learn from errors, something that can’t be done using full-scale exercises and phone-based communication. Following this idea a virtual environment for training in the emergency management protocols has been implemented. Trainers connect to the virtual environment and play the emergency role associated with his/her position. Currently the protocols implemented are the ones for “Control Tower Unit”, “Advance Command Post”, “Principal Command Post”, “Airport Fire and Rescue Service” (AFRS), “Airport Health Service” (AHS) and “Airport Coordination Centre” (ACC) for an emergency type of “Incident in a flying aeroplane”. It is expected that in the future all the roles

A Virtual Environment for Learning Aiport Emergency Management Protocols

231

could be played by either real user or the computer, allowing individual and whole team training on the protocols. At present users can only play the roles of “Control Tower Unit”, “Advance Command Post” and “Principal Command Post”. The emergency simulation can be played in different modes providing a different range of feedback and tips to the user. This way using the easiest mode, “Step by step training”, users will be provided with feedback for each of the actions taken, indicating the compliance with the actual action specified in the actuation plan. On the other hand, in the “Evaluation” mode, no feedback will be provided until the end of the simulation. Once the simulation is finished the participant will be presented with a score based in the deviation between her actuation and the one specified in the actuation protocol. To produce this score, the relevance of the information gathered, accuracy of the communications with the other participants, unnecessary contacts, time spent, etc, will also be considered. When more than one participant plays the simulation a team-score is also computed to evaluate the actuation of the whole group. This simulator has been designed following a participatory design process in which experts on civil protection take part. Experts provide descriptions of the functionalities and characteristics of an ideal application for airport emergency management. Taking as a start point their opinions and the traditional table-top exercises used for training, an initial design of the simulator was produced. The same experts collaborate on its refinement during subsequent meetings until the version here described was produced. Keeping the use of simulator simple was a major goal of the process. 3.1 Interface Description Fig 1 shows a screenshot of the virtual environment. The numbers depicts the different sections in which the simulator screen is divided: information section (1), role section (2), communication section (3), action section (4), auxiliary screen section (5), message section (6), event section (7) and feedback section (8). The participants use the information section to store relevant information about the emergency as it becomes available. Whenever they consider a piece of information is relevant, they can assign one of the slots of this section to save it. This is achieved by selecting the keyword corresponding to that information from the drop-down menu on the left side of the slot. Once a slot has been assigned, he/she can type the value of the data directly in the right hand side box, or obtain the information from other participants. The status colour can be used to indicate whether the data has been computed by the user, received from another role or confirmed. The role section provides a list of the roles involved in the emergency management. Different colours are used to specify if a particular role is played by the computer or by a real player. The communication section provides five options to support the communication between the emergency roles: send, request, confirm, not available and error. Buttons in this section are used in combination with the information and role section to compose and send messages. For instance, whenever a user needs to send data he/she will select the data slot in the information section, select the role or roles he/she wants to receive the data, and finally push the “Send” button. The messages section will then display the composed message and the information section of the selected roles will be refreshed to show the data received. A similar process will be followed when a user requires information or confirmation from another role.

232

T. Zarraonandia et al.

Fig. 1. Screen sections of the Airport Emergency Management Protocols Simulator

In the action section the user will find buttons specific for the actions of the role he/she plays. For instance, user who plays the “Tower Control Unit” role will find one button to fire the emergency alarm and other one to order the recording units custody. On the other hand, the one who plays the “Principal Command Post” role will find buttons to establish the grade of emergency and the coordinates of the command position. The auxiliary screen section will be used to support these role-specific actions, being populated with role-specific maps, templates for generating reports, etc. Depending on the action performed, the event section may display a message to inform of the action taken to some or the rest of the users. Finally, the feedback section provides participants with appropriate hits and advice about the actions they should undertake at each step of the emergency. Hints will be more frequent and precise depending on the game mode selected. 3.2 Example Figure 2 shows a screenshot of a particular moment of the simulation, which can be used to track the users’ actions. The screenshot corresponds to the view of the “Tower Control Unit” during the first stages of the “Incident in a flying aeroplane” emergency management and playing the simulation in a “Step by step” mode. The emergency procedure starts when the air traffic control unit receives a “fire in cabin” notification message from an aeroplane commander (1). The AEM simulator automatically assigns a slot of the information section to store the data received (2). Following the plan of actuation, the user asks the commander for confirmation of the incidence (3), and next the corresponding confirmation message is received (4). The red colour next to the incidence data in the information section reflects this confirmation. Next, as can be seen in the events section, the person responsible of the Tower Control Unit user have fired the alarm to alert of the incidence to the Airport Fire and Rescue Service (AFRS), the Airport Health Service (AHS) and the Airport Coordination Centre (ACC) (5). The three services should send confirmation of the alarm

A Virtual Environment for Learning Aiport Emergency Management Protocols

233

Fig. 2. Screenshot of the Airport Emergency Management Protocols Simulator

notification before to trigger communication with the Tower Control Unit. At this moment only the ACC and AHS sent acknowledgments (6). As he waited for those confirmations the traffic control started to gather information about the incidence from the commander. He assigned slots for data about the identification and type of aeroplane, coordinates and time of the incident and number of passengers (7), and composed a “Request” message (8). The commander sent the requested information (9) which was populated by the simulator to the information section (10). As the game mode selected was “Step by step” the feedback section of the screen reflects the different tips provided at each step of the emergency (11). 3.3 AEM-Simulator Interaction Devices Work has been carried out to explore different possibilities of interaction with the AEM-Simulator. On the one hand, given that the only requirement to run the AEMSimulator is a common web browser, and that the actions are carried out by simply clicking on the screen, the use of mobile devices or PDAs to follow the training becomes an obvious option to be investigated. In practical terms, this means that the application can be used in a wide range of environments. On the other hand, to identify drawbacks and inconsistencies in the emergency plans it would be useful to offer a turn-based game mode, in which all the main players work together using the same device, thus facilitating the free exchange of views and opinions. Interactive whiteboards lend themselves to this type of use of the AEMSimulator, as an ideal medium for collaborative activities such as these (Figure 3). 3.4 The DimensioneX Game Engine The AEM-simulator has been implemented using DimensioneX [10], a free-open source multiplayer game engine provided under the GNU General Public License.

234

T. Zarraonandia et al.

Fig. 3. Use of a AEM-Simulator with an interactive whiteboard

DimensioneX provides a software kit for developing and running adventure games, multiplayer games, multi-user real-time virtual worlds and turn-based games. Among many other features, DimensioneX provides online multiplayer capability, multimedia support, game maps and player tracking, game saving and events handling. The game engine is actually a Java servlet. The games developed for this engine can therefore be run in any servlet container, typically Tomcat. Players connect to the game via a conventional web browser without any additional software required. DimensioneX provides a script language for specifying virtual worlds; this is, to describe the rooms, links, themes, characters and events that can take place during the game. Worlds’ descriptions are stored in plain text files with “.dxw” extension, and processed by the game engine to produce an HTML document. Players interact via browser within each others as well as with the rest of the game elements. As a result of these interaction the world state becomes modified, which in turn triggers the creation of new HTML documents which reflect these changes. The world state is stored in the server and continually updated. We have chosen this engine due to the facilities it provides for implementing the communication between the different players which is a most important requirement in our domain. Modifying the source code of the engine would be relatively straightforward if that were ever necessary, and it is also worth noting that the programming language is simple.

4 Conclusions and Future Work This paper presents a virtual environment to facilitate the training and learning of protocols for the management of emergency in airports. The virtual environment has been implemented making use of an open source game engine named DimensioneX. Users can connect to the simulator via browser, without installing additional software on the machine for players. The project is on its first stages. Currently, the “Incident in a flying aeroplane” emergency plan has been implemented to be play in the “Step by step” mode; this is, providing participants total guidance about each of the actions they should performed

A Virtual Environment for Learning Aiport Emergency Management Protocols

235

at every moment. The next step will be to complete the implementation of the “Evaluation” mode. This will allow starting to test the application with real users and validate its usefulness as a training tool. At the same time work is carried out to implement the rest of the airport emergency plans defined by the Spanish Civil Defense Department. Moreover, the system has to be tested with real users. Even though it has been designed with domain experts the efficacy of any interaction device can only be assessed when end users try to perform their tasks with it. Future work lines consider integrating within the AEM-simulator role specific simulators as, for instance, an Airport Fire and Rescue Services Team Management Simulator. The participant who plays the role of person in charge of these units could then be trainned in emergency protocols and, at the same time, in actions specific of his/her role as emergency analisis, strategy selection and unit personal leading. The output information from one simulator could serve as input and the other provinding a more realistic experience. Acknowledgments. This work is part of the MoDUWEB project (TIN2006-09678), funded by the Ministry of Science and Technology, Spain.

References 1. Rolfe, J., Saunders, D., Powel, T.: Simulations and Games for Emergency and Crisis Management: Simulations and Games for Emergency and Crisis Management. Routledge (1998) 2. McGrath, D., McGrath, S.P.: Simulation and network-centric emergency response, Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC), Orlando, Florida (2005) 3. Brown, J.S., Collins, A., Duguid, P.: Situated cognition and the culture of learning. Educational Researcher 18(1), 32–41 (1989) 4. Prensky, M.: True Believers: Digital Game-Based Learning in the Military LearningCircuits.com (February 2001), http://www.astd.org/LC/2001/0201_prensky.htm (11/02/09) 5. McGrath, D., Hill, D.: UnrealTriage: A Game-Based Simulation for Emergency Response. In: The Hunstville Simulation Conference, Sponsored by the Society for Modeling and Simulation International (October 2004) 6. Virtual preparation for ARFF emergencies, Industrial Fire Journal (October 2008), http://www.hemmingfire.com/news/fullstory.php/ Virtual_preparation_for_ARFF_emergencies.html (11/02/09) 7. McGrath, D., Hunt, A., Bates, M.: A Simple Distributed Simulation Architecture for Emergency Response Exercises. In: Proceedings of the 2005 Ninth IEEE International Symposium on Distributed Simulation and Real-Time Applications (DS-RT 2005) (2005) 8. Dirección General de Protección Civil Española. Subdirección General de Planes y Operaciones. Plan de Emergencia de Aeropuertos (1993) 9. Klenk, J.: Emergency Information Management and Communications. Disaster Management Training Programme. DHA (1997) 10. DimensioneX Online Multiplayer Game Engine, http://www.dimensionex.net

User Profiling for Web Search Based on Biological Fluctuation Yuki Arase, Takahiro Hara, and Shojiro Nishio Department of Multimedia Engineering Graduate School of Information Science and Technology, Osaka University 1-5 Yamadaoka, Suita, Osaka 565-0871, Japan {arase.yuki,hara,nishio}@ist.osaka-u.ac.jp

Abstract. Because of the information flood on the Web, it has become difficult to search necessary information. Although Web search engines assign authority values to Web pages and show ranked results, it is not enough to find information of interest easily, as users have to comb through reliable but out of the focus information. In this situation, personalization of Web search results is effective. To realize the personalization, a user profiling technique is essential, however, since the users’ interests are not stable and are versatile, it should be flexible and tolerant to change of the environment. In this paper, we propose a user profiling method based on the model of the organisms’ flexibility and environmental tolerance. We review the previous user profiling methods and discuss the adequacy of applying this model to user profiling. Keywords: User profile, Web search, biological fluctuation.

1 Introduction As our current life is always surrounded by Internet enabled devices, such as computers, cellular phones, PDAs and game consoles, the highly advanced information society allows us to collect information of concern far easily than the past. However, the larger the amount of information on the Web grows, the harder it becomes to find information of interest. According to a report of Google on July 2008, the number of unique URLs on the Web has already exceeded a trillion and the number of Web pages is practically uncountable. Furthermore, the number of Web pages is still rapidly growing every second. Currently, people use Web search engines to find Web pages containing their information of interest. Most search engines use link structure of Web pages to decide authoritative Web pages, which is based on the idea that authoritative Web pages contain more reliable information than the other minors. When people query a search engine, these authoritative pages are ranked high as the search result. This criterion has been really effective to enable people to find reliable information without bothering by browsing hundreds of junk Web pages. However, the authority based ranking is not enough in the current situation of information flood, since the information on the Web has become too diverse in their semantic meaning to recommend based on J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 239–247, 2009. © Springer-Verlag Berlin Heidelberg 2009

240

Y. Arase, T. Hara, and S. Nishio

only their reliability. As a result, people have to access many reliable but unnecessary Web pages to find pages that exactly match with their interests. To solve this problem, it is effective to personalize Web search results based on users' interests in addition to the current authority based ranking. For this aim, user profiling is essential. However, since the users’ interests are unstable and easy to change, user profiling is not easy task. It is apparent from the fact that although user profiling methods have been actively researched for decades, tracking users’ versatile interests is still difficult. In this paper, we propose a novel approach to realize flexible and dynamic user profiling. We adopt a model called Attractor selection based on biological fluctuation to detect users' intentions. Since the Attractor selection is tolerant to the change of the assumed environment, it is suitable to model users’ versatile interests. Meanwhile, ambient information environment is a recent hot topic, where surrounding computers and embedded sensors in the environment detect users' situations and provide functions to satisfy users' needs without users’ explicit declaration. In the precedent ubiquitous environment, users do access computers to satisfy their requirements. However, in the ambient environment, the environment does make actions to satisfy users' needs. The concept of ambient environment can be applicable to various fields, including user interface on which we are working. We define ambient user interface as interfaces that detect users' intentions and provide information according to them. We regard our personalization method as a realization of ambient interface for Web search. The reminder of this paper is organized as follows. In section 2, we introduce previous user profiling methods and discuss the differences from our approach. In section 3, we briefly introduce the biological Attractor selection model. In section 4, we propose our user profiling method based on Attractor selection, and discuss its potential to solve problems that previous user profiling methods could not settle. Then, in section 5, we conclude this paper and describe our future work.

2 Related Work There have been two directions for user profiling, as one makes use of explicit feedback from users, and the other does implicit feedback. A popular method for the former approach is asking users to input their interest, which is adopted by some portal and news sites. Another method is to ask users to assign a score to their browsed Web pages according to the strength of their interest to contents of the pages. As represented by News Dude [1], a user can specify i) whether they think the content is interesting or not, ii) if they would like to see more similar information, or iii) if they have already seen the information previously. An advantage of this approach is that extracted user profiles tend to be reliable, since users themselves input their interests. A disadvantage is that they trouble users to input their interests, and more, users have to change their profiles each time when their interests change. The latter approach uses the users’ browsing behaviors to extract user profiles. SUGGEST [2] adopts a two-level architecture composed by an offline creation of historical knowledge and an online engine that understands user’s behavior. As the

User Profiling for Web Search Based on Biological Fluctuation

241

requests arrive at this system module, it incrementally updates a graph representation of the Web site based on the active user sessions and classifies the active session using a graph partitioning algorithm. Gasparetti et al. [3] proposed an algorithm with which the system could identify the users’ interests by exploiting Web browsing histories. Claypool et al. [4] investigated different kinds of user behaviors, such as scrolling, mouse clicks, and time on page for extracting user profiles. These methods are based on learning user interests, and thus, a large amount of training data is needed and they seem to take long time to converge to reasonable profiles. Billsus et al. [1] proposed that users’ interests can be classified into two, as longterm and short term interests. The long-term interests can be regarded as the users’ intrinsic interests, and thus, seem to be stable over time. Therefore, they are easier to extract by directly asking users and using learning based methods. On the other hand, the short-term interests can be regarded as interim, reflecting users’ current interests. This feature results in difficulty of tracking change of profiles because of their versatile nature in a short period. Especially for Web search, users’ interests can be classified to the short-term interests, since users usually search information which they need, concern, and get interested in, at that time. Therefore, a user profiling method of flexible and tolerant to environmental change is suitable. For this aim, we adopt the Attractor selection mechanism, which is based on the fluctuation of organisms and realizes flexible and environmental tolerant solutions, to track change of users’ interests on Web search.

3 Attractor Selection In this section we outline the principle of Attractor selection, which is a key component in our method. The original model for adaptive response by Attractor selection is given by Kashiwagi et al. [5]. Attractor selection defines each possible situation as attractor, and evaluates the current situation to select one of better attracters in a dynamic environment. The goodness of the attractor is estimated by the activity value. While the activity is high, the system keeps staying the current attracter. On the other hand, when the situation changes and the activity gets low, the system performs a random walk to find a more suitable attractor. Because of the random walk, the system performs fluctuation. We can basically outline the attractor selection model as follows. Using a set of differential equations, we describe the dynamics of an M-dimensional system. Each differential equation has a stochastic influence from an inherent Gaussian noise term. Additionally, we introduce the activity α which changes the influences from the noise terms. For example, if α comes closer to 1, the system behaves rather deterministic and converges to attractor states defined by the structure of the differential equations. On the other hand, if α comes closer to 0, the noise term dominates the behavior of the system and essentially a random walk is performed. When the input values (nutrients) require the system to react to the modified environment conditions, activity α changes accordingly causing the system to search for a more suitable state. This can also involve that α causes the previously stable attractor to become unstable. The random walk phase can be viewed as a random search for a new solution state and when it is found, α decreases and the system settles in this solution. This behavior

242

Y. Arase, T. Hara, and S. Nishio

is similar to the well known simulated annealing [6] optimization method, with the main difference that the temperature is not only cooled down, but also increased again when the environment changes. The biological model describes two mutually inhibitory operons where m1 and m2 are the concentrations of the mRNA that react to certain changes of nutrient in a cell. The basic functional behavior is described by a system of differential equations, as the following equations show. deg

1 1

The functions syn(α) and deg(α) are the rate coefficients of mRNA synthesis and degradation, respectively. They are both functions of α, which represents cell activity or vigor. The terms ηi are independent white noise inherent in gene expression. The dynamic behavior of the activity α is given as follows.

_

1

Here, prod and cons are the rate coefficients of the production and consumption of α. The term nutrienti represents the external supplementation of nutrient i and nutr_threadi and ni are the threshold of the nutrient to the production of α and the sensitivity of nutrient i, respectively. A crucial issue is the definition of the proper syn(α) and deg(α) functions. To have two different solutions, the ratio between syn(α) and deg(α) must be greater than 2 when there is a lack of one of the nutrients. When syn(α) / deg(α) = 2, there is only a single solution at m1 = m2 = 1. The functions syn(α) and deg(α) as given in [5] are as follows. 6 2 deg The system reacts to changes in the environment in such a way that when it lacks a certain nutrient i, it compensates for this loss by increasing the corresponding mi value. This is done by modifying the influence of the random term ηi through α, as Figure 1 shows. When α is near 1, the equation system operates in a deterministic fashion. However, when α approaches 0, the system is dominated by the random terms ηi and it performs a random walk. In Figure 1, an example is given over 20000 time steps. We can see the following behavior. When both mi values are equal, the activity is highest and α = 1. As soon as there is a lack of the first nutrient (2000 ≤ t < 8000), mi compensates this by increasing its level. When both nutrient terms are fully available again (8000 < t ≤ 10000), the activity α becomes 1 again. An interesting feature of this method can be observed between 10000 < t < 13000. Here, the random walk causes the system to search for a

User Profiling for Web Search Based on Biological Fluctuation

243

Fig. 1. Biological attractor selection model

new solution, however, it first follows a wrong “direction” causing α to become nearly 0 and the noise influence is highest. As soon as the system approaches the direction toward the correct solution again, α recovers and the system gets stable again. Such phases may always occur in the random search phase. As we showed above, Attractor selection cannot always find the best answer. Instead of finding that, it tries to find a better attractor with quite a simple strategy. By this mechanism, Attractor selection ensures robustness in exchange for efficiency. Organisms can survive in such an unstable and dynamically changeable environment in the nature owing to inhering Attractor selection.

4 User Profiling Method Based on Attractor Selection As we discussed in Section 2, the problem of previous user profiling methods is that they take long time to follow the change of users’ interests. As Billsus et al. [1] proposed, users’ interests can be classified into two, as long-term and short term interests. The long-term interests are easier to extract by directly asking users and using learning based methods owing to their stable nature. On the other hand, the short-term interests are difficult to track because of their versatile nature in a short period. To detect the short-term interests, we adopt the Attractor selection scheme, which is suitable for finding solutions in a dynamically changeable environment. 4.1 Design of User Profiling Method We model users' interests as attractors and their change as environmental change in the Attractor selection scheme. We define a user’s profile as a ranking of pre-defined thirteen topics and detect the ranking using the Attractor selection scheme. 4.1.1 Definition of User’s Profile According to the definition of categories of Web sites at YAHOO! Japan [7], we decided the following thirteen topics as the users’ interest candidates.

244

Y. Arase, T. Hara, and S. Nishio

1) 2) 3) 4) 5) 6) 7) 8) 9) 10) 11) 12) 13)

News Entertainment Sports Art Health Politics Economics Life Computer Education Technology Local Others

A user profile is a ranking of these thirteen topics, as User profile = {(1| topic1),..., (13| topic13)}, where (rankk | topick) represents that rankk is the rank of the topic and topick is one of the candidate interest topics. For example, a profile of {(1| Technology), (2| computer),…,(13| Education)} means that this user is most interested in technology and computer related Web pages, while not interested in pages relating educational information. Here, we assume that categories of Web pages are given. Since our definition of users' interest topics based on the Web sites' categories, we can expect to use Web sites' categories assigned by portal sites. Additionally, since there are many previous works conducting automatic Web page categorization, it is also possible to adopt these methods to categorize Web pages as a pre-processing step. 4.1.2 Definition of Activity In Attractor selection, the value of activity α represents the goodness of the current attractor. In our case, α represents how the current user profile matches with the user’s real interests. To evaluate it, we adopt the essence of DCG (discounted cumulative gain) to evaluate the current profile. DCG is a measure of effectiveness of a Web search engine, often used in information retrieval. The premise of DCG is that highly relevant items appearing lower in a search result list should be penalized as the graded relevance value is reduced logarithmically proportional to the rank of the result. The DCG accumulated at a particular rank p is defined as follows:

Where reli is a graded relevance of the result at rank i. In our case, we cannot obtain explicitly graded values for each interest topic. However, it is reasonable to assume that users frequently browse Web pages of topics of interest and their browsing times will be longer compared to other topics. Therefore, we define reli using users’ cumulative browsing times of topics when they browsed a certain number of Web pages. The desired behavior of α is summarized as follows. If we have no information about a user, the candidate topics should be evaluated uniformly. A low α means that the current profile does not match with users’ interests and a new one should be

User Profiling for Web Search Based on Biological Fluctuation

245

detected. We should keep the value of α as 0.0 1.0. The larger α is, the detected profile better matches with the user’s interests. As a whole, after browsing N pages, activity α is determined as follows. (1)

(2)

_

Here, δ and λ are constant parameters to adjust the adaption of α, / is the fraction of the evaluation value of previously found best matching profile over that of the current profile, f(n) is the normalization factor, p is the rank of a topic which should be considered. For example, if p=3, the method evaluates only the top three interests of the user profile instead of examining the whole profile, which results in stressing the topics of the best interest. 4.1.3 Calculation of Interest Ranking As we showed in Section 0, the original form of Attractor selection is two dimensional. Leibnitz et al. extended the form to multi dimensional [8]. We adopt the multi dimensional form since we have to deal with the thirteen topic candidates. For each topic, we decide its weight using Attractor selection, and rank the topics according to their weights. Specifically, we use the following multi dimensional form. ,

deg ,

,

,

(3)

,

deg 1 √2

Here, , is the weight of topic i when being assigned rank j, , is a white noise, β and γ are constant parameters that adjust the effect of the noise term and activity α, respectively. 4.1.4 Flow of User Profiling Detection We can now summarize the basic algorithm for detecting the user profile when a user browsed N Web pages: 1. Calculate activity α based on Equation (1). 2. Initialize the set of topics that are already determined the rank as 3. Conduct the following process for each rank 1, 2, … , 13:

.

246

Y. Arase, T. Hara, and S. Nishio

a. b. c.

For each topic 1, 2, … , 13 , calculate the weight of the each topic , based on Equation (3). . Set max_i as i of the maxim value of , 1, 2, … , 13 Set j as the rank of the topic max_i and add the topic into .

4. Calculate the feedback of the decided user profile based on Equation (2). 5. Update if necessary. 4.2 Discussion The Attractor selection scheme has been applied to some research fields [8][9][10]. The first application is for multi-path routing in overlay networks conducted by Leibnitz et al [8]. They showed that their method based on Attractor selection is noisetolerant and capable of operating in a very robust manner under changing environment conditions. The authors also applied the Attractor selection scheme to routing problem in a mobile ad-hoc/sensor network environment [9]. They proved that their Attractor selection based method can operate entirely in a self-adaptive manner and that it can easily compensate for sudden changes in the topology of the network. On the other hand, Kitajima et al. applied the Attractor selection scheme to set parameters for filtering contents on data broadcasting services [10]. They assume an environment that broadcasting service providers broadcast enormous and various data to users and user clients have to filter out unnecessary data for users. By using the Attractor selection scheme to decide the order of filters, they can reduce the time for filtering in such a dynamic environment. These three previous works show the robustness of the Attractor selection scheme to the change of the environment. In addition, another advantage of the Attractor selection scheme is that it operates without explicit rules and is simply implemented by numerical evaluation of the differential equations. In our case, users’ interests are not stable in nature and versatile in a quite dynamic manner. Therefore, we can expect that our Attractor selection based method successfully track change of users’ interests in a self-adaptive manner. Furthermore, because of its simplicity of implementation and requiring nothing to store users’ histories to learn their profiles, we can implement and distribute our method as a light-weight plug-in to Web browsers, which means that users can receive a benefit of personalization very easily. It also has the advantage that our method is free from violating users’ privacies, since it takes into account the current Web page the user browsed and does not need to store their browsing histories and behaviors.

5 Conclusion and Future Work In this paper, we reviewed the works of user profiling and discussed their problems. Since most of the user profiling methods requires considerable amount of users’ browsing histories as well as the information of their behaviors on the Web pages, it seems difficult to converge to the reasonable user profiles in a practical time, as the users’ interests frequently change. We briefly introduced the Attractor selection scheme that models the fluctuation inhering in organisms, and proposed the Attractor selection based method for user

User Profiling for Web Search Based on Biological Fluctuation

247

profiling in Web browsing. We defined a user’s profile as the ranking of pre-defined topics and decide the ranking using the Attractor selection scheme. As future work, we implement a practical application and conduct user experiments to examine the effectiveness of our method. Since the users’ interests might be quite unstable and versatile, we should confirm the how quickly our method can converge to each attractor.

Acknowledgement This research was partially supported by “Global COE (Centers of Excellence) Program” and Grant-in-Aid for Scientific Research on Priority Areas (18049050) of the Ministry of Education, Culture, Sports, Science and Technology, Japan.

References 1. Billsus, D., Pazzani, M.J.: A Personal News Agent that Talks, Learns and Explains. In: The Third Annual Conference on Autonomous Agents, Seattle, pp. 268–275 (1999) 2. Baraglia, R., Silvestri, F.: Dynamic Personalization of Web Sites Without User Intervention. Communication of the ACM 50(2), 63–67 (2007) 3. Gasparetti, F., Micarelli, A.: Exploiting Web Browsing Histories to Identify User Needs. In: International Conference on Intelligent User Interfaces (IUI 2007), Hawaii, pp. 28–31 (2007) 4. Claypool, M., Le, P., Waseda, M., Brown, D.: Implicit Interest Indicators. In: The Sixth International Conference on Intelligent User Interfaces (IUI 2001), USA, pp. 33–40 (2001) 5. Kashiwagi, A., Urabe, I., Kaneko, K., Yomo, T.: Adaptive Response of a Gene Network to Environmental Changes by Fitness-Induced Attractor Selection. PLos ONE 1(1), e49 (2006) 6. Aarts, E., Korst, J.: Simulated Annealing and Boltzmann Machines. Wiley, New York (1989) 7. Yahoo! Japan, http://www.yahoo.co.jp/ 8. Leibnitz, K., Wakamiya, N., Murata, M.: Resilient Multi-Path Routing Based on a Biological Attractor-Selection Scheme. In: Ijspeert, A.J., Masuzawa, T., Kusumoto, S. (eds.) BioADIT 2006. LNCS, vol. 3853, pp. 48–63. Springer, Heidelberg (2006) 9. Leibnitz, K., Wakamiya, N., Murata, M.: Self-Adaptive Ad-Hoc/Sensor Network Routing with Attractor-Selection. In: IEEE GLOBECOM, San Francisco, pp. 1–5 (2006) 10. Kitajima, S., Hara, T., Terada, T., Nishio, S.: Filtering Order Adaptation Based on Attractor Selection for Data Broadcasting System. In: International Conference on Complex, Intelligent and Software Intensive Systems (CISIS 2009), Fukuoka (2009)

Expression of Personality through Avatars: Analysis of Effects of Gender and Race on Perceptions of Personality Jennifer Cloud-Buckner, Michael Sellick, Bhanuteja Sainathuni, Betty Yang, and Jennie Gallimore Department of Biomedical, Industrial and Human Factors Engineering Wright State University, Dayton, OH, 45435, USA [email protected]

Abstract. Avatars and virtual agents are used in social, military, educational, medical, training, and other applications. Although there is a need to develop avatars with human-like characteristics, many applications include avatars based on stereotypes. Prabhala and Gallimore (2007) conducted research to develop collaborative computer agents with personality. Using the Big Five Factor Model of personality they investigated how people perceive personality based on actions, language, and behaviors of two voice-only computer agents in a simulation. However, these computer agents included no visual features in order to avoid stereotypes. The objective of the current research extends the work of Prabhala and Gallimore by investigating the effects of personality, race, and gender on perceived personality of avatars with animated faces. Results showed that subjects were able to distinguish the different personalities and race and gender significantly affected perceptions on a trait-by-trait basis. Keywords: avatar, virtual agent, personality, Big Five Factor.

1 Introduction Avatars are frequently used in social networking activities, educational contexts, and in medicine for training, telemedicine, collaboration among providers, rehabilitation, and group counseling [1-5]. For the military, avatars have been used to aid in various simulations, including the navigation of unmanned aerial vehicles. In the research of Prabhala and Gallimore [6], the avatar was actually represented by just a voice with personality rather than a visual representation. The voice guided the user through different navigation activities. However, in comparison of text-only, audio-only, audio-video, and audio-avatar chat contexts, the video-assisted and avatar-assisted chats resulted in higher ratings for subject attention than audio-only or text-only chats [7]. With the growing applications of avatars, it is important to note that the perception of avatars’ personality and credibility can be affected by their appearance, behavior, language, and actions. The need for research and development for computer agents or avatars that more closely mimic human behaviors has been noted by many researchers [6, 11]. Using the Big Five Factor model of personality, Prabhala and Gallimore (2007) found that people could perceive personality from avatars through their J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 248–256, 2009. © Springer-Verlag Berlin Heidelberg 2009

Expression of Personality through Avatars

249

actions, language, and behavior [6]. Studies show that appearance, gender, and ethnicity can affect the user’s reaction to the avatar [9-10]. Facial actions like head tilting, nodding, eye brow rising, and blinking are also used as visual cues in perceiving inherent personality [11].

2 Methods 2.1 Objectives, Experimental Design, and Hypothesis The objective was to evaluate if people can perceive personality of computerized avatars based on actions, language, and behaviors and to determine if race and gender as represented in facial and voice features would affect perceptions. Independent variables are avatar’s race (dark or white), gender (male or female), and personality (P1 or P2). These were arranged in a 2x2x2 within-subject experiment. Data were collected on the subjects’ ratings of 16 personality subtraits in the Big Five Factor Model and 4 subjective questions on a 10-point scale (1: trait is not very descriptive of avatar and 10: trait is very descriptive). The hypotheses are as follows: Hypothesis 1: There will be no significant differences in subject’s subtrait ratings by personality. Hypothesis 2: Race and gender will not affect perception of personality. 2.2 Subjects and Apparatus The subjects were recruited from Wright State University via e-mail and class announcements. Thirty-five subjects completed the study (16 female and 19 male). The subjects were not paid to participate and had the option to leave at any time. Test scenarios were presented with two 17-inch computer monitors, a Windows XPequipped computer, and headphones. Scripts were recorded with Audacity audio software from sourceforge.net, and avatars were developed in Haptek’s PeoplePutty software. Statistical analysis was performed in JMP 7.0 and Microsoft Excel. 2.3 Stimuli Avatar Appearance. Shown in Figure 1, avatars consisted of: 1) dark female, 2) white female, 3) dark male, 4) white male. Each used the same face structure.

Dark female

White female

White male

Fig. 1. Avatar Appearance

Dark male

250

J. Cloud-Buckner et al.

The only modifications to the basic face were changing skin color (dark or white), broadening of the male face, and lengthening of the females’ hair. Avatar Personality. The two personality levels were constructed to be completely opposite each other, around 16 of the 30 Big Five Factor model subtraits. These were selected for what could be emulated in scenarios; also, insignificant factors from [6] were omitted, such as artistic interests and liberalism. Traits emphasized in each personality type were given an expected rating of 10, shown in Tables 1 and 2. Personality 1 was designed to be friendly, outgoing, and self-sufficient with a high activity level and some anger. Personality 2 was designed to be introverted, self-conscious, cooperative, orderly, modest, disciplined, and sympathetic to others. Table 1. Emphasized Traits and Actions, Language, and Behavior to Define Personality 1

Personality 1 Subtrait

Expected Rating

Behavior Casual greeting, positive comments, friendly tone, jokes, smiles with teeth showing and wide cheeks

Friendliness

10

Gregariousness

10

Is happy to see a crowd, talkative, likes activities with crowds, makes eye contact

Assertiveness

10

No hesitation, expresses ideas openly, talkative, serves as a leader of groups, makes eye contact

Activity level

10

Involved in many activities, quick information pace

Cooperation

1

Discusses confrontation about issue that angers them

Modesty

1

Talks about being superior to professors or other students, brags about achievements

Sympathy

1

Self-efficacy

10

Orderliness

1

Self-discipline

1

Cautiousness

1

Anxiety

1

Anger

10

Self-consciousness 1

Adventurousness 10 Intellect 1

Jokes about people being sick Uses confident, self-focused phrases (e.g., confident can do anything after graduation), never looks at notes Might jump from point to point in their talk because they don't know the order Procrastinates on assignments / studying for tests Mentions campus security because "it's required," mentions outdoor adventure activities Does not seem nervous about talking Makes snide remarks; has a tirade about something on campus (e.g., parking ticket, bad cafeteria food) Bold opinions no matter what others think Mention study abroad, outdoor adventure activities Prefers to people to ideas; does not like long puzzles

Expression of Personality through Avatars

251

Table 2. Emphasized Traits and Actions, Language, and Behavior to Define Personality 2

Personality 2

Friendliness

Expected Behavior Rating 1 Formal greeting, negative comments, Avoids eyes

Gregariousness Assertiveness Activity level

1 1 1

Cooperation

10

Modesty

10

Sympathy

10

Self-efficacy

1

Orderliness

10

Keeps organized schedule / notes for classes, regular study times, meeting with professor on regular basis

Self-discipline

10

Has regular study schedule ; finishes assignments

Cautiousness

10

Talks about campus security, carrying pepper spray

Anxiety

10

Apologizes for nervousness about talking

Anger

1

Nice about everything, even when it is bad

Self-consciousness

10 1 10

Shy about people; worried that they look stupid Mention others doing study abroad, going outdoors Describes solving puzzles; talks about ideas

Subtrait

Adventurousness Intellect

Shy, avoids crowds, avoids eye contact Hesitates while speaking, not talkative Slower pace of info, involved in fewer activities Avoids confrontation, offers individual help, talks about helping on teams rather than leading them Does not brag about things, works hard in class but does not take credit for being on top Offers extra guidance, condolences on problems Has no control in their life (e.g., parents made them go to that school); unsure about notes

A, L, B in Avatar Scripts. Each avatar script welcomed visitors and introduce a college campus as an online tour guide. The avatar personality was exhibited verbally through actions, language, and behavior (A, L, B) as well as tone of the avatar. The A, L, B for each trait are shown in Tables 1 and 2. Eight gender-neutral scripts were randomly assigned to the faces and each face had one script from P1 and P2. One female recorded all of the female scripts, and one male recorded all the male scripts. Each script was 1.5 to 2 minutes long and followed the same order of items: initial facial expression, greeting, university name, purpose of the talk, why the speaker came to the school, the year the school began, number of students, academics including speaker’s major, professors and classes, study abroad opportunities, facilities, dorms, security, clubs and organizations, and a conclusion. 2.4 Procedure Each subject was randomly assigned to an “order code” that specified the random order in which the 8 scripts would be viewed. After each script, subjects filled in a

252

J. Cloud-Buckner et al.

randomized spreadsheet ratings questionnaire with their rating on a 1-10 scale of how well the subtrait described the avatar, and what A, L, B led to that rating. Each question included the definitions of the subtrait based on the Big Five Factor model, such as “Friendliness describes the quality of being amicable and having no enmity,” This was repeated for all 8 scripts, and the entire process was generally 30-45 minutes. Subjects were also asked four subjective questions about the guide.

3 Results 3.1 General Model and ANOVA Information Each of the 35 subjects answered 160 total ratings questions, resulting in 5600 data points. The ANOVA model included Personality, Race, Gender and Question (20) resulting in a 2x2x2x20 within-subject analysis. Question was included because each question addressed a different subtrait. Since traits like Anger and Friendliness are on opposite scales, performing an analysis across questions can result in an averaging of those traits. We are interested in evaluating how subjects perceived the different subtraits embedded in the personality. The Alpha level was set to 0.05 and resulted in 11 significant factors. To control for sphericity, a conservative Greenhouse-Geisser (GG) correction resulted in 7 significant factors. Significant interactions were analyzed with simple effects f-tests, and main effects were analyzed with Tukey-Kramer Honestly Significant Difference (Tukey HSD) tests. 3.2 Interactions Gender x Personality x Race (GxPxR). This interaction must be viewed by subtrait. Without the G-G correction the RxGxPxQ interaction was significant, and with the GG correction the p-value is 0.0692. Given that the G-G is conservative, we looked at the effects of R and G for each Personality and each subtrait and we find significant main effects of R, G and RxG depending on the trait as indicated in Table 3. Personality x Question (PxQ). PxQ was significant (F(1,34) = 37.71, p < 0.0001). Simple effects F-tests show P is a significant factor for every Q except the one measuring Cooperation. Figures 2 and 3 illustrate the PxQ interaction for P1 and P2, respectively. The traits designed into P1 are rated as more descriptive for P1 than P2. For the traits designed into P2 there are two traits that are more descriptive for P1, Orderliness and Intellect. Cooperation was perceived equally for both P1 and P2. Race x Question (RxQ) was significant (F(1,34) = .0211, p = 0.0211). When analyzed by each question, ratings for race were significantly different for the following traits: orderliness, anger, cooperation, self-consciousness, modesty, self-discipline, intellect, sympathy. For all of these traits, subjects gave higher ratings (more descriptive) when the race was Dark except on the subtrait Anger, in which case the rating was lower (less descriptive). Additionally, the Dark avatars received a significantly higher mean rating on the question asking subjects if they would be willing to trust information from that avatar. Figure 4 shows significant traits.

Expression of Personality through Avatars

8 7 6 5 4 3 2 1 0

253

P1

s ne s ntu

Ad ve

Ac

tiv

i ty

ro us

Le

ti v en es se r As

eg ar Gr

ve l

s

r An ge

s ne s io us

fi c ac Ef lf Se

Fr i en dli n

es

y

s

P2

Fig. 2. Traits designed into P1 are rated on average as more descriptive than P2 Table 3. Instances of Significant Factors for Each Question, where x indicates occurrence of statistical significance of that factor (R, G, RxG) for a given personality and question

Friendliness Self-Efficacy Anxiety Gregariousness Orderliness Anger Assertiveness Activity Level Cooperation Self Consciousness Adventurousness Modesty Self-Discipline Intellect Sympathy Cautiousness Q: Willing to trust? Q: Satisfied w/ tour guide? Q: This guide for later tour? Q: School with this disposition? Total

Personality 1 Race Gender R*G x x

Personality 2 Race Gender R*G

x x

x

x

x x x

x x x x x x x x x x 11

x x x

x

x x x

x x

x

x

x

x x

x x

x x x

x x

7

5

7

5

5

254

J. Cloud-Buckner et al.

7 6 5 4 3

P1 P2

2 1

hy Ca ut io us ne ss Co op er at io n

pa t Sy m

lec t tel In

ip l in e isc

ty f-D

Se l

M od es

ss ci ou sn e

es s

on s

rd er lin

Se lf C

O

A

nx ie ty

0

Fig. 3. Traits designed into P2 are more descriptive for P2 than P1 for 6 of the 9 traits

3.3 Descriptions of Actions, Language, Behavior That Led to Ratings Out of 5600 rating points, 571 comments were provided from subjects. They centered on these areas: tour guide’s direct quotes, guide’s behaviors, tone of voice or accent, stuttering, pauses, forgetfulness, and lack of organization. Most comments referred to verbal rather than visual elements. Comments on appearance mentioned lack of eye contact, looking “boring,” “comfortable” facial expressions, and head movement. Every script element except for school size was mentioned at least once in comments. Some elements had up to 15 comments. The script lines were specifically designed to represent certain personality traits, but those were not always interpreted as intended. For example, when one avatar mentioned living at home with parents, some subjects interpreted that as intelligent budgeting while others saw that as lack of self-efficacy. 7 6 5 4 3 2

Dark White

In te lle ct Sy m pa W th ill y in g to Tr us t?

io n on sc io us ne ss M od es Se ty lfD isc ip lin e

Se lfC

oo pe ra t

ng er A

C

O

rd er lin es s

1 0

Fig. 4. Two-Way interaction of Race x Question (Significant questions only)

Expression of Personality through Avatars

255

4 Discussion 4.1 General Conclusions on Hypotheses The results rejected the null hypothesis of no difference in subtrait ratings by Personality. The two avatar personalities were specifically designed to be different and the ratings indicate that subjects perceived differences in the personalities. These findings are similar to those of Prabhala and Gallimore (2007) who found that subjects could perceive differences in personality even without a face. P1 was rated higher (traits more descriptive) on all 7 traits specifically designed into it. For P2, only 6 of the 9 traits that were built into the personality were perceived as “more descriptive” than P1. Two traits were rated as being more descriptive in P1 than in P2 (Orderliness and Intellect). There was no significant difference in ratings for Cooperation between P1 and P2. The magnitude of differences in the mean ratings varied across traits. For example, the difference in means for friendliness between P1 and P2 is 0.85, while the differences in all other traits built into P1 vary from 1.64 to 3.17. In P2, difference in mean ratings for self-discipline is only 0.47, while the other significant traits range from .0.67 to 2.79). It appears that traits for P2 are more difficult to model into the personalities. Traits that are related to emotion have larger differences (Anger and Anxiety). Adventurousness and activity level also has a larger differences, because scripts for P1 had notable sports and activity information, such as rock climbing or study abroad rather than P2’s swimming laps or walking. The null hypothesis that there would be no difference in ratings based on race and gender was rejected. We expected no differences, but race and gender of the avatar did play some role in perceptions of personality. For example, for P1 and P2, when there was a difference on Race, the dark skin had a higher rating for every item except anger. When there was a difference on Gender, in P1, the male had a higher score, but in P2, the women had a higher score. The interaction of RxG had mixed results across all subtraits. Overall, dark males had higher ratings on many personality traits except anger, which showed the lowest rating. It is difficult to understand the cause of these effects; we could not use race and gender as subject blocking variables so no comparison can be made between the subject’s race or gender and their ratings for the avatars of different races. Approximately 75% of the subjects were white. 4.2 Implications for Future Research The different ways that some script elements were interpreted by subjects indicates that future research should include a personality profile of the subject to see if the person’s preferences affect their favorable ratings on other personality types. Additionally, implications of racial bias in the RxQ results mean that future studies should track the race of the subject to see how the person’s race affects responses. Subjects should clearly understand that their answers are anonymously combined with others’ so that no judgments are made on their individual racial preferences. Some comments mentioned voice tone and accent, so future research should consider using different human and computer-generated voices to address this, especially when non-native English speakers are involved. Personality can also affect interpretation of tone; in one study extroverts were more attracted to extroverted voices; similarly, introverts were more attracted to introverted voices [17].

256

J. Cloud-Buckner et al.

The 571 comments about A, L, B showed that most comments referred to avatar verbal elements rather than visual appearance, possibly indicating that subjects’ biases for avatar appearance were revealed only through ratings, not through the comments. The scenario of avatars presenting a campus tour was a good vehicle to convey personality preferences. Differences between the schools were minimal but still conveyed some personality preferences of the avatars. In future studies, it would be interesting to see how different scenarios affect ratings. With a growing need for computer agents that resemble human behavior, this research confirms that personality can be both modeled and perceived and provides insight on how gender and race affect the perceptions of personality.

References 1. Kang, H.-S., Yang, H.-D.: The visual characteristics of avatars in computer-mediated communication: Comparison of internet relay chat and instant messenger as of 2003. Int. J. Hum.-Comput. St. 64(12), 1173–1183 (2006) 2. Gorini, A., Gaggioli, A., Vigna, C., Riva, G.: A second life for eHealth: Prospects for the use of 3-D virtual worlds in clinical psychology. J. Med. Internet Res. 10(3), e21 (2008) 3. Monahan, T., McArdle, G., Bertolotto, M.: Virtual reality for collaborative e-learning. Computers 50(4), 1339 (2008) 4. Heinrichs, W., Youngblood, P., Harter, P., Dev, P.: Simulation for team training and assessment: Case studies of online training with virtual worlds. World J. Surg. 32(2), 161– 170 (2008) 5. Hilty, D., Alverson, D., Alpert, J., Tong, L., Sagduyu, K., Boland, R.: Virtual reality, telemedicine, web and data processing innovations in medical and psychiatric education and clinical care. Acad. Psychiatr. 30(6), 528–533 (2006) 6. Prabhala, S.V., Gallimore, J.J.: Designing computer agents with personality to improve human-machine collaboration in complex systems. Wright St. Univ. (2007) 7. Bente, G., Rüggenberg, S., Krämer, N.C., Eschenburg, F.: Avatar-mediated networking: Increasing social presence and interpersonal trust in net-based collaborations. Hum. Commun. Res. 34(2), 287–318 (2008) 8. Rizzo, P., Veloso, M., Miceli, M., Cesta, A.: Personality-Driven Social Behaviors in Believable Agents. In: Proceedings of the AAAI Fall Symposium on Socially Intelligent Agents (1997) 9. Nasoz, F., Lisetti, C.L.: MAUI avatars: Mirroring the user’s sensed emptions via expressive multi-ethnic facial avatars. J. Visual Lang. Comput. 17, 430–444 (2006) 10. Masuda, T., Ellsworth, P., Mesquita, B., Leu, J., Tanida, S., De Veerdonk, E.: Placing the face in context: Cultural differences in the perception of facial emotion. J. Pers. Soc. Psych. 94(3), 365–381 (2008) 11. Arya, A., Jefferies, L.N., Enns, J.T., DiPaola, S.: Facial actions as visual cues for personality. Comput. Animat. Virt. W 17(3-4), 371–382 (2006) 12. Lee, K., Nass, C.: Social-psychological origins of feelings of presence: Creating social presence with machine-generated voices. Media Psychol. 7(1), 31–45 (2005)

User-Definable Rule Description Framework for Autonomous Actor Agents Narichika Hamaguichi1, Hiroyuki Kaneko1, Mamoru Doke2, and Seiki Inoue1 1

Science & Technical Research Laboratories, Japan Broadcasting Corporation (NHK) 1-10-11, Kinuta, Setagaya-ku, Tokyo, 157-8510, Japan {hamaguchi.n-go, kaneko.h-dk, inoue.s-li}@nhk.or.jp 2 NHK Engineering Services, Inc. 1-10-11, Kinuta, Setagaya-ku, Tokyo, 157-8540, Japan [email protected]

Abstract. In the area of text-to-video research, our work focuses on creating video content from textual descriptions, or more specifically, the creation of TV program like content from script like descriptions. This paper discusses a description framework that can be used to specify rough action instructions in the form of a script that can be used to produce detailed instructions controlling the behavior and actions of autonomous video actor agents. The paper also describes a prototype text-to-video system and presents examples of instructions for controlling an autonomous actor agent with our designed descriptive scheme. Keywords: Autonomous Actor Agent, Digital Storytelling, Text-to-Video, TVML, Object-Oriented Language.

1 Introduction Research into digital storytelling has attracted considerable interest in recent years, and one approach of producing computer graphics (CG) video content from textual descriptions has inspired a number of studies around the world. (We refer to this approach as “text-to-video” [1]) In text-to-video production, figuring out how to make the actor agents (CG characters) in the video act and behave naturally is critically important. In big production animated films, the mannerisms and behavior of actor agents can be manually edited on a frame-by-frame basis which is extremely costly in terms of man-hours, time, and budgets. But for smaller scale or personal video productions such lavish and costly production techniques are impractical, thus creating a demand for an autonomous method of controlling actor agents. A number of studies have addressed this issue of autonomous actor agents [2]. Researchers have also investigated language-based descriptive methods of producing video content and controlling the actions and behavior of actor agents, including a specially designed scripting approach [3] and a method of controlling the behavior of actor agents using natural language instructions [4]. Most of these J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 257–266, 2009. © Springer-Verlag Berlin Heidelberg 2009

258

N. Hamaguichi et al.

studies of autonomous actor agent action and ways of describing such actions are only able to describe such actions using a limited vocabulary based on rules set up in advance under limited conditions. The problem is that if the user wants to add a new autonomous action rule or wants to modify an existing rule, there are very few schemes giving users access to the rules, and the expandability of those that do provide access is quite limited. This led us to design a system that is functionally separated into two parts—a video content production part and an object-oriented description part— and instead of using a special proprietary language, the object-oriented description part uses a dynamic programming language that can be run as is from the source code without compiling the code beforehand. Users are thus able to add, modify, and reuse actor agent action rules by directly accessing the source code. Moreover, because this approach is based on an existing programming language, it is infinitely expandable according to the whims and desires of the user. In the next section, we will lay out the principle issues that will be addressed in this paper.

2 Requirements In this section we consider the requirements needed to enable users to represent rules that control the behavior and actions of actor agents. 2.1

Openly Modifiable Object-Oriented Language

The object-oriented approach permits functions to be encapsulated which are highly beneficial in terms of reusability, and today many advanced programming languages have adopted the object-oriented approach. There are essentially two types of objectoriented languages: languages that are executed after the source code is compiled (compiler languages), and languages that are not pre-compiled but are interpreted at run-time (dynamic programming languages or interpretive languages). In compiler languages, the source code is separated from the executable file, and because the executable file is a black box, the user is unable to modify functions or copy and reuse portions of functions even if he is able to use functions created by others. To achieve our objectives, we need a language that will enable users to add, modify, and reuse rules for controlling the behavior and actions of actor agents. In short, we need an object-oriented dynamic programming language that permits the user to access the source code. 2.2

Versatile Layered Structure for Different Types of Users

With the goal of using descriptive language to control the actions of actor agents, our first concern is to achieve the desired behavior or action using the simplest possible expressions without inputting detailed instructions. The problem with the objectoriented languages described earlier is that expressions inevitably become much more complicated than can be handled by a simple script language the more encapsulation and reuse is involved.

User-Definable Rule Description Framework for Autonomous Actor Agents

259

The level of language used also varies depending on the type of user and the intended use. For example, if one wants to produce a simple video clip with a minimum of time and effort, the level of language abstraction and types of data manipulated are very different than if one wants to produce content in which very detailed actions and timing of the actor agents are critically important. In order to address these issues, we adopted a three-layer structure that can be tailored to different kinds of users and different intended uses. The lower layer (detailed description instructions) is for pros enabling descriptions supporting detailed video content to produce professional-grade video content. The upper layer (simple description instructions) is for beginners or amateurs. It hides the complexity of detailed descriptions, and allows amateur users to produce video content using relatively simple descriptions. The rules that control the actions and behaviors of actor agents are described in the middle layer sandwiched between the upper and lower layers. Essentially, this layer converts the relatively simple instructions received from the upper layer to the more detailed expressions required by the lower layer.

3 Language Design and Prototype System Based on the requirements outlined in the previous section, we designed a prototype system consisting of two parts as illustrated in Fig.1: an Object-Oriented Description Module and a Presentation Module. The Object-Oriented Description Module is a three-layer structure as described above. It consists of a Simple Scripting Layer, an Automatic Production Layer, and a Detailed Description Layer, and runs using the dynamic programming language Python. A series of rough sequential instructions similar to the script for a TV program are described in the upper Simple Scripting Layer. The instructions are in a format that even someone with little or no experience with programming languages can understand and edit. The rules controlling the behavior and actions of the actor agents are described in the middle Automatic Production Layer. The layer receives the rough instructions described in the upper Simple Scripting Layer, acquires the situation using the functions of the lower Detailed Description Layer, then automatically determines the specific actions and behavior of the agents based on rules in the middle layer, which are sent to the lower Detailed Description Layer for execution. The lower Detailed Description Layer provides a simple wrapped interface with a TVML Player [5] in the Presentation Module, and through this intermediary wrapper, the Detailed Description Layer obtains the video states and delivers the instructions. Instructions are created using a descriptive language called TVML (TV program Making Language), and the states are acquired using an external application program interface called TvIF. TVML is a self-contained language featuring all the capabilities needed to create TV program-like video content, including detailed control over the speech and movement of actor agents, cameras, display of subtitles, and so on. Essentially, complex descriptive instructions are substituted for detailed control. The TVML Player uses software to interpret a TVML Script, then generates video content using 3D computer graphics, synthesized speech, and other production

260

N. Hamaguichi et al.

Fig. 1. Layered structure of the object-oriented video content production framework

techniques. The TVML Script only provides the TVML Player with a one-way stream of instructions, and the states are returned by way of the TvIF interface. Moreover, the TVML Player was developed using C++. This relieves the user of dealing with the technically challenging aspects of production such as handling 3D computer graphics which is done by the TVML Player, but the internal operation of the TVML Player itself is unalterable. By adopting the layered approach outlined above, users can employ whichever layer is best suited to their skills and objectives. And because users have direct access to the source code of each layer, they can add, modify, reuse and inherit classes of rules controlling the actions of actor agents.

4 Application Examples for Each Layer In this section we will provide description and application examples for each layer. 4.1

Simple Scripting Layer

The Simple Scripting Layer is the upper layer. It is based on a very intuitive format: a sequential string of instructions without any inherent control structure, much like the script of a TV program. It is thus transparent and easily manipulated by anyone, even people with little or no experience with programming languages. Application Example. Fig.2 shows a typical example of an application in which a description in the Simple Scripting Layer is edited, and the internal descriptions written in Python are the followings. These descriptions in the Simple Scripting Layer consist of a simple line-by-line sequence of unstructured instructions. So using a tool such as illustrated in Fig.2, the user can easily edit the script in much the same way as

User-Definable Rule Description Framework for Autonomous Actor Agents

261

Fig. 2. Application example for editing the descriptions in the Simple Scripting Layer

using a word processor. Any user capable of using a word processor is thus capable of producing video content! Description Examples import apetest

#Import of Automatic Production module

ape=apetest.APETest()

#Constructor

ape.title(“Script Example”) ape.text(“I wanna go over there!”) #Speech ape.action_walk_to_goal()

#Action

ape.subimage(“goal.jpg”)

#Show image

ape.action_look_at_camera() ape.text(“You see?”) ape.end() 4.2

Automatic Production Layer

The specific rules that control the actions and the behavior of actor agents on the basis of instructions received from the Simple Scripting Layer are described in the Automatic Production Layer. Description Examples. Here is an example of descriptions in the Automatic Production Layer. Action rules are represented in classes, and inherit a new class called APEBase. Basic action rules are defined in APEBase, so in order to create a

262

N. Hamaguichi et al.

new action rule, a user only needs to create or describe a rule that is different from that in the APEBase. The module for producing new action rules in this way is called the Automatic Production Engine (APE) [6]. goal.x=4 goal.z=0 class APETest(APEBase): … def setup(self): #Initialization self.A=tvml.Character(filename=”bob.bm”, x=-4) self.obst1=tvml.Prop(filename=”tsubo.obj”, x=1,… … def text(self, value):

#Speech

self.A.talk(text=value) … def subimage(self, value): #How to show image self.img=tvml.Prop() self.img.openimageplate(filename=value, platesizeh=3.6, platesizev=2.7) self.img.position(y=2, pitch=270) … def action_walk_to_goal(self):

#How to walk to goal

props=getPropNameList() #Get all prop names for prop in props: loc=findPath(obstacle=prop)

#Find a path per prop

self.A.walk(x=loc.x, z=loc.z)

#Simple walk

self.A.walk(x=goal.x, goal.z) As one can see, several method subroutines are defined: the setup method deals with initialization, the text method relates to the speech of actor agents, and the subimage method relates to how images are presented. The action_walk_to_goal method is a new subroutine created by the user that instructs the actor agent to avoid obstacles as it proceeds to a goal (the actual description has been simplified in this example). Previously, the only walk-related method defined in the lower Detailed Description Layer instructed the action agent to proceed in a straight line from its current position to the goal. The new action_walk_to_goal subroutine calculates a path from the position and size of obstacles (3D bounding box), thus

User-Definable Rule Description Framework for Autonomous Actor Agents

263

Fig. 3. Operation of the action_walk_to_goal method

enabling the user to define an action rule permitting the agent to proceed to a goal without bumping into things. Output Example. Fig.3 shows an example of how the action_walk_to_goal method is run based on descriptions in the Simple Scripting Layer. Users are thus able to add and modify the rules controlling the actions and behavior of agents in the Automatic Production Layer. Significantly, newly added action rules can be used and manipulated using an easy user-friendly format from the upper Simple Scripting Layer just like any other instruction. Here we have discussed a userdefined action rule enabling agents to avoid obstacles, but all sorts of powerful rules can be created in the same way, such as: − − - Actor agent actions playing to a particular camera: − - Actions of an agent can be controlled to play to a particular camera by acquiring the names, positions, and angles of the cameras. − - Actor agent actions can be synched to a movie file: − Actions of an agent can be synchronized to the playback timing of a movie by acquiring the playback timing of the movie file. − - Actor agent behavior can be synched to speech:

264

N. Hamaguichi et al.

− The expressions and gestures of an agent can be synched to the character strings of the synthesized speech lines spoken by the agent. − - Evaluation of the output screen layout: − The layout or composition of the output video screen can be evaluated and the agent's actions adjusted to the layout by acquiring an on-screen 2D bounding box. These various types of automatic production rules are actually executed by the functional capabilities of the Detailed Description Layer. Let us next take a closer look at the Detailed Description Layer, which must be endowed with powerful capabilities in order to execute these rules. 4.3

Detailed Description Layer

The TvIF/TVML are wrapped by the method subroutines that are incorporated in the Detailed Description Layer. This layer can obtain a comprehensive range of states in the TVML Player including the prop bounding box, camera information, movie playback timing, and a host of other states. Table 1 shows some of the state acquisition methods that are incorporated in the Detailed Description Layer. Note too that all of the states incorporated in the TVML Player—orientation angle and coordinates of the actor agents, speed, timing, and so on—can be directly controlled by the TVML Script. This allows more experienced users who want direct control over the production and editing of their video content to directly work at this layer. Description Examples. Here are some typical examples of Detailed Description Layer descriptions. These descriptions enable detailed control over the movement of actor agent joints, camera movements, and a host of other variables. buddy.gaze(pitch=-30, wait=NO) buddy.turn(d=-120, speed=0.5) buddy.definepose(pose=GetWhisky, joint= LeftUpperArm, rotx=-105.00, roty=25.00) buddy.definepose(pose=Getwhiskey, joint=Chest, rotx=5.00, roty=-15.00, rotz=0.00) buddy.pose(pose=Getwhiskey, speed=0.25, wait=NO) tvml.wait(time=0.7) cam1.movement(x=-0.49, y=1.57, z=1.75, pan=400, tilt=-5.00, roll=1.00, vangle=45.00, transition=immediate, style=servo) whisky.attach(charactername=buddy, joint=RightHand, switch=ON) Output Example. Fig.4 illustrates how the agent actually moves based on the Detailed Description Layer descriptions listed above.

User-Definable Rule Description Framework for Autonomous Actor Agents

265

Table 1. Typical state acquisition methods incorporated in the Detailed Description Layer

getCharacterLocation getCharacterTalkingText getCameraCurrent getCameraLocation getPropNameList getPropBoundingSolid getPropBoundingBox getMovieCurrentTime

Current position of an actor agent Character string currently spoken by an actor agent Name of camera currently selected Current location and angle of camera List of prop names 3D bounding box of a prop 2D bounding box of an on-screen prop Playback position of a movie file

Fig. 4. Movement based on Detailed Description Layer descriptions

5 Conclusions In this work we designed an object-oriented scheme for building a descriptive language framework enabling users to add, modify, and replay rules controlling the behavior and actions of autonomous actor agents. By dividing the object-oriented description into three layers—the Simple Scripting Layer, the Automatic Production Layer, and the Detailed Description Layer—we have implemented a structure that can be tailored to different kinds of users and different intended uses. This scheme allows users themselves to describe rules for controlling the behavior and actions of autonomous actor agents by editing the Automatic Production Layer. Leveraging this Automatic Production Layer based scheme, we plan to design a wide range of autonomous actor agents and develop applications that use the agents.

266

N. Hamaguichi et al.

References 1. Bindiganavale, R., Schuler, W., Allbeck, J., Badler, N., Joshi, A., Palmer, M.: Dynamically Altering Agent Behaviors Using Natural Language Instructions. In: The 4th International Conference on Autonomous Agents, Proceedings, Barcelona, Spain, pp. 293–300 (2000) 2. Funge, J., Tu, X., Terzopoulos, D.: Cognitive Modeling: Knowledge, Reasoning and Planning for Intelligent Characters. In: The 26th International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 1999), Proceedings, Los Angeles, USA, pp. 29–38 (1999) 3. Hamaguchi, N., Doke, M., Hayashi, M., Yagi, N.: Text-based Video Blogging. In: The 15th International World Wide Web Conference (WWW 2006), Proceedings, Edinburgh, Scotland (2006) 4. Hayashi, M., Doke, M., Hamaguchi, N.: Automatic TV Program Production with APEs. In: The 2nd Conference on Creating, Connecting and Collaborating through Computing (C5 2004), Kyoto, Japan, pp. 20–25 (2004) 5. http://www.nhk.or.jp/strl/tvml/ 6. Perlin, K., Goldberg, A.: IMPROV: A System for Scripting Interactive Actors in Virtual Worlds. In: The 26th International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 1996), Proceedings, New Orleans, USA, pp. 205–216 (1996)

Cognitive and Emotional Characteristics of Communication in Human-Human/Human-Agent Interaction Yugo Hayashi and Kazuhisa Miwa Graduate School of Information Science, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, 464--8601, Japan {hayashi,miwa}@cog.human.nagoya-u.ac.jp

Abstract. A psychological experiment was conducted to capture the nature of Human-Human and Human-Agent Interactions where humans and computer agents coexist in a collaborative environment. Two factors were manipulated to investigate the influences of the 'schema' about and the 'actual partner' on the characteristics of communication. The first factor, expectation about the partner, was controlled by the experimenter's instruction, manipulating with which partner (human or computer agent) participants believed to be collaborating. The second factor, the actual partner, was controlled by manipulating with which partner (human or computer agent) participants actually collaborated. The results of the experiments suggest that the degree of the refinement of the conversation controlled as the actual partner factor affected the emotional and cognitive characteristics of communication; however the schema about the opponent only affected the emotional characteristics of communication. Keywords: Collaboration, Human-Human Interaction, Human-Agent Interaction, Communication.

1 Introduction Communication across different computers connected by the Internet continues to increase due to the development of computer network technologies. In such situations, research on technologies for supporting such collaboration using computer agents has appeared [6]. In the fields of Human Computer Interaction (HCI), there are studies focusing on the nature of humans and computer agents [3]. In our study, we conduct a psychological experiment to capture the nature of Human-Human Interaction (HHI) and Human-Agent Interaction (HAI) in a setting where humans and computer agents coexist. In daily life, we make inferences and decisions about an opponent based on information received from it. For example, when he reacts politely, we may guess his character and attitudes through his reactions. This indicates that in communication the contents of conversation are important for determining the characteristics of communication. In contrast, in the initial stage of communication, information about an opponent is limited. Therefore, people rely on the related knowledge of an opponent obtained in advance and J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 267–274, 2009. © Springer-Verlag Berlin Heidelberg 2009

268

Y. Hayashi and K. Miwa

infer him based on this knowledge. Actually, social psychological research has indicated the importance of top-down processing based on the knowledge about the speaker such as schema and stereotypes in interpersonal cognition [1]. In communication through the Internet where we do not face an opponent directly, the above two perspectives, "actual conversation" and "schema," function as follows: (1) either the opponent is believed to be a human or computer agent (based on the schema), and (2) either the actual opponent's conversation is sophisticatedly adaptive or simply machine-like. We performed a psychological experiment in which the two factors were manipulated to investigate the characteristics of communication where humans and computer agents coexist.

2 Method 2.1 Task We assume that the above interesting aspects of communication emerge remarkably in a situation where interpersonal conflicts emerge during collaboration. To establish such a situation, we used an experimental paradigm designed by Hayashi, Miwa, and Morita [2], in which two participants, each having a different perspective, communicate with each other. As shown in Figure 1, a stimulus is constructed where black and white unit squares are randomly arranged on a 6- by 6-grid. We call each surface comprised of black and white unit squares an 'object'. In Figure 1, there are a total of ten objects: five black and five white. This stimulus is displayed on either a black or white background. This stimulus is presented to one participant on a black background; to the other on a white background. Based on Gestalt psychological principles, the participants acquire a single perspective focusing on objects whose color is the opposite of the background color. Each participant informs the other of the sequence of the number of objects he/she perceives during the task (Figure 2). In the initial stage of the experiment, one experiences difficulty understanding the perspective of the other; with miscommunication, interpersonal conflict occurs, which the participants must resolve to complete the task.

Fig. 1. Example of stimuli

Cognitive and Emotional Characteristics of Communication

269

Fig. 2. Series of stimuli presentations

2.2 Experimental System Figure 3 illustrates an example screen shot. The stimulus is presented in the center. Below it, there is a text field where the participants input their messages and receive their partner's messages. Just one sentence per trial is permitted, and at most 30 words are accepted. Buttons for changing the slides, sending messages, and terminating the experiment are placed at the bottom of the screen. We developed a natural language conversation agent whose responses are generated based on scripts. The agent is constructed to respond to the sentences input from the participants. This agent has mechanisms for conversation such as extracting keywords, activating scripts, and generating responses utilizing keywords drawn from the partner's sentences inputted.

Fig. 3. Example screenshot

270

Y. Hayashi and K. Miwa

3 Experiment Design 3.1 Summary The experiment has a 2 x 2 between-subjects factorial design. The first factor was controlled by the experimenter who manipulated with which partner (human or computer agent) the participants believed themselves to be collaborating. This represents the manipulation for the schema about the opponent. The second factor was controlled by manipulating with which partner (human or computer agent) the participants actually collaborated. This represents the manipulation of the degree of refinement of the conversation as the actual partner. In the following, we use small letters to express characters in the first factor, human and agent, and capital letters to express characters in the second factor, HUMAN and AGENT. For example, in the agent/HUMAN condition, the participants were instructed that the collaborative partner was a computer program; however they conversed with a human partner (details are discussed below). We constructed four conditions: human (instruction)/HUMAN (actual partner), agent/HUMAN, human/AGENT, and agent/AGENT. 3.2 Experiment Situation One hundred and three undergraduates participated in the experiment (male = 57, female = 46, M age = 18.82 years). They were set up to always speak first in the AGENT conditions where the participants conversed with the agent. On the other hand, the participants were set up to speak both first and second in the HUMAN conditions where the participants conversed with real people. Therefore, twice as many participants were assigned to the HUMAN conditions. Table 1 shows the number of participants assigned to each condition. The experiment was performed in small groups consisting of eight to twelve participants. Two types of computers were set up in a laboratory: machines connected to the Internet by wireless LAN and those running independently of other computers. These computers were placed so that no participant could peek at other screens (Figure 4). For manipulating the first factor, the participants were instructed that the collaborative partner was either: (1) someone in the room or (2) a program installed in the computer. For manipulating the second factor, the actual partner was controlled by assigning either: (1) a computer connected to someone in the room through wireless LAN or (2) one in which the conversation agent was running independently from others. Table 1. Experimental design and number of assigned participants

Actual partner

HUMAN AGENT

Instruction humans agents 34 34 18 17

Cognitive and Emotional Characteristics of Communication

271

Fig. 4. Experimental situation

3.2 Questionnaires In our study we utilized a questionnaire developed by Tsuduki and Kimura [4] answered by the participants after their conversations to solve the task were terminated. This questionnaire, which asks about the psychological characteristics of the media in communication, was comprised of 16 questions scored with a five-point scale. We classified the 16 questions into three measures. The first measure denotes the "interpersonal stress" factor consisting of five questions about such feelings as tension, severity, and fatigue. The second measure denotes the "affiliation emotion" factor consisting of eight questions about such feelings as friendliness to the opponent, ability to discuss personal matters, and happiness. The third measure denotes the "information propagation" factor consisting of three questions about such feelings as purpose and effectiveness in collecting information. In each of the three measures, the rating scores were totaled and divided by the number of subordinate questions. These rating scores (i.e., mean numbers) were used for analysis.

4 Results Figure 5 indicates the results. The vertical axis represents the mean value of the ratings, and the horizontal axis represents each measure. For participants who talked first, a 2 x 2 ANOVA was conducted on each measure with the factor of instruction (human vs. agent) and the factor of actual partner (HUMAN vs. AGENT) as a between-subject factor. For participants who talked second, a t-test was conducted on each measure. These participants were assigned only to two conditions: human/HUMAN and agent/HUMAN; therefore, only the effect of the instruction factor was examined.

272

Y. Hayashi and K. Miwa

5 4.5 4

humanHUMAN agentHUMAN

humanAGENT agentAGENT

3.5 3 2.5 2 1.5 1 Interpersonal stress

Affiliation emotion

Information propagation

(a) Participants who talked first

5 4.5

humanHUMAN

agentHUMAN

4 3.5 3 2.5 2 1.5 1 Interpersonal stress

Affiliation emotion

Information propagation

(b) Participants who talked second Fig. 5. Questionnaires

4.1 Interpersonal Stress For participants who talked first, interaction on interpersonal stress was significant (F(1,65)=7.34, p

Device 1 2

pad + fixed electrodes (active)

Fig. 13. Left: Functionalizing packaging. The stamp of the cover wafer deflects the comb permanently. Right: SEM graph of a test structure. The left comb is torsional deflected.

510

H. Schenk et al.

Preliminary simulation results show that a scan angle of up to ±10° mechanically in the static mode can be achieved (depending on the scan/switching frequency). Experimentally, the principle was proved by test structures. An example is shown in Fig. 13, right. A more detailed description of the approach including simulation results is found in 4.

6 Summary and Conclusions 2D micro scanning mirrors were presented with resonant driving for both axes. Large scan angles of up to 112° x 84° optically are supported by deploying degressive springs for the suspension of the mirror plate and the movable frame. An ultra compact laser projector and an ultra compact laser imager were presented. With the availability of small RGB-laser modules both systems can be further miniaturized. In principle, both systems can be combined and realized with one single 2D scanning mirror, only. To support higher resolutions a novel approach to realize a quasistatic scanner was presented. Here, the permanent out-of-plane deflection of the driving comb for the slow axis is achieved by a functionalizing packaging.

Acknowledgment The authors would like to thank Dr. Andreas Bräuer from FhG-IOF for the cooperation with respect to the laser projector and Microvision for the cooperation regarding the endoscope.

References 1. Yalcinkaya, A.D., Urey, H., Brown, D., Montague, T., Sprague, R.: Two-axis electromagnetic microscanner for high resolution displays. J. of Microelectromechanical Systems 15, 786–794 (2006) 2. Drabe, C., James, R., Klose, T., Wolter, A., Schenk, H., Lakner, H.: A new micro laser camera. In: Proc. of SPIE, San Jose, USA, vol. 6466, pp. 64660I-1–8 (2007) 3. James, R., Gibson, G., Metting, F., Davis, W., Drabe, C.: Update on MEMS-based Scanned Beam Imager. In: Proc. of SPIE, San Jose, USA, vol. 6466, pp. 64660J-1–11 (2007) 4. Jung, D., Kallweit, D., Sandner, T., Conrad, H., Schenk, H., Lakner, H.: Fabrication of 3D Comb Drive Microscanners by mechanically induced permanent Displacement. In: Proc. of SPIE, San Jose, USA, vol. 7208, pp. 72080A-1–11 (2009)

Understanding the Older User of Ambient Technologies Andrew Sixsmith Gerontology Research Centre, Simon Fraser University 2800-515 West Hastings Street, Vancouver, BC V6B 5K3, Canada www.sfu.ca/grc Abstract. This paper reports on the user-driven research and development (R&D) approach adopted with the EU-funded SOPRANO project (http://www.soprano-ip.org/) to develop an “ambient assisted living” (AAL) system to enhance the lives of frail and disabled older people. The paper describes the conceptual framework and methods used within SOPRANO and briefly presents some of the results from requirements capture, use case development and initial prototype development. The focus of the research is on understanding the potential user of the SOPRANO AAL system using a holistic ecological model of person and context and using methods that aimed to explore different experiential “realities”. The results demonstrate the usefulness of the approach for involving user in all stages of R&D and in generating and evaluating ideas for prototype development. Keywords: Ambient assisted living, older people, user-driven research.

1 Introduction AAL systems use technologies such as sensors, actuators, smart interfaces and artificial intelligence to create an intelligent, interactive and supportive home environment by to enhance safety, support independence and social participation. SOPRANO (Service Oriented Service-oriented Programmable Smart Environments for Older Europeans) is a consortium of commercial companies, service providers and research institutes with over partners from in Greece, Germany, UK, Netherlands, Spain, Slovenia, Ireland and Canada. Emerging information and communication technologies (ICTs), such as “pervasive computing”, “ubiquitous computing” and “ambient assisted living” have the potential for enhancing the lives of frail and disabled older people. However, this paper argues that there are considerable problems of mapping out this new area of technology in a way that is sensitive to the everyday needs, preferences and objectives of a very heterogeneous group. The paper describes the conceptual framework and user-driven approach used within the SOPRANO project and briefly presents the some of the results from requirements capture, use case development and initial prototype development.

2 Background - Conceptualizing the Socio-technical Domain As well as technical development, R&D within AAL has to deal with a number of challenges, not least being how should we explore, visualize and map out this J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 511–519, 2009. © Springer-Verlag Berlin Heidelberg 2009

512

A. Sixsmith

unchartered area in order to exploit this potential. Within the general field of technology and ageing, there has been a focus on health-related problems associated with old age. Moreover, R&D is often driven by a technological agenda that defines R&D in terms “engineering problems and solutions”, while evaluation of needs and outcomes focus on “usability” of systems and devices. However, this approach often misses the point of how technology is embedded in the everyday lives of older people and how it could positively enhance their quality of life. If these issues are not taken seriously, there is a danger that ill-conceived technologies will at best be irrelevant or inappropriately-designed and at worst will reinforce some of the negative ageist assumptions that frame much of society’s response to ageing [1]. In a very real way, the experience of ageing is a socially constructed phenomenon and the SOPRANO system and its technical components should be explicitly conceived and designed as part of a “socio-technical” system that incorporates both the human and machine domains within a coherent conceptual framework. Within SOPRANO this framework for understanding the user comprised two perspectives. A first perspective involved building a model of the personal and contextual domains relevant to AAL. Much of the literature around assistive technologies focuses on the functional impairments of the person and suggests solutions that directly or indirectly compensate for that impairment- what could be described as a “problem-need-solution” approach. Examples of this are prosthetic devices or powered wheelchairs for physically impaired people. In addition, much of the research and development in relation to older people has focused on the problems associated with advanced old age, particularly focusing on issues of safety and security (such as personal response systems, wandering alerts, fall detectors, hazard detection). While these kinds of help and support are important they ignore the interplay of the personal and contextual factors in the sense that “disablement” is an expression of the relationship of impairment within its environmental context. Moreover, the problem-need-solution approach ignores the creative and adaptive responses and resilience of the person to deal with their everyday life situations and the need to develop an approach that is about “enablement” as well as “care”. In light of this, the SOPRANO project adopted a simple ‘ecological” model of the person (3) in order to map out the main domains of the socio-technical AAL system (Fig 1). The model makes a distinction between objective and subjective components. The objective components of the model refer to: characteristics of the person (physical and cognitive status, psychological variables, etc.); characteristics of the context in which they live (physical environment, care provision, social network etc.); activities that are carried out in everyday life (social interaction, activities of daily living, leisure etc.). The ecological conceptualization suggests that these personal and contextual components are somehow in balance, for example, that the person’s functional capacities are sufficient to meet the demands of the context within which they live. The subjective components refer to the way the person experiences their everyday life in terms of personal meanings and well-being. For example, two people may be struggling to perform some activity (e.g. dressing or climbing stairs ) because of some physical problem. For one person this may be a source of frustration, but for the other

Understanding the Older User of Ambient Technologies

513

Person

Support Network

Social Network

Activities

Meaning

Well-being

Physical Environment

Culture

Fig. 1. Ecological model of socio-technical system [2]

it may result in a sense of achievement. The holistic nature of this model allows the researcher to explore the interplay of personal and contextual factors, their relation to everyday life (in terms of activities) and how these impact on the person’s subjective experience and well-being. A second perspective for understanding the user required research methods that addressed different levels of “reality”. Drawing on Dovey [3], it is possible to identify three modes of engagement that have both theoretical and methodological relevance to understanding place experience: • Contemplative engagement: This refers to the “conscious”, articulated perspectives that a person has about their lives. These are the ideas and meanings that a person has developed themselves or articulated in response to questioning, e.g. through interviews and surveys. • Everyday engagement: Much of everyday experience happens at a level of practical activity where individuals are only aware of a few key aspects of their experience at any given time, with phenomena only emerging when the familiar and predictable order of things is “threatened” or disrupted. Much of this experience is unarticulated and forgotten by the person and methods to explore this level of reality require to be situated within the “flow of experience” e.g observation or time use diaries.

514

A. Sixsmith

• Instrumental engagement: This can be seen as the immediate relationship between person and the things around them. In this sense, the life situation comprises features that facilitate or constrain the activities of the person. As with “everyday engagement” this requires methods that are able to directly examine this relationships as a person’s accounts may be inaccurate. A key issue here is that researching these different modes of engagement with the everyday world generally, and technology specifically, requires different approaches and methods. For example, the use of methods that depend on a person’s accounts, such as questionnaires and qualitative interviews, may bring out certain information, but miss out others. Table 1 provides a framework of possible methods for researching these different modes of engagement. The table also categorises these methods according to particular stages within the research and development cycle: user requirements; technology development; evaluation.

3 User-Driven Approach and Methods SOPRANO has aimed to be innovative in its approach to the research and development process by using an ethnographic approach to develop a holistic understanding of potential users and to involve them in all stages of the work to explore, visualize and map out an AAL system that will practical benefits for users in their everyday lives. Potential users were involved in order to gather their feedback on the key challenges to independence/quality of life [1]. This was done without reference to technological solutions in the first instance as the aim was to identify “opportunities” for technological support, without being driven by a predefined technical agenda. To this end, 14 focus groups (with more than 90 users) were conducted with older people, informal carers and care professionals in the UK, the Netherlands, Spain and Germany. Also interviews with older individuals were carried out in Germany, Spain and the Netherlands. A next stage of research and development involved the mapping out of use-cases (descriptive models) for potential technological solutions and exploring how these might work in a real-life context. A challenge was to help potential users to both in terms of design idea generation and design idea evaluation. Visualisation techniques such as theatre groups as well as specially designed focus groups applying multimedia demonstrators [1] were used to transform each use case into a drama session or animated within a multimedia demonstrator, which were viewed interactively and discussed within small user groups. A total of 72 potential users participated in 27 sessions conducted in the 4 different countries. All the research conformed to the ethical guidelines for each country. Sessions were audio or video recorded for subsequent analysis. 3.1 Results The paper outlines some of the results of the user research and development within SOPRANO in areas such as encouraging and reminding people to take medication, helping them to follow exercise programs, enhancing social interaction and living safely. The paper presents results from various stages of the user research from initial requirements analysis through to prototype development and refinement.

Understanding the Older User of Ambient Technologies

515

Table 1. Multi-method approach to research different realities Level of engagement

Perspective

Requirements methods

Development methods

Evaluation methods

Contemplative engagement

• Subjective: • Accounts based research • (quantitative and qualitative) • e.g. scaling • grounded theory

• Interviewsqualitative and quantitative • Focus groups • Surveys

• Intervention visualisation • Interviews • Focus groups

• Interviewsqualitative and quantitative • Focus groups • Surveys

Everyday engagement

• Subjective/o bjective: • Phenomenol ogy • Time/space usage analysis • Case studies

• Diaries • Prolonged researcher engagement • Observation • Activity monitoring using sensors • Experience sampling

• Working prototypes • Observation

• Diaries • Prolonged researcher engagement • Observation • Activity monitoring using sensors • Experience sampling • Logging of interactions by “smart” technologies

Instrumental engagement

• Mainly objective • Subjective to some extent: • Ergonomics • Occupationa l therapy • Personenvironment fit • Human factors

• Observation • Checklists • User assessments • Environment al assessments

• Lab prototypes • Mock-ups and wizard of Oz techniques • Observation

• Observation • Checklists • User assessments • Environment al assessments

4 Opportunities for Socio-technical Support The user research suggested that ambient assisted living has a potential role in a number of key areas. Social isolation has profound negative outcomes such as

516

A. Sixsmith

loneliness, depression, boredom, social exclusion and disruption of patterns of daily living. Safety and Security issues that were highlighted include falls, control of household equipment and access into the home. Forgetfulness appears to be a challenge to independence for many and concerns, for example, taking medication or remembering to switch off devices. Mobility inside and outside the home includes challenges to personal mobility, walking in the neighbourhood and use of public transport. Some of the other themes to emerge from the user research focused on how to enable people to lead a more active and participative lifestyles. Keeping healthy and active included physical and mental activity, exercise, good nutrition and daily routines. Community participation and contribution to local community was a priority for some people. Accessing information/keeping up to date was important in maintaining indpendence, as well as finding help and tradesmen to do little jobs around the home. Getting access to shops and services was problematic for people who have difficulty getting out of the house. The research also highlighted the potential of indirect help and support. Quality management of care provision is an important issue to ensure that the right amount and right quality of care is delivered in people’s homes.

5 SOPRANO Use Cases The user research provided the starting point for developing a set of use cases (descriptive models) that describe in a straightforward way the interactions between users and the AAL system: Safety and security focused on some of the more hazardous aspects of everyday life: Open Door addressed safety and security aspects of access into and out of the home; Safe involved environmental and activity monitoring for signs of problems; Fall focused on detection and response to falls at home. Forgetfulness were use cases aimed at helping people with mild cognitive impairments: Medication Reminding addressed how SOPRANO could help a person who keeps forgetting to take medicine. Easy-to-use Home Automation demonstrated smart home components supporting independent living. Remembering provided reminders and encouragement for people with every tasks of living. Active lifestyles were use cases that aimed to promote healthy and active living: Exercise focused helping older people to follow rehabilitation programs after discharge from hospital; Active aimed at monitoring and supporting healthy and active routines. Social activities were use cases that aimed to enhance social participation: In Touch aimed at improving social interaction; Entertained focused on supporting leisure activities.

6 Use Case Validation and Refinement - An Example It is not possible to describe all the use cases listed above. However, it useful to present a single use case to illustrate the value of the approach adopted within SOPRANO. The Exercise use case addressed the need to help a person follow a

Understanding the Older User of Ambient Technologies

517

programme of rehabilitation exercises that are typical for someone who has recently had an operation such as hip replacement. The key elements of the use case are: • Roger (user) has had hip replacement surgery and completes the in-hospital course of exercises • The hospital discharge planning service contacts Wendy (Therapist) and tells her that Roger is leaving hospital, that he has been prescribed exercises to do at home. • Wendy and Roger and Pamela decide whether Roger will: Receive assistance [from a person] at home to do the exercises Travel to the hospital physiotherapy department to do the exercises Do the exercises at home by himself with or without assistance from SOPRANO. • Wendy feels that if Roger does not want to travel to the hospital, he will still need support to do them correctly at home and that Roger might 1) might give up too easily, 2) forget some of the exercise steps or 3) need reminding when to start or encouragement to keep going. Roger feels that he would manage at home, but he does not refuse assistance. It is agreed that Roger will do the exercises at home with support from SOPRANO system. • The SOPRANO system uses a speech interface and on-TV avatar to prompt and encourage Roger to carry out his exercises as prescribed. Example dialogue is: “Good afternoon Roger, it’s time for your exercises. Do you want me to remind you what the first exercise is? Please say yes if you do or no if you do not”. The system provides a series of simple prompts and reminders to encourage Roger to carry out the exercises and to log if Roger responds “yes” at the completion of each stage. The Exercise use case was presented to a group of older people in a theatre session held in Newham London. This theatre session involved short plays to be enacted based on the use case to help the audience to visualise the system working in a reallife situation. The plays worked through all the different elements of the use case in a mocked-up AAL system interacting with actors. The short plays were followed up by audience feedback sessions moderated by a trained researcher. The main results of the feedback were: •

• • •

Perceived importance of the Exercise reminder: It was clear that older people receive very little direct support for rehabilitation from formal and informal sources. Everyone recognises the need to do exercise, but the spirit and motivation is often weak. People recognise the need for external support, providing evidence that further validates the use case. Look and feel of system: The tone of the support is very important- any help has to be done with sympathy and empathy to the situation of the person, but at the same time be firm in terms of clearly defining the need for doing the exercises. Possible drawbacks: Participants highlighted the possible isolation associated doing exercises alone in the home. This may be very important to the success of the rehab programme. Context: It is important that the person’s living environment is assessed to ensure exercises can be safely and appropriately carried out in the house.

518

• • •

A. Sixsmith

Demonstration: The user needs some kind of demonstration on how to do the exercise- e.g. the person has to be shown how to do it safely. Intrusiveness: The system should not be intrusive on the person’s everyday life and exercise reminders should be geared to the routines, preferences and activities of the person. Monitoring: It is important that a person receives feedback and also that they are monitored to ensure that exercises are actually carried out.

The results of the feedback form the theatre group were extremely important in refining the use-case and developing the system specification for the SOPRANO system. The example presented here illustrates the value of utilising innovative methods for involving users directly in the design process and to help them visualise the system as part of everyday “reality”.

7 Conclusions and Future Directions This paper has discussed the methods and results of a user-driven approach to developing an AAL system. The focus of the research has been on understanding the potential user of the SOPRANO AAL system using a holistic ecological model of person and context and using methods that aimed to explore different experiential “realities”. The paper presented some of the high level results of the SOPRANO user research and illustrated the usefulness of the approach through a more detailed examination of a particular use case supported by the SOPRANO system. The paper demonstrates the usefulness of the approach for involving user in all stages of R&D and in generating and evaluating ideas for prototype development. User involvement has been facilitated through continuous dialogue using methods that have been tailored to meet the specific information needs at each stage of the research process. Later phases of the SOPRANO project involve the development of prototype system and components and the deployment of the system in demonstration facilities and field trials. Large scale field trials of limited version of the SOPRANO system are planned to evaluate its impact in real-life situations with 300 users and carers across Europe. The large-scale trials will fully support the “medication”, “remembering” and “fall” use cases and partially support the “active” and “entertained” use cases. As with earlier phases of SOPRANO the user research will specified by the approach and methodological framework outlined earlier.

Acknowledgements SOPRANO (http://www.soprano-ip.org/) is an Integrated Project funded under the EU’s FP6 IST programme Thematic Priority: 6.2.2: Ambient Assisted Living for the Ageing Society (IST – 2006 – 045212). The authors acknowledge the input and role of the SOPRANO consortium and would also like to thank the many people who volunteered in the various stages of user research described in this paper.

Understanding the Older User of Ambient Technologies

519

References 1. Sixsmith, A.: SOPRANO An Ambient Assisted Living System for Supporting Older People at Home. In: Paper for ICOST 2009 7th International Conference on Smart Homes and Health Telematics: Ambient Assistive Health and Wellness management in the Heart of the City, Tours, France, July 1-3 (2009) 2. Sixsmith, A., Gibson, G., Orpwood, R., Torrington, J.: Developing a technology ‘wish-list’ to enhance the quality of life of people with dementia. Gerontechnology 6(1), 2–19 (2007) 3. Dovey, K.: Home as Paradox. In: Rowles, G., Chaudhury, H. (eds.) Home and Identity in Late Life. Springer Publishing Company, New York (2005)

Multi-pointing Method Using a Desk Lamp and Single Camera for Effective Human-Computer Interaction Taehoun Song1, Thien Cong Pham1, Soonmook Jung1, Jihwan Park1, Keyho Kwon2, and Jaewook Jeon2 1

School of Information and Communication Engineering, Sungkyunkwan University, Suwon, Korea {thsong,pham,kuni80,fellens}@ece.skku.ac.kr 2 School of Information and Communication Engineering, Sungkyunkwan University, Suwon, Korea {khkwon,jwjeon}@yurim.skku.ac.kr

Abstract. Multi-pointing has become an important research interest, and is used in many computer applications to allow users to interact effectively with a program. Multi-pointing is used as an input method, and can also be fun and very user-friendly. However, in order to use the method, a complex and expensive hardware configuration is required. This paper presents a new and low cost method of multi-pointing based on a simple hardware configuration. Our method uses dual hand recognition, a table lamp, and a single CMOS camera. The table lamp provides a steady illumination environment for image processing, and the CMOS camera is mounted to maintain good stability. A single camera is used for dual hand recognition to achieve multi-pointing. Therefore, image processing does not require intensive computing which allows us to use a stand-alone system (including a 32 bit RISK processor). The results of the proposed method show that effective control navigation of applications such as Google Earth or Google Maps can be achieved. Keywords: Multi-Pointing, Hand Recognition, Human-Computer Interaction.

1 Introduction Multi-pointing on a computer screen is an effective inputting method that is generating considerable research interest. Multi-pointing studies have increased recently due to the development of touch-pads and sophisticated camera applications. The latest display screens, used in personal computers, are typically liquid crystal displays(LCDs), which are also used as the displays in embedded systems. Embedded systems usually use seven inch LCD screens with touch-pads. Unfortunately, the normal display systems of personal computers cannot use such touch-pads because, in J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 520–525, 2009. © Springer-Verlag Berlin Heidelberg 2009

Multi-pointing Method Using a Desk Lamp and Single Camera

521

general, PC’ display resolution is much greater then the resolution in embedded systems. Research on camera-based multi-pointing is limited since camera-based methods can increase the frame rate of the system, thereby increasing hardware configuration requirements for real-time performance. The input devices used in PC systems normally include keyboards and mice, and possibly joysticks and tracker-ball devices [1]-[4]. These input methods do not support multi-pointing and commercial touchpads have limited multi-pointing ability with which to support desktop PC display systems.

2 Motivation The proposed multi-pointing method is performed using dual hand recognition, using a table lamp and a single CMOS camera. The aim is to support multi-pointing using a normal LCD monitor through a simple hardware configuration without intensive image processing. Section III describes multi-pointing environments and the multipointing strategy, including skin color information extraction, dual hand recognition, and hand position estimation. Section IV discusses the implementation scenario and the required interface commands. Section V presents results from our proposed multipointing system, and some conclusions from our research.

3 Simple Strategy for Multi-pointing 3.1 Overview of Multi-pointing Environments The experimental environments considered in this study are shown in Fig. 1. A CMOS camera mounted with a table lamp views the LCD monitor of the PC'. The images captured by the camera are transmitted to a stand-alone system consisting of a Marvell’s PXA320 processor through DMA, via the PXA320's Quick Capture Interface port. Dual hand recognition can be obtained through the received CMOS camera’s image using the multi-pointing strategy described in this section. We show that the Google Earth application program can be controlled by the commands generated from the dual hand recognition system. This allows screen control commands to move forward and backward, turn left and right, and zoom in and out on the axis of the PC screen. These screen-based control commands can be control Google Earth program, and can also control the navigational aspects within the Google Maps application. Figure 2 shows a block diagram for the experimental environments. The blocks of Fig. 2 represent all development and applications. The main components of proposed multi-pointing method are shown in five bold rectangle blocks. Our proposed multipointing method system is not heavy computing performance, and can be embedded to small size.

522

T. Song et al.

Fig. 1. Multi-pointing experimental environments

Fig. 2. Multi-pointing system block diagram

3.2 Multi-pointing Strategy A. Skin color extraction Our skin color extraction approach uses a red-green-blue (RGB) color-based threshold method, whereas previous studies used the HSV color-space for better

Multi-pointing Method Using a Desk Lamp and Single Camera

523

illumination tolerance [5], [6]. The working conditions in [5], [6] changed very quickly, the authors detected the skin in real videos, but the algorithm needed an extensive skin color database requiring many skin sample images. To avoid this dependence on an extensive database, our approach uses a much narrower working environment and requires much higher accuracy. The skin color extraction can be used on an input image by manually extracting a trial color so that the RGB characteristics of the hand’s color can be obtained, and the threshold values can be used (Fig. 3). The proposed method is performed in two steps: 1. Hand skin color extraction. 2. Skin color threshold value created by the proposed method: sample images are captured in the same working environment as the target hand color extraction program. After the skin color extraction step, background and noise effects remain in the resulting image, and future work will address these problems by adopting appropriate filtering of the affected pixels. B. Background and small noise removal This method is referred to in [7] and follows these steps to remove any small noise effects: 1. Every connected component is extracted. 2. Small connected components containing less than a pre-defined number of pixels are removed. End of this step, there are still some big components remaining; these will be filtered next. C. Hand recognition The relative information between any two components is considered in the recognition of the hands (Fig. 5). The hand extraction process described in [7] states that many conditions exist for effective hand recognition, and thus many evaluation functions must be applied to assess the result. However, our proposed method removes many of these hand recognition conditions by using a virtual grid, as shown in Fig. 4. In this way, an evaluation function to recognize the hands can be used to detect some key features that form the important connected components.

Fig. 3. Color Extraction

Fig. 4. Virtual Grid

Fig. 5. Multi-Pointing

524

T. Song et al.

The proposed hand recognition is performed in three steps: 1. Size of a component A: The number of pixels in component A. 2. Size index of a component A: The index of A, using its size information, compared to the other remaining components. 3. Estimate the start and end positions of a component A: Estimate the component’s start and end positions using the virtual grid on the screen.

4 Implementation The intended implementation of the proposed multi-pointing method is to control the Google Earth application. Google Earth is a global map searching program, based on the World Wide Web. Our implementation can control Google Earth in the following ways: move up image (by the forward command); move down image (by the backward command); turn left image (by using the left command); and turn right image (by using the right command). It can also control the image zoom in and out by using the zoom-in/out commands. Each command generated by the results of the hand recognition procedure is shown in Fig. 6.

Fig. 6. Implementation Google Earth Control

5 Conclusion The proposed multi-pointing method, using a desk lamp and single camera, can generate four directional commands: forward, backward, left, and right. In addition, it can generate zoom-in and zoom-out commands, based on human hand position and figure extraction. The Google Earth application controlled using the new approach working in real-time. This study focused on developing an effective input method using a simple multi-pointing approach which can promote the user experience and be fun to use when communicating with computer applications. It is hoped that the

Multi-pointing Method Using a Desk Lamp and Single Camera

525

proposed multi-pointing method can be a useful and effective way to replace traditional input devices such as keyboards, mice, and joysticks. Acknowledgments. This research was supported by MKE, Korea under ITRC IITA2009-(C1090-0902-0046).

References 1. Zhai, S.: User performance in Relation to 3D Input Device Design. Computer Graphics 32(4), 50–54 (1998) 2. Zhai, S., Kandogan, E., Barton, A.S., Selker, T.: Design and Experimentation of a Bimanual 3D Navigation Interface. Journal of Visual Languages and Computing, 3–17 (October 1999) 3. Ku, J.Y., Hong, J.P.: A Study on the Controlling Method for Remote Rehabilitation Assisting Mobile Robot Using Force Reflection Joystick. Journal of the Institute of Electronics Engineers of Korea 40, 26–34 (2003) 4. Lapointe, J.F., Vinson, N.G.: Effects of joystick mapping and field-of-view on human performance in virtual walkthroughs. In: Proceeding of 1st International Symposium on 3D Data Processing Visualization and Transmission Padova, Italia, June 18-21 (2002) 5. Sigal, L., Sclaroff, S., Athitsos, V.: Skin Color-Based Video Segmentation under TimeVarying Illumination. IEEE Trans. Pattern Analysis and Machine Intelligence, 862–877 (2004) 6. Dadgostar, F., Barczak, A.L.C., Sarrafzadeh, A.: A Color Hand Gesture Database for Evaluating and Improving Algorithms on Hand Gesture and Posture Recognition. Research Letters in the Information and Mathematical Sciences 7, 127–134 (2005) 7. Pham, T.C., Pham, X.D., Nguyen, D.D., Jin, S.H., Jeon, J.W.: Dual Hand Extraction Using Skin Color and Stereo Information. In: IEEE International Conference on Robotics and Biomimetics, Bangkok, Thailand, February 21-26 (accepted, 2009)

Communication Grill/Salon: Hybrid Physical/Digital Artifacts for Stimulating Spontaneous Real World Communication Koh Sueda1, Koji Ishii2, Takashi Miyaki1, and Jun Rekimoto1 1

The University of Tokyo, 7-3-1, Hongo, Bunkyo-ku, Tokyo, Japan [email protected], [email protected], {miyaki,rekimoto}@acm.org

Abstract. One of the problems encountered in face-to-face communication involves conversational imbalances among the participants caused by differences in conversational interests and social positions. It is common for us not to be able to communicate well with an unfamiliar person. On the other hand, old customs in the real world, such as the Japanese tea ceremony, effectively use physical artifacts to enable smoother conversation. In this project, we designed two communication systems that facilitate casual communication using physical/digital artifacts, such as a meal and text-chat, in order to clarify that real world communication can be supported by digital technology. The first system, called the "Communication Grill," connects a grill for cooking meat to a chat system. The grill is heated by the chatting activity. Thus, people must continue conversing to roast the meat. The second system is called the “Communication Salon.” It is a computer-enhanced tea ceremony with a chat screen displayed at a tearoom. Using these systems, we conducted user evaluations at SIGGRAPH and other open events. Based on the chat logs at these events, we found that conversational topics gradually shifted from topics about the systems to more general topics. An analysis of these chat logs revealed that the participants began to communicate spontaneously using this system. Keywords: Augmented reality, Chat, Chat-augmented meal, merging virtual and real, Communication Grill/Salon.

1 Introduction Face-to-face communication is very important in our daily life, even though online communication technologies now support communication. One of the problems encountered in face-to-face communication involves conversational imbalances among the participants caused by differences in conversational interests, social positions and so on. Some situations require casual conversation, such as brainstorming and blind dates. At the same time, some researchers have reported that virtuality activates communities in online spaces, such as the Internet [1][2]. The virtuality allows an individual to be separated from the real world. Therefore, in J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 526–535, 2009. © Springer-Verlag Berlin Heidelberg 2009

Communication Grill/Salon: Hybrid Physical/Digital Artifacts

527

online spaces, it is easy to create a unified community that transcends the conditions of the real world, such as appearance, culture, and social position [3]. Other problems that often cause us to hesitate to communicate with others involve differences in conversational interests or meeting someone for the first time, regardless of whether it is online or face-to-face. It has been found that some games or ceremonies transcend these hesitations by imposing conversational restrictions in the form of rules or goals. In this project, we proposed a system that stimulates spontaneous real world communication by inserting the advantages of an online communication system into face-to-face conversation and placing restrictions on the communication.

Fig. 1. The places used for the demonstrations: the systems require conversation to eat or enjoy a cup of tea. The imbalance in statements was found to be less significant than in traditional chat systems, and the conversational topics gradually shifted from topics about the systems to more general topics. Communication Grill Chang-tei (Left); Communication Salon Chang-tei (Right).

2 Approach In Japan, the tea ceremony has many rules and artifacts that provide topics for communication between the participants. These methods are seen in the tea ceremony because it is based on strict aesthetics and ideology. The tea ceremony was developed to optimize face-to-face communication in a feudal society [4]. It has been found that these kinds of methods stimulate casual face-to-face communication in the real world. These methods aim to optimize communication using ceremonies and the design of a physical space to reduce the gaps in the positions and social conditions of the participants.

Fig. 2. Restrictive rules are used to optimize a tearoom for face-to-face communication while every guest and a master enjoy a cup of tea (Left). Confessional: the penitent can confess anonymously by stepping into a small, enclosed booth for a face-to-face conversation (Right).

528

K. Sueda et al.

These are seen in the features of a Zen temple, including its tearoom and confessional in the cathedral, which is a venue for courtesy ceremonies (Figure 2: left). These methods have the following points in common: 1. They provide spatial or mental immersion by means of a restrictive procedure. 2. Daily activities, including conversation and a meal, are conducted under non-daily circumstances. In addition, real-time online communication, such as text-chat, has these common points [5][6]. Text-chat provides only verbal information under real-time communication. This kind of restriction forces participants to imagine non-verbal information more profoundly than during normal face-to-face communication. In this study, we focused on and considered the potentials of these two points. Our aim is to propose a system that allows spontaneous real world communication by the fusion of a text-chat and daily communication, such as during a meal. We designed two communication systems, the "Communication Grill" and "Communication Salon," to realize this idea. The “Communication Grill/Salon” stimulates a conversation by using systems that require the participants to chat in order to eat. Demonstrations were conducted and evaluated at exhibitions that included ACM SIGGRAPH and Ars Electronica. As a result, we found the following two points by stimulating casual conversations between participants using the “Restriction” that the system required conversation in order to eat. • The bonds between participants were deepened as a result of the meal. • The common goal such as enjoying meal or tea bring the participant positive utterance whether they were first meetings / join on the way or not. 2.1 Conceptual Background “Communication Grill/ Salon” makes participants to recognize a gap between verbal communication and nonverbal communication by the system requiring irrational textchat on a face-to-face communication in the real space. When the participants recognize inconsistent situations, they try to concentrate on the nonverbal information from each other. According to Mehrabian’s report, the spoken word is only 7% effective during face-to-face communication when we receive ambiguous messages, such as the words spoken are inconsistent with the tone of voice or body language of the speaker [7]. Normally, face-to-face conversations contain visual, tone of voice, and verbal information. However, it is difficult to recognize the importance of nonverbal information, because we usually receive both verbal and nonverbal information at the same time, and subconsciously. On the other hand, the “Communication Grill/Salon” provides a way for the participants to receive the two forms of information separately. Requiring conversation to eat is an easy way to overcome an awkward situation, because the participants have to talk with others quickly to heat up the grill. Thus the participants try to receive more nonverbal information from each other, and thus deepen their

Communication Grill/Salon: Hybrid Physical/Digital Artifacts

529

mutual understanding by more profound observations. This is why the system imposes restrictions on the participants, such as requiring them to converse in order to eat or drink. The participants become sensitive to nonverbal information because of the requirement of verbal based conversation with the others through the use of this system. The conversations on the absurdity of a system that requires communication in order to eat allowed the participants to gain more insights about each other.

3 System 3.1 Chat System The Communication Grill/Salon Chang-tei system is comprised of chatting devices, such as a browser phone, PDA, and laptop PC, chat applications that control a heater and indicate the remaining power time, and an electric heater powered by the conversation. When these various parts work together via a network, the act of chatting results in the heating of the heater (Figure 3).

Fig. 3. System Diagram: (The countdown indicators) The timer and meter indicating the time remaining before shutdown and urging the participants to converse

3.2 Interaction Design This system is equipped with a timer that controls the heating element of the heater. This timer turns off the power to the heater if the conversation stops. The chat application interface provides a countdown timer that indicates the remaining time before the power shutdown, urging the participants to converse (see Figure 4:right and Figure 5:center). In addition, the heater is equipped with a red pilot gauge that visually indicates the operating status (Figure 5). 3.3 Communication Grill The Communication Grill consists of an electric grill controlled by the participants’ text-chat, a chat application to communicate with the others sitting with them, a

530

K. Sueda et al.

network (enabling the use of non-Internet connections), and the tableware for eating (Figure 4).

Fig. 4. The communication grill (all in one edition): an LCD (1) to indicate an IP address, an Ethernet port (2) to receive operating signals from chat applications, and an indicator lamp (3) (Left). The Communication Grill Application: a screen shot of the chat application. The bar meter indicates the remaining time before power shutdown, implemented on the right side of the window (Right).

3.4 Communication Salon There are some differences between the Communication Grill and Communication Salon. These two systems provide almost the same interaction, but the interfaces are different. The Communication Salon consists of an Internet connected chat server, a power controller for the heater, Internet connected client devices, a hanging scroll onto which the chat is projected and the remaining time before power shutdown is indicated, and a tea set (Figure 5). One of the big differences compared to the communication grill is the hanging scroll display of chat logs.

Fig. 5. Equipment for the Communication Salon: power-controlling device (Left), hanging scroll chat-log display (Center), and electric heater (Right)

The participant are only able to join and communicate with each other in the same location, because the chat logs are only displayed on a hanging scroll in the tearoom. The hanging scroll also shows a countdown timer for the remaining time before power shutdown. Therefore the hanging scroll is very important in this system. Every participant has to watch the hanging scroll throughout the conversation. In the typical tea ceremony, a hanging scroll is a symbol of the event, and each participant respects

Communication Grill/Salon: Hybrid Physical/Digital Artifacts

531

the hanging scroll, which depicts the theme of the event. Thus the participants see the symbol of the conversation automatically through the use of the Communication Salon. 3.5 Chat Application The chat application for the Communication Grill was installed on a client PC, to provide a normal chat system and a status meter to indicate remaining operating time for the grill (see Figure 4). This chat application allowed the system to be used with a network, and controlled the timer to operate the electric grill (a remark turned on the heater for 5 s). The bar meter, which used animation to indicate the remaining operating time, was placed on the right side of the chat application window. The chat application was supported by multicast IP and so did not need chat servers and an Internet connection. The Communication Salon requires only an Internet connection, a power control device (Figure 5 (left)) for the electric heater, a client device to display the chat-log (Figure 5 (center)), and an electric heater (Figure 5 (right)). 3.6

Demonstrations

The Communication Grill was demonstrated at several exhibitions at open events, including Ars Electronica 2003. We sampled over 16,000 remarks in English or Japanese from the participants’ conversation logs, which contained first meetings. In this research, we analyzed 6,808 remarks from a total of 16 sessions and 33 participants. The Communication Salon was exhibited at open events that included Yokohama EIZONE 2007 and SIGGRAPH 2007. Just as with the Communication Grill, we sampled over 5,000 remarks in English or Japanese from the participants’ conversation logs, which contained first meetings. In this research, we analyzed 2,200 remarks from a total of 31 sessions and 57 participants. Individuals were invited to participate at the venues of the exhibitions.

4 Results/Evaluation 4.1 Overview The results of the chat log analysis showed that almost all of the participants tried to communicate with the others spontaneously. The following observations were also made. (1) There was not a big difference in the remark frequency based on whether or not they were meeting for the first time. (2) Conversational topics gradually shifted from topics about the systems to more general topics. (3) It was easy to shift to a personal topic because of the virtuality of the chat, even though the participants were in the same space. (4) Some of the participants noticed and mentioned the effects of the system. 4.2 Differences between the Grill and Salon The results with the Grill and Salon did not show big differences, except that the speaking frequency of the Salon was longer than the Grill’s. This difference was

532

K. Sueda et al.

caused by the fact that the Salon system worked longer than the Grill per message (Grill: 5 s per message; Salon: 20 s per message). 4.3 Chat Log Analysis As can be seen in Figure 6, there was no relationship between the remark frequencies and participation time. In the figure, the remark frequencies of two participants (they joined/left the conversation) were not lower than the others. (All of the participants were meeting for the first time and all of conversations were in English.) On the other hand, participant A (a non-native English speaker) spoke less than the other participants. This result showed two features of the system: it did not support language differences and did not stimulate meaningless remarks just for operating the heater.

Fig. 6. Relations between remarks and participation time: there was no relationship between the remark frequencies and participation time (participant B joined in the middle of the conversation)

Figure 7 indicates that the participants had become friendly by degrees through the use of this system. This graph classifies the contents of the conversations into three categories: “The topic was operating the heater,” “A topic that resulted from chatting (e.g. impressions about eating, chatting to maintain the heat)” and “other general topics.” The graph indicates the percentages with a 20 remark resolution. As shown in

Fig. 7. The shifting process of conversation topics: the conversational topics gradually shifted from topics about the systems to more general topics

Communication Grill/Salon: Hybrid Physical/Digital Artifacts

533

the figure, the conversations began by exchanging greetings and talking about the operation of the heater. After a while, the participants began to talk more about the topic of eating. Finally, the participants began to talk about personal topics. 4.4 Development of the Conversations Figure 8 shows samples from the beginning and middle of a chat log. As can be seen in the figure, the conversation had been changing to personal topics. In addition, the participants become to talk more naturally. These results show the possibility that the system enables spontaneous interaction by providing a sense of achievement through the use of restrictions. ---------------Early stage---------------7 b says: lets get it cook 8 b says: huhu 9 a says: huh 10 b says: try 11 a says: try 12 a says: ? ---------------After a while---------------143 a says: u wanna JP 144 a says: girlfriend? 145 b says: lost 146 b says: just a week ago 147 a says: uh hmm 148 b says: have a tea

---------------- Early stage -------------3 a says: I’m very hungry 4 b says: you haveta chat!! 5 c says: hungry… 6 b says: more and more~~~~ 7 a says: roast, roast --------------- After a while ---------------1095 b says: tell us beginning of your love..! 1096 d says: sausages’ve gotten coooool 1097 b says: anyway,, 1098 b says: u tell us 1099 a says: I don’t have a girl friend:-P

---------------- Early stage ---------------1 a says: Hello 2 b says: hi! 3 a says: How are you doing? 4 b says: bit sleepy... 5 a says: Have you been here all day? ---------------- After a while ---------------68 a says: did you see takeo igarashi speak at all? 69 b says: i m also 1st time SIG 70 b says: origarashi? 71 a says maybe. I may have butchered his name 72 a says: He made the teddy sketching system

[a: Male, b: Male, c: Female, d: female (Japanese) ] (Left-upside) [a: Male (Austrian), b: Male (Japanese) (Right-upside) [a: Male (Canadian), b: Male (Japanese) ] (Downside)

Fig. 8. Shift of conversational topics: Comparisons of the chat contents, early stage and after a while. As shown in the lists, the conversational topics gradually shifted from topics about the systems to more general topics.

5 Discussion 5.1 “Communication Grill/ Salon” System The goal of this text-chat system was to stimulate communication by mixing the virtual and the real worlds. This system is expected further development in the point of activating communications by situation such as urge, restriction, and reward. These situations provide environments where a user observes, and then considers others. Typical communication technologies pursue speed, efficiency, and accuracy. These are very important ways to promote communication. In this project, we proposed another way to promote communication that considered it from the viewpoint of culture, habits, and rewards. There are some studies that have surveyed the usability

534

K. Sueda et al.

of a communication system from these aspects [8][9]. These achievements should be applied to a greater extent in the HCI design process. 5.2 Future Works These features can be applied to communication that requires casual and spontaneous remarks, such as counseling and brainstorming. They can also be applied to amusement communications, because the restrictions of the system promote the discernment of other participants and the shared environment. It is a good example of using a restriction to trigger the same potential that is displayed by a blind person, who makes good use of aural information and memory to recognize environments [10]. This kind of example shows the possibility of "Real-World User Interfaces" that effectively use environmental information. In addition, this system requires quantitative analysis by comparing it with a typical online chat in real space or surveying related projects [11].

6 Conclusion The proposed Communication Grill/Salon is a system that promotes spontaneous remarks by applying the online text-chat virtuality to face-to-face communication. Observations of the chat logs showed that participants began with casual remarks about using the system. (1) The imbalance of remarks was improved by using this system in conversations, even under circumstances where the participants would normally hesitate to speak, such as first meetings or joining an ongoing conversation. (2) The system provided the participants with a motivation for conversation by requiring text-chat in order to eat. (3) In addition, as result of the chatting, the groups became more communicative throughout the meal. In the future, this research should pursue another new interface design that promotes spontaneous communication by applying the culture or customs of daily life.

Acknowledgments We thank all of the participants on this project, and all of the publication’s support personnel and staff, who provided helpful comments on this paper. Some of the references cited in this paper are included for illustrative purposes only.

References 1. Kraut, R., Patterson, M., Lundmark, V., Kiesler, S., Mukophadhyay, T., Scherlis, W.: Internet Paradox; A Social Technology that Reduces Social Involvement and Psychological Well-being? American Psychologist 53(9), 1017–1031 (1998) 2. Bordia, P.: Face-to-Face Versus Computer-Mediated Communication: A Synthesis of the Experimental Literature. Journal of Business Communication 34(1), 99–118 (1997)

Communication Grill/Salon: Hybrid Physical/Digital Artifacts

535

3. Kraut, R., Kiesler, S., et al.: Internet Paradox Revisited. Journal of Social Issues 58(1), 49– 74 (Spring 2002) 4. Okakura, K.: The book of tea. G.P. Putnam (1906) 5. Hall, E.T.: The Hidden Dimension. Doubleday, New York (1966) 6. Turkle, S.: Life on the Screen: Identity in the Age of the Internet. Simon & Schuster Trade (1995) 7. Mehrabian, A.: Silent Messages. Wadsworth Publishing Company, Inc., Belmont (1971) 8. Mainwaring, S., March, W., Maurer, B.: From meiwaku to tokushita!: lessons for digital money design from Japan. In: Proc. CHI 2008, pp. 49–74 (2008) 9. Wyche, S.P., Aoki, P.M., Grinter, R.E.: Re-placing faith: reconsidering the secularreligious use divide in the United States and Kenya. In: Proc. CHI 2008, pp. 11–20 (2008) 10. Ito, K., Miyamoto, E., Tanahashi, K.: Ample information picked up by blind travelers at the train station. In: International Conference on Environment Behavior Studies for the 21st Century, pp. 323–328 (1997) 11. Rekimoto, J., Ayatsuka, Y., Uoi, H., Arai, T.: Adding another communication channel to reality: an experience with a chat-augmented conference. In: CHI 1998 Summary (1998)

Motion Capture System Using an Optical Resolver Takuji Tokiwa1,4, Masashi Yoshidzumi2, Hideaki Nii3, Maki Sugimoto4, and Masahiko Inami5 1

School of Engineering, Tokyo University 7-3-1, Hongo, Bunkyo-ku, Tokyo 113-8656, Japan Department of Mechanical Engineering and Intelligent Systems [email protected] 2 University of Electro-Communications 1-5-1, Chofugaoka, Chofu-shi, Tokyo 182-8585 Japan [email protected] 3 Graduate School of Information Science and Technology, Tokyo University 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan [email protected] 4 Media Design Institute, Graduate School of Media Design, Keio University 4-1-1 Hiyoshi, Kohoku-ku, Yokohama, Kanagawa 223-8526, Japan [email protected] 5 Graduate School of Media Design, Keio University 4-1-1 Hiyoshi, Kohoku-ku, Yokohama, Kanagawa 223-8526, Japan [email protected]

Abstract. In this paper, we present a novel position measurement method that makes use of a couple of plane light sources created from an IR-LED matrix array and a photo-detector. The light sources emit light with the same frequency, but different phases, while the optical axes of the sources are set up orthogonally. Then, the signal place is diffused by the space with phase differences in each position. Finally, the signal received by the photo-detector is analyzed to determine the position. Keywords: Motion Capture, Position Detection.

1 Introduction With ubiquitous computing, the input and output of information for users is performed through various interfaces. Interfaces that are optimized for the location and circumstances of users are necessary, as well as a standardized interface. An optimized interface facilitates the input and output of information, disperses it effectively and offers the user the opportunity for new knowledge. It is important in creating such an interface to ensure that the design retains basic functionality, that is, a combination of the accepted requirements and an environment that makes this possible, so that information appliances can be attached if necessary. Besides, an information environment not only responds to requests received from a user through an information terminal device, but can also detect via various sensors J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 536–543, 2009. © Springer-Verlag Berlin Heidelberg 2009

Motion Capture System Using an Optical Resolver

537

the position and actions of the user if necessary. The information given to a user is presented by means of information terminal devices, and also a combination of information appliances in the information environment. The presenting method depends on the location, circumstances and requests from the user. It is important to realize such an environment that has the technology to detect both a user and the position of an information appliance easily in the information environment. Consequently, techniques have been proposed and developed to detect a user and position, such as how to use RFID tags[1,2], how a marker is detected using image analysis technology[3-5], how distance is measured by supersonic waves, and so on, together with resolvers and “Polhemus”[6]. These techniques, however, have several weaknesses. For example, measurement precision depends on measurement time, while the system architecture is complicated as detection requires high-performance computers for analysis of images and data. Therefore, a technique has been suggested to measure information projected in the real world[7,8]. The measurement system is this case is simplified and easy to use. Recently we proposed a technique that uses two plane light sources which flash on and off continually with a fixed phase difference and the same frequency. This creates a signal field with phase difference changes in a particular direction in the space, that makes the detection of a position possible, when the signal is measured by a photodetector. Using this technique, device construction is simplified. Moreover, detection does not depend on the frame rate as is the case in movie capture using a video camera and position detection is continuous at high speed. The implemented system has, however, been developed only as a proof of concept. The device construction is complicated and cannot really be used in a ubiquitous computing environment. Consequently, in this paper, we examine an alternative method to detect phase difference on a computer to simplify device construction. The paper is organized as follows. The basic concepts of the proposed technique are explained in the next section. An implementation as proof of concept and the results are reported in Section 3. An application of the proposed technique is described in Section 4, while potential future problems are considered in Section 5.

2 Optical Resolver When light from two plane light sources, with a constant phase difference and that flickers with the same frequency, is composed in space, a signal field develops where a phase difference changes in a specific direction. This technique occurs as a result of the following three principles. 1. The light emitted by a plane light source irradiates a finite space and has directionality. The photo-detector changes according to the angle between itself and the plane light source, if the distance to the photo-detector of the received luminance is fixed, and if the photo-detector always faces the center of the plane light source. 2. The synthetic light is measured at the position where the light output from multiple light sources overlaps.

538

T. Tokiwa et al.

3. When two signals, with identical frequency but whose undulating phases differ, are synthesized, a signal is obtained in which only the phase deviates in proportion to the ratio of the strength of the two signals. The uniaxial angle detection system developed from these principles is explained in the next section. 2.1 Uniaxial Angle Detection System The concept of the uniaxial angle detection system is illustrated in Fig. 1. Plane light sources S1 and S2 flicker at Asin(ωt) and Bcos(ωt) (where A and B denote amplitude, and ω is the modulated angular frequency). The luminance, however, does not become negative. Then, the AC component is changed by adding a positive offset to the sinusoid. Henceforth, only the change in this alternating current is noticed. Similarly, the luminance of the other light source is also changed to alternating current.

Fig. 1. Uniaxial Angle Detector

The following are determined as depicted in Fig. 1: the origin O, X-axis, Y-axis, Zaxis, and rotation angle θ. Luminance Ls, which is mesured by a photo-detector placed in the region X,Y > 0 (Z is constant), is defined by Eq. 1. (1)

Motion Capture System Using an Optical Resolver

539

Phase difference φ is defined as φ = tan-1 (Bsin(θ) / Acos(θ)). This becomes φ = θ, if A = B. Angle θ, measured between the X-axis and the original direction, is equal to the phase difference of the composite signal and the drive signal of the source of light that is detected by the light receiving element. The wavefront develops radially at the Z-axis center in the first quadrant of the XY plane. The proper phase difference φ is measurable by the phase detection. It is possible to ensure that the sampling rate does not depend on the modulation frequency of the light source. In addition, in practice, a light receiving element exhibits directivity. However, the problem of directivity is removed by the diffusion board which is installed at the front of the photo-detector and spherical solar batteries without directivity. As a result, the directionality of the photo-detector is not considered in Eq. 1. 2.2 Development of Uniaxial Angle Detector A block diagram of the system produced is shown in Fig. 2. A source of light arranged as a chip type infrared LED with wide directivity in the array state is substituted for the plane light sources. A “CL-190IRS-X” infrared LEDm manufactured by CITIZEN ELECTRONICS CO., LTD., was used as the LED. The plane light source mounted in the LED array is shown in Fig. 3. According to the basic principles given in Section 2, the light source is driven by a sine wave. However, it is not easy to allow the luminance of an LED to be changed accurately in the sinusoid. Furthermore, this complicates the composition of the equipment. Thus the LED was actually made to flicker in accordance with a 1[kHz] rectangular wave. The higher harmonics of rectangular wave were cut off by the filter in the phase difference detection circuit. A “TPS615” phototransistor manufactured by the Toshiba Semiconductor Company was used as the photo-detector. This component has directionality. The photo-detector was installed in such a way that it always faces the direction of the intersection of the light sources in the experiment. A “CD-505R2”

Fig. 2. System block diagram

540

T. Tokiwa et al.

Fig. 3. PlaneLightSource

phase detector from NF Corporation. was used as a noise filter and for cutting out the higher harmonics and elements in the phase difference detection circuit [9]. The “CD505R2” consists of the following: input differential amplifier, two post amplifiers, bandpass filter, phase shifter, phase detector, and lowpass filter. It is possible to decide the characteristics of each circuit by its resistance and the capacitor of the outside attachment. The phase-detector uses a rectangular wave in the reference signal, takes the inner product with the signal passed through the bandpass filter, and finally outputs a direct current through the lowpass filter.

3 Improvements to the System The system described in Section 2 was developed to verify the concepts of the proposal, and uses a specialized custom-made analog circuit with masking. In addition, an AD interface board is used to allow the measurement result to be analyzed by a computer. These components are expensive, and the equipment itself is bulky, making it difficult to include the system in other equipment. A ubiquitous computing environment comprises various components of information-processing equipment. For the proposed technique to be used in such an environment, it needs to be incorporated in this information-processing equipment. For this reason, it needs to be low-cost and small and should be simple to use. On the other hand, computer performance has improved remarkably in recent years, making it possible to carry out digital signal processing, even without the use of specialized circuitry. In the field of media arts, it has been proposed that the computer's audio input port and audio interface for music creation be used instead of the AD interface board [10]. Audio interfaces provide a large number of input ports at low-cost and enable system development using a software environment for realtime signal processing for audio and multimedia, such as “MAX”, “PD (PureData)”, “jMax”, and so on [11-15].

Motion Capture System Using an Optical Resolver

541

Since precision of processing is not guaranteed in these software environments, verification is necessary. Consequently in the simulation, we examined the accuracy of a program for phase difference detection created in “MAX”. The results of the simulation are shown in Fig. 4. In the simulation, a square wave (5[Hz]) was used as the driving signal of the plane light sources. The angle varied between zero and 90 degrees, with an interval of 10 degrees. The signal driving S1 was used as the reference signal. The band pass filter extracts signals of the basic frequency from the reference signal and the signal measured by the photo-detector. Signals extracted using filters are segmented by means of a flip-flop method. The difference in the starting time of the two signals is measured and denotes the phase difference.

Fig. 4. Simulation results. (Signal field generated in a square wave 5[Hz], and measured with sampling rate 96k[Hz])

4 Application of the Proposed Technique 4.1 Two-Dimensional Position Detection System Measurement in two dimensions becomes possible when the installed system is orthogonal to the direction of the signal field formed by two plane light sources. Multiple signal fields can coexist by using a frequency that differs for every signal field. Each signal field can be separated by orthogonal detection and a bandpass filter. All angles and positions can be measured, if the phase difference is detected in every signal field.

542

T. Tokiwa et al.

4.2 Ubiquitous Computing Environment The proposed technique can be used in an indoor ubiquitous computing environment. Plane light sources are included in lighting equipment, while receivers are included in information devices that comprise the ubiquitous computing environment. Then, information devices would be able to grasp positions autonomously, if these are within a signal field irradiated from the light source. If combined in a information terminal for ubiquitous computing that allows a user to feel the direction of the source of information such as CoBIT [16], not only the perception of the user but also the terminal itself can grasp the direction of the source of information.

5 Conclusion Conventional implementations of the proposed position sensing technique depended on a specialized and custom-made analog circuit module and an AD interface board. This made it difficult for incorporation in information devices used in a ubiquitous computing environment. To solve these problems, an experimental phase difference detection program was developed and validated in a simulation in which it was implemented as an audio interface in a realtime audio/multimedia programming environment.

References 1. Tokiwa, T., Tokuhisa, S., Honna, Y., Shinozaki, T., Kusunoki, F., Nishimura, T., Iwatake, T.: Surround CoBIT: A method for presenting auditory information as a virtual acoustic field. In: International Workshop on Smart Appliances and Wearable Computing 2004 (2004) 2. Kusunoki, F., Isyama, A., Tokiwa, K., Nishimura, T.: The sensing board enhanced by interactive sound system for collaborative work. In: Proceedings of the 2004 ACM SIGCHI international Conference on Advances in Computer Entertainment Technology (2004) 3. Kato, H.: Marker tracking and hmd calibration for a video-based augmented reality conferencing system. In: Proc. 2nd IEEE/ACM Int. Workshop on Augmented Reality (IWAR 1999) (1999) 4. Rekimoto, J., Ayatsuka, Y.: CyberCode: Designing Augmented Reality Environments with Visual Tags. In: Designing Augmented Reality Environments (DARE 2000) (2000) 5. Kaltenbrunner, M., Bencina, R.: reacTIVision: A Computer-Vision Framework for TableBased Tangible Interaction. In: Proceedings of the first international conference on Tangible and Embedded Interaction (TEI 2007) (2007) 6. Polhemus: Motion Tracking, 3D Scanning, and Eye Tracking Solutions from Polhemus, http://www.polhemus.com/ 7. Kagotani, G., Kojima, M., Sugimoto, M., Nii, H., Inami, M.: PTS: Projector-based Tracking System, International VR. In: Media Art and Technology 2004 Proceedings, p. 27 (2004)

Motion Capture System Using an Optical Resolver

543

8. Raskar, R.: Rfig lamps: Interacting with a self-describing world via photosensing wireless tags and projectors. ACM Transactions on Graphics (TOG) SIGGRAPH 23, 406–415 (2004) 9. NF Corporation, CD-505R2, http://www.nfcorp.co.jp/english/pro/fm/pha/index.html 10. Jo, K.: Audio Interface as a Device for Physical Computing. In: Audio Mostly 2008 - a conference on Interaction with Sound, Piteå, Sweden, October 22-23, pp. 123–127 (2008) 11. Cycling 1974, Max/MSP (1974), http://www.cycling74.com/products/max5 12. Puckette, M., Apel, T.: Real-time audio analysis tools for Pd and MSP. In: Proceedings of International Computer Music Conference, pp. 109–112. International Computer Music Association, San Francisco (1998) 13. Déchelle, F., Borghesi, R., De Cecco, M., Maggi, E., Rovan, B., Schnel, N.: jMax: A New JAVA-Based Editing and Control System for Real-Time Musical Applications. In: Proceedings of the 1998 International Computer Music Conference. International Computer Music Association, San Francisco (1998) 14. Geiger, G.: PDa: Real Time Signal Processing and Sound Generation on Handheld Devices. In: Proceedings of the International Computer Music Conference, Singapore (2003) 15. Breidenbruecker, M., Geiger, G., Brossier, P., Hazan, A., Barknecht, F., McCormick, C., Nordgren, A.: RjDj, http://www.rjdj.me 16. Nishimura, T., Itoh, H., Nakamura, Y., Yamamoto, Y., Nakashima, H.: A Compact Battery-Less Information Terminal for Real World Interaction. In: Ferscha, A., Mattern, F. (eds.) PERVASIVE 2004. LNCS, vol. 3001, pp. 124–139. Springer, Heidelberg (2004)

The Effects of an Anti-glare Sleeve Installed on Fluorescent Tube Lamps on Glare and Reading Comfort Shiaw-Tsyr Uang1, Cheng-Li Liu2, and Mali Chang1 1 Department of Industrial Engineering and Management, Minghsin University of Science and Technology, Hsinchu 304, Taiwan 2 Department of Industrial Management, Vanung University, Taoyuan 302, Taiwan [email protected], [email protected], [email protected]

Abstract. Our previous study has demonstrated the benefits of a reflective sleeve to redirect lighting and to enhance luminous intensity of fluorescent tube lamps in certain light projecting angles. A reflective sleeve is composed of a plastic reflector and a transparent refractor. However, the intensive centralized lighting may increase the possibilities of producing glare. In this study, the transparent refractor of the sleeve is replaced with a diffuser to compose an anti-glare sleeve. This study adopts measurement, optical software simulation, and experiment methods to investigate the effects of an anti-glare sleeve on redirecting lighting and reducing glare. The results demonstrate that luminous intensity towards viewing objects of a fluorescent tube lamp enhances after adopting an anti-glare sleeve. In addition, software simulation indicates an antiglare sleeve increases light uniformity and reduces glare. The subjective evaluation also shows that florescent tube lamps with anti-glare sleeves produce less light reflection on various papers and more comfortable reading. Keywords: Glare, Reading comfort, Fluorescent tube lamp, Lamp sleeve.

1 Introduction According to previous studies, up to 20% of world's electrical energy consumption is used for lighting purposes [1]. Fluorescent lamps have the advantages of larger light emitting area, more evenly light uniformity, lower temperature on tube surface, similar color to sunlight, and longer life period than incandescent lamps do. Hence, fluorescent lamps are the most commonly and widely used artificial light sources in indoor spaces nowadays [2-4]. Our previous study has demonstrated the benefits of installing a reflective sleeve on a fluorescent tube lamp to redirect lighting and to enhance luminous intensity in certain light projecting angles around 80% [5]. A reflective sleeve is composed of a plastic reflector and a transparent refractor to control light distribution and density. However, the intensive centralized lighting may increase the possibilities of producing glare which cause eye discomfort and/or performance decrement. J.A. Jacko (Ed.): Human-Computer Interaction, Part III, HCII 2009, LNCS 5612, pp. 544–553, 2009. © Springer-Verlag Berlin Heidelberg 2009

The Effects of an Anti-glare Sleeve Installed on Fluorescent Tube Lamps

545

Regarding reducing glare, Japuntich [6] found that the use of a linear polarized light source helps to minimize specular glare by darkening the reflected image of the light source on the document. Theoretical predictions and light measurement analysis of specular glare reduction were compared with empirical results from testing on a panel of humans on semi-gloss finish and matte finish papers. This study showed that with the right alteration of a polarized light source position, specular glare may be significantly reduced, and correlations exist between the theory, empirical measurements and the human response to specular glare reduction. Osterhaus [7] reviewed and discussed the advantages and limitations of using existing glare indices for daylighting conditions. It concluded that available assessment and prediction methods are of limited practical use in daylit situations and currently have no provision for integrated systems that combine daylighting and electric lighting. This paper also presented selected findings from a case study of daylit office environments which identify a number of important design considerations. Iwataa et al. [8] measured subjective response to intense light, or glare caused by a wide source. Three glare indices have been investigated in this study: the Building Research Station glare index, the CIE glare index and the Cornell daylight glare index. They have also examined the glare vote and have proposed a new glare evaluation scale, as well as asking the subjects to vote on the condition's acceptability. The Cornell formula most accurately predicts glare discomfort, but it is found to be inadequate for a range of wide-source glare conditions. Both the discomfort sensation and the glare ratings which they proposed correlate well with the percentage of subjects dissatisfied when looking directly at the light source. Koga Kim [9] proposed a practical method of determining the background luminance in the evaluation of discomfort glare. Two experiments were conducted, a visual sensitivity test and a glare sensitivity test. The results show that the luminance of threshold and the luminance of discomfort glare are mainly determined by the luminance of the immediate background of the source, rather than by the average background luminance. Velds [10] tried to draw up a relation between glare assessments and measured quantities. For this purpose two test rooms were used: one room was occupied by the subject, and the required measuring equipment was placed in the other one. An electronic questionnaire was developed for these studies and installed on the computer of the subject. Continuous measurements were necessary to link subjective assessments to measured quantities that were obtained at the same time. The study showed that the vertical illumination measured near the facade and the average sky luminance measured from the back of the room are good measures to monitor visual comfort under intermediate and overcast sky conditions. In this study, the transparent refractor of the sleeve is replaced with a diffuser to compose a newly designed lamp sleeve, which we called it an anti-glare sleeve (see Fig. 1). The reflector of this sleeve is used to redirect lighting towards viewing objects. On the other hand, the diffuser is used to diffuse lights, thus to reduce glare. Hence, the purpose of the present research is to investigate the effects of an anti-glare sleeve on redirecting lighting and reducing glare.

及

546

S.-T. Uang, C.-L. Liu, and M. Chang

Reflector

Diffuser

Fig. 1. A fluorescent tube lamp with an anti-glare sleeve

2 Methods This study adopts measurement, optical software simulation, and experiment methods to investigate the effects of an anti-glare sleeve on redirecting lighting and reducing glare. These three methods are described below in this section. First, the luminous intensity distribution curves of 10, 20 and 40 watts T8 florescent tube lamps with no sleeve, a reflective sleeve and an anti-glare sleeve in different light projecting angles (0°, 45°, 90°) were measured and recorded by a goniophotometer system. This measurement can obtain the maximum and average luminous intensity (unit: cd), as well as the shape of light distribution. Fig. 2 is the apparatus used in this study and their configuration.

Fig. 2. The measurement apparatus and their configuration

The Effects of an Anti-glare Sleeve Installed on Fluorescent Tube Lamps

547

Next, the luminous intensity collected was imported to the optical software— Lumen Micro 2000. This software was then used to build up simulated lighting condition of one classroom and to calculate illumination levels and glare indexes in order to evaluate the effects in enhancing luminous intensity and reducing glare. Simulation has the advantage in estimating lighting conditions without installing real lamps and lighting fixtures. In addition, Lumen Micro 2000 is capable of computing the glare indexes (visual comfort probability, VCP) of the users. Finally, 30 subjects participated in a laboratory experiment. All 30 subjects have no difficulty in discriminating colors and have normal or corrected-to-normal visual acuity. Subjects evaluated reading effects and their visual comfort while viewing words and graphs on matt finish papers, copy papers, and dowling papers with different kinds of florescent tube lamps (no sleeve, a reflective sleeve and an antiglare sleeve). Subjects experienced all nine (3*3) experimental combinations in random order. Their evaluation was based on a 7-point Likert-type scale, which “1” denotes “strongly disagree” and “7” denotes “strongly agree”. Fig. 3 is our experimental configuration.

Fig. 3. The experimental configuration of this study

3 Results This study adopts measurement, optical software simulation, and experiment methods to investigate the effects of an anti-glare sleeve on redirecting lighting and reducing glare. This section provides the findings from these three research methods respectively.

548

S.-T. Uang, C.-L. Liu, and M. Chang

3.1 Results of Measurement A goniophotometer was used to collect the luminous intensity data of different watts (10W, 20W, 40W) T8 fluorescent tube lamps with different sleeve conditions (no sleeve, a reflector sleeve, an anti-glare sleeve) from -90 to 90 in different light projecting angles (0°, 45°, 90°). 27 paired t tests were conducted to compare the difference of luminous intensity. The results indicate 25 of the 27 t tests are statistically significant. It implies the adoption of sleeve can produce significant difference in luminous intensity. In order to explore this finding further, the luminous intensity diagram curves from -90 to 90 were draw and used to compare the shapes of lighting. These curves demonstrate similar and consistent findings; therefore, only the diagrams of a 10W fluorescent tube lamp in 90 with different sleeve conditions are provided in this paper (shown in Fig. 4~6). The lighting of a fluorescent tube lamp dispersed widely from -90 to 90 (Fig. 4). However, with a reflective sleeve (Fig. 5) or an anti-glare sleeve (Fig. 6) cause the light distribution centralized toward the illuminated surface.

°

°

°

°

°

°

°

Fig. 4. Luminous intensity curve of a 10W fluorescent tube lamp in 90°

Fig. 5. Luminous intensity curve of a 10W fluorescent tube lamp with a reflective sleeve in 90°

The Effects of an Anti-glare Sleeve Installed on Fluorescent Tube Lamps

549

Fig. 6. Luminous intensity curve of a 10W fluorescent tube lamp with an anti-glare sleeve in 90°

The maximum and average luminous intensity (unit: cd) in light projecting angle 0°, 45° and 90° are summarized in Table 1~3. Compared to no sleeve condition, a reflective sleeve or an anti-glare sleeve increases the maximum and average luminous intensity. Besides, the increment of these two sleeves seems to have similar trends. Table 1. The maximum and average luminous intensity in 0° Sleeve Items

Maximum cd Average cd

10W

No sleeve 20W

40W

A reflective sleeve 10W 20W 40W

An anti-glare sleeve 10W 20W 40W

55.4

111.8

179.7

90

211.5

373.4

90.1

219.6

414.9

45.6

91.7

147.8

74.2

174.3

307.9

73.7

180.5

343

Table 2. The maximum and average luminous intensity in 45° Sleeve Items

Maximum cd Average cd

10W

No sleeve 20W

40W

A reflective sleeve 10W 20W 40W

An anti-glare sleeve 10W 20W 40W

55.9

105.9

177.1

95.4

204

317.8

90.6

213.6

412.6

46.0

87.9

144.1

76.1

166.2

258.7

73.8

173.3

338.3

Table 3. The maximum and average luminous intensity in 90° Sleeve Items

Maximum cd Average cd

10W

No sleeve 20W

40W

A reflective sleeve 10W 20W 40W

An anti-glare sleeve 10W 20W 40W

57.2

121.3

206.7

94.4

208.5

347.1

89.5

215.5

395.7

52.9

111.8

171.1

74

159.6

272.4

73

174.7

322.8

550

S.-T. Uang, C.-L. Liu, and M. Chang

3.2 Results of Simulation The simulated space of this research is a classroom of one junior high school in Taiwan. Fig. 7 is the photo of this classroom, and Fig. 8 is the same classroom build up in Lumen Micro 2000.

Fig. 7. The photo of the classroom being simulated

Fig. 8. The simulated classroom by Lumen Micro 2000

The average horizontal illumination measured in this classroom is 251.27 lux, which is closed to the simulated illumination, 255.7 lux. The average horizontal illumination while using reflective sleeves is estimated to be 333.9 lux by Lumen Micro 2000; and 377.2 lux while using anti-glare sleeves. Visual comfort probability (VCP) represents the degree of visual comfort of persons. The larger the VCP value is, the smaller the glare is. Smaller max/min VCP value indicates the uniformity distribution of lighting in a space. The VCP values estimated by Lumen Micro 2000 are summarized in Table 4. An anti-glare sleeve has larger average VCP (54.7) and smaller max/min VCP (3.6). In other words, an antiglare sleeve seems to be beneficial in reducing glare. Table 4. A summary table of the visual comfort probability (VCP) values VCP

Average VCP

Maximum VCP

Minimum VCP

Max/Min VCP

No sleeve

50.5

99

16.4

6.1

A reflective sleeve

50.1

98

20.1

4.9

An anti-glare sleeve

54.7

97.9

27.1

3.6

Sleeve

Unit: Percentage (%)

3.3 Results of Experiment The experiment contained two factors: three kinds of sleeve conditions (Sleeve), no sleeve, a reflective sleeve and an anti-glare sleeve; and three types of paper materials

The Effects of an Anti-glare Sleeve Installed on Fluorescent Tube Lamps

551

(Paper), matt finish papers, copy papers and dowling papers. Each participant (n=30) randomly experienced the sequences of all the 9 combinations of Sleeve and Paper. The 3*3 within-subject two-way ANOVA analyses were conducted to examine the main effects and interactions of Sleeve and Paper on reading effects and visual comfort. Reading effects. There are four subjective ratings related to reading effects as described in the followings. An ANOVA on “I can clearly read the words on the paper.” revealed one significant interaction between Sleeve and Paper, F(4,116)=5.619, p