Lecture Notes in Artificial Intelligence Edited by J. G. Carbonell and J. Siekmann
Subseries of Lecture Notes in Computer Science
3814
Mark Maybury Oliviero Stock Wolfgang Wahlster (Eds.)
Intelligent Technologies for Interactive Entertainment First International Conference, INTETAIN 2005 Madonna di Campiglio, Italy, November 30 – December 2, 2005 Proceedings
13
Series Editors Jaime G. Carbonell, Carnegie Mellon University, Pittsburgh, PA, USA Jörg Siekmann, University of Saarland, Saarbrücken, Germany Volume Editors Mark Maybury The MITRE Corporation, Information Technology Center (ITC) 202 Burlington Road, Bedford, MA 01730-1420, USA E-mail:
[email protected] Oliviero Stock Center for Scientific and Technological Research (ITC-irst) via Sommarive 18, 38050 Povo (Trento), Italy E-mail:
[email protected] Wolfgang Wahlster Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI) Stuhlsatzenhausweg 3, 66123 Saarbrücken, Germany E-mail:
[email protected] Library of Congress Control Number: 2005936394
CR Subject Classification (1998): I.2, H.5, H.4, H.3, I.3, I.7, J.5 ISSN ISBN-10 ISBN-13
0302-9743 3-540-30509-2 Springer Berlin Heidelberg New York 978-3-540-30509-5 Springer Berlin Heidelberg New York
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springeronline.com © Springer-Verlag Berlin Heidelberg 2005 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 11590323 06/3142 543210
Preface
From November 30 to December 2, 2005, INTETAIN 2005 was held in beautiful Madonna di Campiglio, on the majestic mountains of the Province of Trento, Italy. The idea to hold the first international conference that would have as topic “Intelligent Technologies for Interactive Entertainment” seemed to be timely. In the previous couple of years there had been other more specific — or more generic — events where some of the relevant themes had made it to the front stage. With INTETAIN we were aiming at establishing a conference where intelligent computational technologies are at the basis of any interactive application for entertainment. As “intelligent computational technologies” we mean adaptive media presentations, recommendation systems in media scalable crossmedia, affective user interfaces, intelligent speech interfaces, tele-presence in entertainment, collaborative user models and group behavior, collaborative and virtual environments, crossdomain user models, animation and virtual characters, holographic interfaces, augmented, virtual and mixed reality, computer graphics and multimedia, pervasive multimedia, creative language environments, computational humor, and so on. We also believe that there is an important role for novel underlying interactive device technologies, for example, mobile devices, home entertainment centers, haptic devices, wall screen displays, holographic displays, distributed smart sensors, immersive screens and wearable devices. Interactive applications for entertainment include, but are certainly not limited to, intelligent interactive games, intelligent music systems, interactive cinema, edutainment, interactive art, interactive museum guides, city and tourism explorer assistants, shopping assistants, interactive real TV, interactive social networks, interactive storytelling, personal diaries, websites and blogs, and comprehensive assisting environments for special groups (challenged, children, the elderly). The conference attracted a good number of the best practitioners from throughout the world. Papers were submitted from Europe, Asia and America. Twenty-one long papers were accepted out of 39 submissions, making it a good-quality program. To this we added 15 short papers presented as posters, and a rich program of live system demonstrations. The program also included invited speakers, special events such as a design garage, where participants performed in groups a hands-on limited task of designing intelligent entertainment applications; and also a chess challenge between Almira Skripchenko, the women’s European champion, and Deep Junior, the 2004 world champion computer chess system. This volume includes all material accepted for presentation: long papers, short papers and demonstration papers, separated into three sections.
VI
Preface
We thank the colleagues of the international Program Committee for their active contribution in the selection process of long and short papers, and Tsvi Kuflik and Carlo Strapparava for taking care of the selection of the demonstration papers included in the special section. We gratefully acknowledge the sponsorship of this event by CREATE-NET, by ITC-irst, and by DFKI (VHNET). Finally, we would like to thank Dina Goren-Bar and Oscar Mayora for their hard work in organizing the conference,and Susana Otero for her help in putting together the camera-ready material.
December 2005
Mark Maybury Oliviero Stock Wolfgang Wahlster Program Co-chairs INTETAIN 2005
Organization
Organizing Committee Steering Committee Chair Imrich Chlamtac
Create-Net Research Consortium, Trento, Italy.
General Chairs Dina Goren-Bar
Oscar Mayora
Ben-Gurion University of the Negev, Israel. Center for Scientific and Technological Research (ITC-irst), Trento, Italy. Create-Net Research Consortium, Trento, Italy.
Program Chairs Mark Maybury Oliviero Stock Wolfgang Wahlster
Information Technology Center (ITC), MITRE, Bedford, MA, USA. Center for Scientific and Technological Research (ITC-irst), Trento, Italy. German Research Center for AI (DFKI), Saarbr¨ ucken, Germany.
Demo Chairs Tsvi Kuflik Carlo Strapparava
MIS Department, University of Haifa, Haifa, Israel. Center for Scientific and Technological Research (ITC-irst), Trento, Italy.
Design Garage Dina Goren-Bar
Fabio Pianesi
Ben-Gurion University of the Negev, Israel. Center for Scientific and Technological Research (ITC-irst), Trento, Italy. Center for Scientific and Technological Research (ITC-irst), Trento, Italy.
VIII
Organization
Local Organization Chair Giuliana Ucelli
Graphitech, Trento, Italy.
Program Committee Elisabeth Andr´e
Multimedia Concepts and Applications, Institute of Computer Science, Augsburg University, Germany.
Liliana Ardissono
Department of Computer Science, University of Torino, Italy.
Steffi Beckhaus
Interactive Media Group, University of Hamburg, Germany.
Kim Binsted
Information and Computer Sciences Department, University of Hawaii, HI, USA.
Shay Bushinsky
Caesarea Rothschild Institute (CRI), University of Haifa, Israel.
Yang Cai
Visual Intelligence Studio, CYLAB, Carnegie Mellon University, Pittsburgh, PA, USA.
Antonio Camurri
Department of Communication, Computer and System Sciences (DIST), University of Genoa, Italy.
Phil Cohen
Center for Human-Computer Communication (CHCC), Oregon Health Science University, Portland, OR, USA.
Ron Cole
Center for Spoken Language and Understanding (CSLU), University of Boulder, CO, USA.
Bo Dahlbom
The Swedish Research Institute for Information Technology, SITI AB, Stockholm, Sweden.
Dina Goren-Bar
Ben-Gurion University of the Negev, Israel. Center for Scientific and Technological Research (ITC-irst), Trento, Italy.
Marco Gori
Department of Information Engineering, University of Siena, Italy.
Organization
Koiti Hasida
Information Technology Research Institute (ITRI), AIST, Tokyo, Japan.
Kristina H¨ oo¨k
Swedish Institute of Computer Science (SICS), Kista, Sweden.
Tristan Jehan
Hyperinstruments GroMIT, Media Lab, Cambridge, MA, USA.
Lewis Johnson
Center for Advanced Research in Technology for Education at the USC/ISI, Marina del Rey, CA, USA.
Antonio Kr¨ uger
Institute for Geoinformatics, University of M¨ unster, Germany.
Henry Lowood
History and Philosophy of Science Program, Stanford University, CA, USA.
Blair MacIntyre
Graphics and User Interfaces Lab, Georgia Tech, Atlanta, GA, USA.
Don Marinelli
Drama and Arts Management and Computer Science, Entertainment Technology Center (ETC), Carnegie Mellon University, Pittsburgh, PA, USA.
Michael Mateas
College of Computing, Georgia Tech, Atlanta, GA, USA.
Oscar Mayora
Create-Net Research Consortium, Trento, Italy.
Anton Nijholt
Computer Science, University of Twente, Enschede, The Netherlands.
Paolo Petta
Department of Medical Cybernetics and Artificial Intelligence, Austrian Research Institute for Artificial Intelligence, Vienna, Austria.
Charles Rich
Mitsubishi Electric Research Laboratories, Cambridge, MA, USA.
Isaac Rudomin
Computer Graphics Group, Monterrey Institute of Technology, ITESM, Mexico.
Ulrike Spierling
FH/University of Applied Sciences, Erfurt, Germany.
IX
X
Organization
Bill Swartout
Institute for Creative Technologies at USC, Marina del Rey, CA, USA.
Barry Vercoe
Music, Mind and Machine Group, MIT Media Lab, Cambridge, MA, USA.
Yorick Wilks
Computer Science, University of Sheffield, UK.
Kent Wittenburg
Mitsubishi Electric Research Laboratories, Cambridge, MA, USA.
Massimo Zancanaro
Center for Scientific and Technological Research (ITC-irst), Trento, Italy.
Table of Contents
Long Papers COMPASS2008: Multimodal, Multilingual and Crosslingual Interaction for Mobile Tourist Guide Applications Ilhan Aslan, Feiyu Xu, Hans Uszkoreit, Antonio Kr¨ uger, J¨ org Steffen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3
Discovering the European Heritage Through the ChiKho Educational Web Game Francesco Bellotti, Edmondo Ferretti, Alessandro De Gloria . . . . . . . . .
13
Squidball: An Experiment in Large-Scale Motion Capture and Game Design Christoph Bregler, Clothilde Castiglia, Jessica DeVincezo, Roger Luke DuBois, Kevin Feeley, Tom Igoe, Jonathan Meyer, Michael Naimark, Alexandru Postelnicu, Michael Rabinovich, Sally Rosenthal, Katie Salen, Jeremi Sudol, Bo Wright . . . . . . . . . . . . .
23
Generating Ambient Behaviors in Computer Role-Playing Games Maria Cutumisu, Duane Szafron, Jonathan Schaeffer, Matthew McNaughton, Thomas Roy, Curtis Onuczko, Mike Carbonaro . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
34
Telepresence Techniques for Controlling Avatar Motion in First Person Games Henning Groenda, Fabian Nowak, Patrick R¨ oßler, Uwe D. Hanebeck . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
44
Parallel Presentations for Heterogenous User Groups - An Initial User Study Michael Kruppa, Ilhan Aslan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
54
Performing Physical Object References with Migrating Virtual Characters Michael Kruppa, Antonio Kr¨ uger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
64
AI-Mediated Interaction in Virtual Reality Art Jean-luc Lugrin, Marc Cavazza, Mark Palmer, Sean Crooks . . . . . . . . .
74
XII
Table of Contents
Laughter Abounds in the Mouths of Computers: Investigations in Automatic Humor Recognition Rada Mihalcea, Carlo Strapparava . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
84
AmbientBrowser: Web Browser for Everyday Enrichment Mitsuru Minakuchi, Satoshi Nakamura, Katsumi Tanaka . . . . . . . . . . . .
94
Ambient Intelligence in Edutainment: Tangible Interaction with Life-Like Exhibit Guides Alassane Ndiaye, Patrick Gebhard, Michael Kipp, Martin Klesen, Michael Schneider, Wolfgang Wahlster . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 Drawings as Input for Handheld Game Computers Mannes Poel, Job Zwiers, Anton Nijholt, Rudy de Jong, Edward Krooman . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 Let’s Come Together — Social Navigation Behaviors of Virtual and Real Humans Matthias Rehm, Elisabeth Andr´e, Michael Nischt . . . . . . . . . . . . . . . . . . . 124 Interacting with a Virtual Rap Dancer Dennis Reidsma, Anton Nijholt, Rutger Rienks, Hendri Hondorp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 Grounding Emotions in Human-Machine Conversational Systems Giuseppe Riccardi, Dilek Hakkani-T¨ ur . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 Water, Temperature and Proximity Sensing for a Mixed Reality Art Installation Isaac Rudomin, Marissa Diaz, Benjam´ın Hern´ andez, Daniel Rivera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 Geogames: A Conceptual Framework and Tool for the Design of Location-Based Games from Classic Board Games Christoph Schlieder, Peter Kiefer, Sebastian Matyas . . . . . . . . . . . . . . . 164 Disjunctor Selection for One-Line Jokes Jeff Stark, Kim Binsted, Ben Bergen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 Multiplayer Gaming with Mobile Phones - Enhancing User Experience with a Public Screen Hanna Str¨ omberg, Jaana Leikas, Riku Suomela, Veikko Ikonen, Juhani Heinil¨ a . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Table of Contents
XIII
Learning Using Augmented Reality Technology: Multiple Means of Interaction for Teaching Children the Theory of Colours Giuliana Ucelli, Giuseppe Conti, Raffaele De Amicis, Rocco Servidio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 Presenting in Virtual Worlds: Towards an Architecture for a 3D Presenter Explaining 2D-Presented Information Herwin van Welbergen, Anton Nijholt, Dennis Reidsma, Job Zwiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Short Papers Entertainment Personalization Mechanism Through Cross-Domain User Modeling Shlomo Berkovsky, Tsvi Kuflik, Francesco Ricci . . . . . . . . . . . . . . . . . . . . 215 User Interview-Based Progress Evaluation of Two Successive Conversational Agent Prototypes Niels Ole Bernsen, Laila Dybkjær . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 Adding Playful Interaction to Public Spaces Amnon Dekel, Yitzhak Simon, Hila Dar, Ezri Tarazi, Oren Rabinowitz, Yoav Sterman . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 Report on a Museum Tour Report Dina Goren-Bar, Michela Prete . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 A Ubiquitous and Interactive Zoo Guide System Helmut Hlavacs, Franziska Gelies, Daniel Blossey, Bernhard Klein . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 Styling and Real-Time Simulation of Human Hair Yvonne Jung, Christian Kn¨ opfle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 Motivational Strategies for an Intelligent Chess Tutoring System Bruno Lepri, Cesare Rocchi, Massimo Zancanaro . . . . . . . . . . . . . . . . . . 246 Balancing Narrative Control and Autonomy for Virtual Characters in a Game Scenario Markus L¨ ockelt, Elsa Pecourt, Norbert Pfleger . . . . . . . . . . . . . . . . . . . . . 251 Web Content Transformed into Humorous Dialogue-Based TV-Program-Like Content Akiyo Nadamoto, Adam Jatowt, Masaki Hayashi, Katsumi Tanaka . . . 256
XIV
Table of Contents
Content Adaptation for Gradual Web Rendering Satoshi Nakamura, Mitsuru Minakuchi, Katsumi Tanaka . . . . . . . . . . . . 262 Getting the Story Right: Making Computer-Generated Stories More Entertaining K. Oinonen, M. Theune, A. Nijholt, D. Heylen . . . . . . . . . . . . . . . . . . . . 267 Omnipresent Collaborative Virtual Environments for Open Inventor Applications Jan Peˇciva . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 SpatiuMedia: Interacting with Locations Russell Savage, Ophir Tanz, Yang Cai . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 Singing with Your Mobile: From DSP Arrays to Low-Cost Low-Power Chip Sets Barry Vercoe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 Bringing Hollywood to the Driving School: Dynamic Scenario Generation in Simulations and Games I.H.C. Wassink, E.M.A.G. van Dijk, J. Zwiers, A. Nijholt, J. Kuipers, A.O. Brugman . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
Demos Webcrow: A Web-Based Crosswords Solver Giovanni Angelini, Marco Ernandes, Marco Gori . . . . . . . . . . . . . . . . . . 295 COMPASS2008: The Smart Dining Service Ilhan Aslan, Feiyu Xu, J¨ org Steffen, Hans Uszkoreit, Antonio Kr¨ uger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 DaFEx: Database of Facial Expressions Alberto Battocchi, Fabio Pianesi, Dina Goren-Bar . . . . . . . . . . . . . . . . . 303 PeaceMaker: A Video Game to Teach Peace Asi Burak, Eric Keylor, Tim Sweeney . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 A Demonstration of the ScriptEase Approach to Ambient and Perceptive NPC Behaviors in Computer Role-Playing Games Maria Cutumisu, Duane Szafron, Jonathan Schaeffer, Matthew McNaughton, Thomas Roy, Curtis Onuczko, Mike Carbonaro . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
Table of Contents
XV
Multi-user Multi-touch Games on DiamondTouch with the DTFlash Toolkit Alan Esenther, Kent Wittenburg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 Enhancing Social Communication Through Story-Telling Among High-Functioning Children with Autism E. Gal, D. Goren-Bar, E. Gazit, N. Bauminger, A. Cappelletti, F. Pianesi, O. Stock, M. Zancanaro, P.L. Weiss . . . . . . . . . . . . . . . . . . . 320 Tagsocratic: Learning Shared Concepts on the Blogosphere D. Goren-Bar, I. Levi, C. Hayes, P. Avesani . . . . . . . . . . . . . . . . . . . . . . 324 Delegation Based Multimedia Mobile Guide Ilenia Graziola, Cesare Rocchi, Dina Goren-Bar, Fabio Pianesi, Oliviero Stock, Massimo Zancanaro . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328 Personalized Multimedia Information System for Museums and Exhibitions Jochen Martin, Christian Trummer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332 Lets Come Together - Social Navigation Behaviors of Virtual and Real Humans Matthias Rehm, Elisabeth Andr´e, Michael Nischt . . . . . . . . . . . . . . . . . . . 336 Automatic Creation of Humorous Acronyms Oliviero Stock, Carlo Strapparava . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337 Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
COMPASS2008: Multimodal, Multilingual and Crosslingual Interaction for Mobile Tourist Guide Applications Ilhan Aslan1 , Feiyu Xu1 , Hans Uszkoreit1 , Antonio Kr¨ uger1,2, and J¨ org Steffen1 1
2
DFKI GmbH, Germany University of M¨ unster, Germany
Abstract. COMPASS2008 is a general service platform developed to be utilized as the tourist and city explorers assistant within the information services for the Beijing Olympic Games 2008. The main goals of COMPASS2008 are to help foreigners to overcome language barriers in Beijing and assist them in finding information anywhere and anytime they need it. Novel strategies have been developed to exploit the interaction of multimodality, multilinguality and cross-linguality for intelligent information service access and information presentation via mobile devices.
1
Introduction
More and more people rely on Smartphones and PDAs as their companions helping them to organize their daily activities. Having initially just focused on traditional PIM applications (such as telephone and date book), the last generation of devices equipped with GPS, enhanced connectivity (Wireless Lan and UMTS) as well as powerful processors is well prepared to support a variety of location based services. Mobile services for tourists are one of the promising domains to profit from this new development. Especially tourists in a foreign country who are often unfamiliar with the local language and culture could extremely benefit from well designed mobile services and interfaces. Tourists need help in a variety of situations, e.g. when ordering food, using public transportation or booking museum tickets and hotel rooms. However, the complex task of designing adequate mobile user interfaces for these services still remains a challenging problem. The mobile interface has to reflect the individual user’s interests and background as well as the specific usage situation in order to support smooth interaction. Often, relevant content is only available in the local language and therefore not accessable by the majority of tourists. Finally, mainly due to their small screen estate and reduced interaction capabilities (i.e. the lack of a keyboard), the interaction with mobile devices is more cumbersome than with regular scale computer systems. Users experience difficulties, for instance, when posing a query or accessing a service if this includes browsing through menus with a deep hierarchy. In the past, different lines of research have addressed these two problems. On the one hand, multimodal interfaces have been investigated as one possible solution to improving the interaction with mobile devices (see [10]). By using speech M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 3–12, 2005. c Springer-Verlag Berlin Heidelberg 2005
4
I. Aslan et al.
and gesture, users are able to compensate for smaller screens and a missing keyboard or mouse. On the other hand, research on multilingual and crosslingual information access has advanced the possibilities of users to exploit information sources in languages other than their own (see [7], [8] and [11]). In this paper we would like to bring together both lines of research by introducing translation techniques, multilinguality and cross-linguality into the design of mobile multimodal interfaces. We will show that such an interface will not only combine the benefits of both approaches but will also provide a new interaction style which allows to combine modalities in different languages. This will allow tourists to communicate much more effectively with automated services and local people by using the mobile device as a mutual communication platform in their own and in the foreign language. We will demonstrate our concepts in the context of the Compass2008 (COMprehensive Public informAtion Services) project for the Olympic Games 2008 in Beijing1 . Compass2008 is a Sino-German cooperation aiming at integrating advanced technologies to create a high-tech information system that helps visitors to access information services during the 2008 Olympic Games in English and Chinese and a few other languages. The next section will provide an overview of the services and the scenarios we had in mind while designing Compass2008. Section 3 elaborates on the interaction concepts and translation services and section 4 discusses the overall architecture of the system. The paper closes with a brief review of related work and draws a few conclusions.
2
Multimodal, Multilingual Service Access and Presentation
COMPASS2008 is built on top of the monolingual service platform FLAME2008 [4] that provides users services according to their personalized demands and allow service providers to register their services. However, the service access option in FLAME2008 is restricted to category-based navigation and no intelligent user interface technologies are applied. The current focus of COMPASS2008 is Olympic Games related information services. A service taxonomy is defined to describe the COMPASS2008 service scope, having taken some existing ontology and taxonomies into account, namely, the service ontology designed in FLAME2008, the tourism ontology resulted from the multilingual tourism information system MIETTA (see [11]) and the Olympic Game information classification available on the Athens 2004 Olympic Game web site. The service ontology serves for service classification and service content structuring and is very important for the service adaptive search and result presentation design. 1
Funded by German Federal Ministry of Education and Research, with grant no. 01IMD02A and selected for cofunding by the Chinese Ministry for Science and Technology. The COMPASS2008 project partners: The Institute of Computing Technology, T-Systems International GmbH, the Frauenhofer Institute for Software and Systems Engineering and CAPINFO Limited Company.
COMPASS2008: Multimodal, Multilingual and Crosslingual Interaction
5
COMPASS2008 services are classified into three groups: (a) Information service: weather info, eating and drinking, city info, olympic info, etc. (b) Transaction service: translation services and e-commerce service (c) Composed service: services that integrate various services to deal with a complex situation (e.g. taxi dining service that makes use of Taxi Dialog Assistance and Smart Dining service) We follow a two-level strategy for the service access and presentation: first level is the general service retrieval and second level is the service specific information retrieval. The general service retrieval options contain ontology-based service retrieval and keywords search. Users can click, handwrite or speak out the service category names, (e.g. city info and/or enter keywords such as forbidden city). Services belonging to the chosen category and containing information about forbidden city will be returned to the users in their preferred modalities. Figure 1a depicts the navigation in the service taxonomy, stepping from the general category eating and drinking to its subcategory smart dining. The second level strategy allows users to enter more precise and specific queries when they have selected a specific service category. A specific query template is designed for each service category. For example, users can ask for temperature and air pressure of a specific region, when they are interested in weather information. In the following sections, we will give a detailed description of the taxi dialog assistance and dining services. Both services provide multimodal, multilingual and crosslingual interaction concepts and help foreign tourists and Olympic Game participants to overcome the language barriers in the everyday life in Beijing. 2.1
Taxi Dialog Assistance
One of the biggest problems for foreigners in Beijing is how to make the taxi driver understand their destination requests, because most taxi drivers in Beijing can only speak and understand Chinese. Our Taxi Dialog Assistance acts as a mediator between taxi drivers and foreign visitors. It translates the destination request into Chinese and speaks to the taxi driver and asks him to enter the estimated price and distance information on the mobile device. If the location-based service is available, COMPASS2008 system will also provide interactive map and route information which eases communication and understanding between the taxi driver and the clients. If a COMPASS2008 user chooses the service Taxi Dialog Assistance, a specified service query page will appear for entering the destination name. The current location of the user will be treated as the default starting point, appearing on the screen parallel to the destination calculated by the location-based service. The user is asked to specify the category of the destination, because it can be a restaurant name, a building name, an organization or institution name. Same name belonging to different categories can have different addresses. Experiences tell us that the category information is an important resource for destination disambiguation. Furthermore, we allow users to enter the addresses of the destination as additional information. Given the destination and its category, the
6
I. Aslan et al.
translation engine will look up in the bilingual dictionary. In comparison to the traditional bilingual dictionary, our dictionary contains not only the translations between terms in different languages, but also tourism relevant categories, to which they belong, such as hotel, restaurant, hospital, shop, school, company, etc. These names are related to the address and location information available in the location-based services.
Keyword search
Next category
Subcategory Prev. category Upper category
Fig. 1. Navigation in the service taxonomy
The smart dining translation assistance is a service that helps foreigners in Beijing to find the right restaurant or food, according to their taste and preferences by providing a vivid and attractive user interface, using multimedia data. We have designed a very fine-grained multilingual database for food and restaurant information, containing e.g., Chinese food names and their audio records, name translations, food/restaurant images, taste descriptions, restaurant addresses, and even related video information. As described in Figure 1, a COMPASS2008 user can perform a general keyword search, for example, by entering chicken and then chooses from the matched restaurants. Furthermore, we also allow multimodal interactions in the queries, for example, (1) speech: ”show me only vegetarian food” (2) speech: ”show me the ingredients of this dish” + gesture: tap on image of dish from a list of dishes (3) speech: ”compare this dish” + gesture + speech:”to this dish” + gesture (4) speech: ”translate this Chinese writing” + handwriting (or gesture) For ordering the food by the restaurant staff, the mobile device can speak out the food name in Chinese and the preferences of the user such as ”vegetarian” or ”no garlic” in Chinese. These are example services, which showcase the synergetic usage of inteligent multilingual, crosslingual and multimodal interaction. Multilingual writing, multilingual voice in/output and gestures in combination with multimedia data presentation gives us the flexible means for the convenient and service adatpive multilingual and crosslingual information access and presentation via mobile devices.
COMPASS2008: Multimodal, Multilingual and Crosslingual Interaction
3
7
Interaction Concepts
3.1
Mobile Multimodal Interaction
The vast variety of interaction modalities available on mobile devices result from the fact, that mobile and handheld devices are used in everyday life, in different situations and context. Therefore, it seems to be difficult to provide a static interaction modality that is always suitable. For example the usage of speech input within a crowded stadium seems to be difficult, because of the background noise. The usage of a stylus to tap or write on the handheld device will only be possible if the user has both hands free. The interaction modalities we support in COMPASS2008 are based on the results of a user study describe in [10]. This study investigated preferences of users, which had no or little experience with handheld PCs, considering multimodal interaction with a mobile and multimodal interactable shopping assistant in a public environment. The subjects preferred (in addition to unimodal interaction modalities; that is, interaction with only one modality) to interact with speech in combination with gesture performed on the display of the mobile device (e.g. with the stylus). In COMPASS2008 we also support combinations of speech2 and gesture combined with handwriting. We believe that writing is a modality that is important in a tourist and multilingual environment. Multimodal interaction is realized in COMPASS2008 with the use of data container pairs. In this work we refer to data container as a data containing file in XML format. A data container pair consists of one XML file that encodes information of all single target objects and another XML file that encodes multimodal grammar definitions that can be used to access information about the target objects. 3.2
Translation Services
COMPASS2008 is the first service platform that combines traditional general machine translation services with some specified translation services, in order to cover different demands of the foreign visitors and tourists in Beijing. We call our translation services ”COMPASS2008 Translation Center”. Figure 2 depicts the four major translation services in COMPASS2008. As highlighted in the above figure, different service requests need different translation services. We provide two kinds of services in the general machine translation functionality, namely, hybrid machine translation service and the name translation service. In the hybrid machine translation service, we integrate on the one hand available online machine translations, such as Google or Babelfish, and on the other hand, offer a more restrictive but also more reliable translation service through an extended digital tourism phrase book. We expect 2
We currently use IBM ViaVoice for speech input and Scansoft TTS for speech output, Chinese speech recognition and synthesis modules developed in the Chinese partner project MISS2 are also under consideration.
8
I. Aslan et al.
Fig. 2. COMPASS Translation Center
that current progress in statistical MT especially for the language pair ChineseEnglish as well as enhanced selection and voting methods will strongly improve the reliability of the full text translation service. For the time being, reliability is mainly achieved through the digital phrase book. Its purpose is to provide foreigners with essential key words, phrases and sentences they need in the most urgent situations and for communication in hotels, airports, train stations, restaurants, Olympic stadiums and other travel sites. The hybrid machine translation service will use the results of the Digital COMPASS Tourism Phrase Book if a translation can be found there. Otherwise it will call the online free machine translation services. The name translation service is especially designed for the mobile device. It helps foreigners to recognize Chinese characters and translate them into the preferred languages. In addition to the regular Chinese character input methods, foreigners have two options to enter expressions made up of one or several Chinese characters: (a) handwriting: drawing Chinese characters via stylus, (b) photo capturing: capturing the Chinese characters via digital camera The first option is supported by a software for Chinese handwriting on the Pocket PC such as the CE-Star Suite for Pocket PC 2003 2.5. Users can draw the Chinese characters, the system will then suggest the most similar characters and let the users choose the right one. For the second option, the corresponding Chinese OCR software still needs to be selected from a range of available options. The Taxi Dialog Assistance and the Smart Dinning services described above use the Taxi Translation Service and the Smart Dining Translation Assistance for the translation task. Both specialized translation services reach a higher accuracy by employing fine-grained dictionaries modelled for the specific domains. 3.3
Multilingual and Crosslingual Interaction
In addition to the number of modalities, the COMPASS system has to deal with multiple languages. Some modalities are connected to a language (e.g. speech and handwriting). Others are language independent (e.g. gesture) but can nevertheless be used to facilitate or complement language communication. The COMPASS platform allows the combination of multimodal techniques and communication on several languages. The system is multilingual in the sense that three languages are currently supported with the options of adding more.
COMPASS2008: Multimodal, Multilingual and Crosslingual Interaction
9
Since the completed information system for the Olympic Games will not support more than five to six languages, we still have to expect a large number of users for whom none of these supported languages will be their mother tongue. For these users who have a certain command of e.g. English or German but no knowledge of Chinese, multimodal presentation techniques can greatly facilitate communication. In addition to this multilingual setup, the system also offers crosslingual functionalities that help to overcome language barriers. These functionalities are important for the communication between people who do not speak a mutual language. They can also help tourists to find their way in an environment where signs, menues, instructions are expressed in a language they do not master. However, some of the target users of the COMPASS2008 system may be bilingual to a certain degree (e.g. will be able to speak English and some Chinese). These users are also allowed to input their queries and commands in mixedlanguage modalities; for example, a bilingual user may ask a question in English and relate it to a Chinese writing ( e.g. they will ask ”How do I pronounce this” and write down in Chinese characters the name of a location or object). To achieve cross-lingual modalities, the data container pairs that are used for multimodal interaction need to be changed. There are two approaches to do this, either a second data container pair that describes the data in the second language can be used or the existing data container pair can be extended.
4
Compass2008 Architecture
The COMPASS2008 service platform has two main parts: the frontend system and the backend system. The backend system is mainly responsible for preparation of data resources for service retrieval, multilingual services, multimodal interaction and location-based services. In this paper, we will focus on the frontend system architecture. The frontend system allows users to register their profiles and retrieval and access the useful services. In this context, we will only describe the central system architecture and concentrate on the multilingual and multimodal service retrieval and access. We distinguish the server architecture from the client architecture. The server architecture includes computing and storage intensive processes. The relevant processes for multimodal interaction are mainly embedded in the client architecture. The right hand side of Fig. 3 depicts the main components in the COMPASS2008 server. The COMPASS2008 frontend system contains two managers: the task manager and the database manager. The task manager is responsible for the workflow and the interaction between the sub-components, while the database manager takes care all database access activities. The main subcomponents are (i) Query Processing Server : Responsible for processing the user queries sent by the clients, including multi-modal interaction and query translation tasks. A query can be speech commands, key-words, questions, service categories, locations and filled forms etc. (ii) Search Server : Responsible for index and database search. It includes both general service retrieval and the
10
I. Aslan et al.
Fig. 3. Overview of client and server archtiecture
service specific retrieval. (iii) Service Manager Server : Responsible for servicespecific applications (iv) Presentation Server : Responsible for the generation of the presentation pages displayed in the browsers. It includes components for multilingual and multimodal presentation. (v) User Profiling and Notification Server : Responsible for setting and updating of user pro-files and notifying the user when new information is available or updated. In Fig. 3, on the left hand side an overview of the client architecture is given. A communication module uses the HTTP protocol to make requests to the services server. The services server presents information (e.g. data container pairs) in XML format. The XML files are parsed, the grammar is loaded and the user interface manager is informed. The main subcomponents are: (i) Modality Fusion: Responsible for handling input modalities. (ii) UI-Manager : Controls connection between Flash based ( Flash is a vector animation software for the web, because they are so lightweight) UI set and C++ based control logic. (iii) Communication: Pipes requests to the server component. (iv) Engine: Coordinates threads. (v) Data (and Grammar) Manager : Manages data container pairs. (vi) Flash based UI set : Pipes user input to UI-Manager and is responsible for data presentation. The user interface manager is connected via socket technology with a topological ordered set of Flash user interface templates. A main Flash template serves as the entry point for the flash user interface. Depending on the commands the main Flash template receives from the user interface manager, the main template pipes information to child templates and initiates the presentation of data and reports interaction with the user interface.
5
Related Work
In this section, we compare multimodal systems that have been developed in the past to assist users in tourist domains. One of the most prominent mobile spatial information systems is the GUIDE system [3], which provides tourists
COMPASS2008: Multimodal, Multilingual and Crosslingual Interaction
11
with information on places of interest in the city of Lancaster. The GUIDE system allows simple point gestures only and was not explicitly designed to explore multi-modal research issues. The HIPS [6] project aimed at designing a personalised electronic museum guide to provide information on objects in an exhibit. The presentations were tailored to the specific interests of a user with the help of a user model and the user’s location within the rooms of a museum. The implementation platform was a notebook and touchscreen which allowed for simple gestures and speech commands, but both modalities were not fused and processed in parallel. The REAL system [2] is a navigation system that provides resource adapted information on the environment. The user can explicitly perform external gestures by pointing to landmarks in the physical world to obtain more information. REAL does not allow for speech interaction. In contrast, DEEP Map [5], another electronic tourist guide for the city of Heidelberg, combines both speech and gestures (mainly pointing) to allow users to interact more freely with the presented mapbased presentations. SmartKom [9] is one of the first systems that follow the paradigm of symmetric modality (see Section 3.3). Input to SmartKom can be provided through the combination of speech and gestures. SmartKom then provides travel assistance for the city of Heidelberg through synthesised speech and through gestures performed by a life-like character. None of the described systems allows for a multilingual and crosslingual interaction comparable to the Compass2008 system. Most related work has been designed for one primary language, the support for multiple languages in the context of multimodal systems is to our knowledge a new concept. Description of additional mobile navigation systems (without multimodal interaction capabilities) for tourist applications can be found in the survey [1].
6
Conclusions
We have tried to demonstrate that a systematic and conceptually thoughtthrough combination of multimodal input and output techniques, multilingual and crosslingual communication and location-sensitive functionalities can greatly enhance tourist assistance systems. Such a combination can yield much more than an aggregation of functionalities. It enables relevant new functionality and poses a number of exciting research challenges. Especially the combination of language technology with other modalities and location sensitivity offers many novel opportunities such as: (i) Multimodal output can help the user to understand information presented in one of the supported languages even if this language is not the user’s mother tongue. (ii) Multimodal presentation can help the user to interpret untranslatable expression such as certain food names and names of places. (iii)To know the location and situational context can help the translation service in disambiguation and selecting the most appropriate output. (iv) Multimodal input can help the user to enter unfamiliar script and facilitate interpretation. (v) Because of the modular architecture in which services are specified and parametrized by means of a complex ontology, new services and combinations can be added. In our ongoing and planned research and devel-
12
I. Aslan et al.
opment the COMPASS2008 platform serves three purposes: (i) It is a research tool for investigating currently existing and any new forms of interaction among multimodal input, multimodal presentation, multilingual setup, crosslingual capabilities and location-sensitive functionalities. (ii) It is a tool for developing, testing and demonstrating functionalities to be offered for the information services of the 2008 Olympic Games. (iii) It is an extendable and adaptable base for developing navigation, information and assistance services for general tourism, cultural exploration and large international events.
Acknowledgements We are grateful to the useful discussions with our project partners, in particular, Weiquan Liu from Capinfo, Bernhard Holtkamp and his colleagues from ISST.
References 1. J. Baus, K. Cheverst, and C. Kray. In Map-based mobile services - Theories, Methods and Implementations. Springer. 2. J. Baus, A. Krger, and W. Wahlster. A resource-adaptive mobile navigation system, 2002. 3. K. Cheverst, N. Davies, K. Mitchell, A. Friday, and C. Efstratiou. Developing a context-aware electronic tourist guide: some issues and experiences. In CHI, pages 17–24, 2000. 4. Gartmann, R.; Han, Y.; Holtkamp, B. FLAME 2008 - Personalized Web Services for the Olympic Games 2008 in Beijing. In Cunningham, P.: Building the Knowledge Economy. Issues, Applications, Case Studies. Vol.1, Amsterdam, Nederlands, 2003. 5. C. Kray. Situated interaction on spatial. In: DISKI 274, Akademische Verlagsgesellschaft Aka GmbH, 2003. 6. R. Oppermann and M. Specht. A context-sensitive nomadic exhibition guide. In HUC ’00: Proceedings of the 2nd international symposium on Handheld and Ubiquitous Computing, pages 127–142, London, UK, 2000. Springer-Verlag. 7. H. Uszkoreit. Cross-lingual information retrieval: From naive concepts to realistic applications. In Proc. of the14th Twente Workshop on Language Technology, 1998. 8. H. Uszkoreit and F. Xu. Modern multilingual and crosslingual information access technologies. In Proc. of Multilingual Information Service System for the Beijing 2008 Olympics Forum, CHITEC, Beijing, China, 2004. 9. W. Wahlster, N. Reithinger, and A. Blocher. SmartKom: Multimodal Communication with a Life-Like Character. In Proc. of Eurospeech 2001, pp.1547-1550., 2001. 10. R. Wasinger, A. Kr¨ uger, and O. Jacobs. Integrating intra and extra gestures into a mobile and multimodal shopping assistant. In Pervasive, pages 297–314, 2005. 11. F. Xu. Multilingual WWW — Modern Multilingual and Crosslingual Information Access Techologies, chapter 9 in Knowledge-Based Information Retrieval and Filtering from the Web, pages 165–184. Kluwer Academic Publishers, 2003.
Discovering the European Heritage Through the ChiKho Educational Web Game Francesco Bellotti, Edmondo Ferretti, and Alessandro De Gloria Department of Electronic and Biophysical Engineering, University of Genoa {franz, ed, adg}@elios.unige.it http://www.eliosmultimedia.com
Abstract. The rapid success of the Internet has spurred a continuous development of web-based technologies. Several applications also feature multimedia environments which are very appealing and can effectively convey knowledge, also in a life-long learning perspective. This paper presents aims and developments of the ChiKho EU project. ChiKho has designed a web-distributed educational game which allows players to share and improve their knowledge about the heritage of European cities and countries. The paper also describes the ChiKho’s logical structure, the interaction modalities and the technical architecture. We finally presents preliminary usability results from tests with final users which we performed in the four ChiKho exhibition sites (London, Genoa, Plovdiv and Kedainiai), where we launched a prototype version of the web game.
1 Introduction Information and Communication Technologies (ICT) have shown a great potential in providing new educational tools and services, as they are useful to loosen the time and space constraints that have traditionally limited the knowledge-acquisition processes. A recent trend has extended the multimedia education approach by inserting the educational contents in a game framework, which is very appealing, in particular to youngsters, and has an intrinsic cognitive value [1] [2]. Samples of educational games include Ultima Online, an on-line multiplayer game played in a fantasy setting, and Real Lives 2004, an interactive life simulation game. These games, however, intend to stress entertainment aspect rather than the educational aspect. The ChiKho educational web-game (an early prototype is available online on the ChiKho website [3]) stems from the need for personally involving high-school students in the understanding of the European cultures, through the discovery of the common roots and the local differences. The game consists in a virtual trip through several European cities (four, in this experimental phase: Genoa in Italy, East London in England, Plovdiv in Bulgaria and Kedainiai in Lithuania). In any city, players have the opportunity of investigating local artistic and cultural heritage through a set of trials that reward the players’ enterprise and sense of sight. The aim of the game is to arouse participants’ curiosity and interest, so that they learn in a personal and engagM. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 13 – 22, 2005. © Springer-Verlag Berlin Heidelberg 2005
14
F. Bellotti, E. Ferretti, and A. De Gloria
ing way. The game intends to provide positive impressions and cues that should stimulate the player’s interest toward the presented places. The ChiKho web-game has been designed and implemented by the ELIOS Lab of the University of Genoa in the context of the ChiKho project (Cultural Heritage Interactive), co-funded by the European Union under the Culture 2000 programme. The main contribution of this paper consists in the proposal of a web-based model for a virtual tour of art, history, culture and environment in a number of European cities. In particular, we discuss its fundamental elements and its impact on the users through the analysis of early usability tests.
2 ChiKho Targets and Principles The main target of ChiKho is to offer a pleasant and useful virtual tourist experience in several European cities. The experience relies on active virtual exploration of the territory (streets, squares, churches, palaces, ports, etc.), its people and their activities, in order to have a better understanding and appreciation of the city’s cultural heritage. The experience is delivered through a game frame, allowing to capture and appeal a wide audience and to exploit specific educational and cognitive aspects of computer games. The main target audience are high-school students aged 14 to 18. Nevertheless, in a life-long perspective, the game has been designed for the general public, trying to reduce at a minimum usability barriers. The game can be played by individuals at home and groups of school mates in classrooms (namely, the ChiKho Solo game modality). There is also the possibility of organizing synchronous international games (the ChiKho match modality, not yet fully implemented as of writing). A third game modality is the ChiKho International Event, where every team is composed of players coming from all the participating countries, in order to spur and favor collaboration in an international context. A first International Event test has been played in Genova, East London, Kedainiai and Plovdiv on June 21st. ChiKho’s pedagogical principles mostly rely on the constructivistic learning philosophies [4], which stress the importance of constructing knowledge by situating cognitive experiences in real-world, authentic activities [1]. Moreover, especially according to our previous experience [5], we have complemented this constructivistic approach by introducing objectivistic learning features (e.g. providing text introductions and conclusions to the single games), in order to consolidate the implicit experience with clear and explicit verbal knowledge.
3 The Game Architecture Relying on the presented principles we have designed a game structure similar to a hurdle-path game. Players go through a sequence of stages, one stage per each city. At each stage, the player faces a number (5 in the present implementation), of trials. The trial games are instances of 12 typologies that intend to stimulate different skills. Snapshots of trial games samples are reported in Fig. 1.
Discovering the European Heritage Through the ChiKho Educational Web Game
15
We broadly divide games in three categories, according to the cognitive skills they mostly involve: observation games, reflection games, and action videogames. A detailed description of the categories is provided in [5]. In fact, the categories are those of VeGame, a mobile territorial game we developed for enhancing a cultural tour through a city of art as Venice. In the case of ChiKho, the objective is similar (we intend to promote a credible experience, not a virtual trip disjointed from reality) even if players cannot physically experience a city of art while playing. In adapting the game typologies from a territorial game to a web-game we paid particular attention to promote an analysis of the documents based mainly on reasoning (rather than on the sense of sight to explore the surrounding of a physical space), analysis of symmetries, of documents available in web repositories and of experience gained in previous games.
Fig. 1. Snapshot from sample game trials: MissingDetails, CatchIt, TextQuestion and Memory
ChiKho does not involve temporal constraints, since its purpose is to support discovery and appreciation of the cultural heritage, not to promote a rush through the cities. Thus, we do not consider time as the main factor for rewarding players. The winner is not the team which completes the path first, but the one which completes all the games best, gaining the highest score. The percentage of the score given by trialcompletion times is limited to up to 30%. Trial games mostly reward accuracy and effectiveness. For instance, the result of the Puzzle game depends on the number of moved elements, and the Memory game rewards players able to remove all cards in
16
F. Bellotti, E. Ferretti, and A. De Gloria
the minimum number of steps. Through this simple policy we intend to promote a (virtual) tour style which induces accurate analysis of the heritage in a pleasant and engaging fashion. Action videogames are an exception, since in this case we reward efficiency - the ability to complete tasks within a limited time -, which is a fundamental factor for the information processing skills stimulated by videogames. An important aspect of ChiKho concerns communication among teams. We considered that frequent interaction between teams, may distract players and prevent them from an attentive performance of the trials (e.g. answering to an urgent question/need from team B may disturb team A while engaged in solving a puzzle). Instead, we want inter-team communication to be functional to the main target of the game, which is an entertaining experience of the European heritage. Thus, we moved most of the inter-team communication aspects outside the core of the game. We achieved this by implementing a chat and messaging system (not yet fully implemented as of writing). This system does not require real-time interaction: a message can be read after a while without loosing precious time/information nor making other teams wait. This kind of non-invasive inter-team communication is not strictly necessary to complete the game, but is important: teams can freely help each other, exchange opinions to improve their field experience, comment on their activity, exchange personal information, and use these communication channels with extreme freedom.
4 The ChiKho User Interface Specifications The ChiKho overall design aims at supporting constructive edutainment (education + entertainment). This involved design of suited interaction modalities (i.e. tools and user interface elements) whose specifications are described in this section. Not all of the mentioned functionalities have yet implemented, as writing.
Fig. 2. The sketch of the ChiKho main page
Discovering the European Heritage Through the ChiKho Educational Web Game
17
The game’s main page is inspired by the model of the news interactive TV channels. We use this metaphor to convey the message that cultural heritage is an important information item that can be of great interest and appeal to the public. The sketch of the layout is reported in the figure 2. The main page contains a main panel which can host several different activities: o o o
o
Trial-games. Where the users play the actual trial-games. Game management. Score and statistics about the games. See Figure 3. Message management. In this area a player can send messages to specific groups (e.g. the remote components of the same team) or post generic messages for everybody. Players can also read incoming messages. Work tool panel: use of some “magic” tools that are useful. Sample tools include: a simple translator, a local-site search engine, a global search engine, a search wizard (in an indexed local-database). These objects are simple samples of “constructivistic tools” that support personal knowledge acquisition. They are gained as prizes at a successful completion of some games.
Fig. 3. The sketch of the ChiKho game management page
The main panel is framed by peripheral areas (right and bottom, which provide the players with tools and peripheral information). At the right of the main panel there are: o o
o
o
The game management button, to access the relevant area. The Bonus icon (button), where bonus icons may suddenly appear. By clicking on the bonus icon, a new bonus game can be played in the main panel, in order for the player to increase his/her score. The Message icon (button). The Message icon changes color in case of unread incoming messages. By clicking on the button the user can access the message management area in the main panel. The tools area where tool icons may appear as a reward for a player’s success. Icons can be clicked by the user to open the work panel of the selected tool.
18
F. Bellotti, E. Ferretti, and A. De Gloria
o
A useful links panel. Links in the panel are incrementally added, in order to reward successful players (this is to be specified in the panel). The links lead to local or remote pages that provide useful information to solve some quizzes or to get more in-depth information about items of interest. A special link leads to the “introductions and conclusions” page, where introductions and conclusions of the trial-games already played by the team are collected.
At the bottom, sliding text contains highlights about art, history, architecture, etc. in the involved sites. Players can click on the highlight to follow a link to a web page where they may get more in-depth information to solve quizzes or for personal information.
5 Technical Architecture The prototype ChiKho implementation currently online involves 4 cities with 5 trial games per city. At each game session, trials are chosen randomly from a set of available trials. Trials contents have been developed by students of the participating countries, according to our specifications and under the supervision of local educational experts. The set of trials is ever growing (in terms of typologies and of instances). At present ChiKho features 32 instances, for a total of about 13 MB of contents. All contents are indexed and stored in a database in order to support separation of contents from interaction modalities, flexibility, extensibility, and to favor search of contents by players to have more in-depth information to solve quizzes. The ChiKho computing and communication architecture relies on a central server where all the game information and contents are collected and managed. Contents are stored in a SQL server database. All the server functionalities are available as web services, which rely on standard Internet technologies (XML, TCP/IP) and are independent of the platform, the development language and of the transport (e.g. HTTP, SMTP, etc.). We have implemented an XML-based protocol for data exchange. On the client’s side, requests to Web services are encoded in SOAP strings. Since current Macromedia Flash technology, that we have chosen to implement the game animations and the user interfaces on the client, does not support the .net Dataset, the server’s results are encoded and serialized as specific objects which are ad-hoc interpreted by the clients. The server side logic, which essentially translates clients’ requests into database stored procedures, has been implemented in C# in the .net environment. ChiKho involves a huge amount of multimedia contents that are delivered before and during the games. We have tested several network configurations, obtaining good results but with a 56K modem connection, where loading times are high (about 20 seconds), the introductory animation is frequently interrupted and playing the trialgames is sometimes not smooth.
6 Tests We conducted field tests with experts and users from the early phases of the project forward, performing evaluations to inform future design.
Discovering the European Heritage Through the ChiKho Educational Web Game
19
A prototype version of the ChiKho web game is online since June 16th. The launch of ChiKho has been done in four contemporaneous exhibitions that have been held at cultural institutions and palaces in Kedainiai, London, Genoa and Plovdiv from June 16th to 21st. On the last day, a special ChiKho International Event Match has been played with mixed teams in order to support also international collaboration. At all the exhibitions we conducted test sessions in order to evaluate the impact of ChiKho on users and check its validity as an edutainment tool. The sessions involved a total of 48 teams each constituted by two players. Players were students aged 11-18. In its present form, the paper only presents results from the Plovdiv and Genoa sites, for a total of 54 participants (68% females, 32% males). Students came from three major types of schools (scientific, humanistic and linguistic schools), representing a wide range of backgrounds. Students had been invited through school teachers and local newspaper articles. Once at the venue, students were presented with the ChiKho computers and started playing without any introduction, in order to simulate the behaviour of normal Internet users (anyway, an introduction to ChiKho is available online at the beginning of the game). Supervisors from local ChiKho partners were at the venue and observed the players in order to get feedback from their behaviour. Students were free to perform the game without any constraints on time, number of trials, score. By no means was the game experience related with any school activity (i.e. no marks were foreseen for participants). When a game ended, we asked participants to complete a questionnaire and also interviewed users informally about their experience. The average length of a trial game (including introductions and conclusions) was 71 seconds. The total game length ranged from 20 to 50 minutes (average 28). The students completed 100% of the trials, which shows a high level of participations. A couple of teachers accompanied some students and were surprised that a game based on cultural items on such remote places could gain the interest of their students. Differently from usual videogames, where players are alone in front of the screen, ChiKho users typically play in couple, one player holds the mouse and the other one provides suggestions. ChiKho encourages interaction between team-mates because it favors content verbalization, such as defining a picture’s properties and details. It also encourages expression and discussion of proposals, knowledge sharing, critical reasoning, and hypothesis evaluation. Moreover, as an entertaining game, humoristic comment were also exchanged during the matches. In general, we observed that just a minority of the players followed the suggested links to get more information (the highlights shown in Fig. 2, which may be useful to solve difficult quizzes, or just for personal information). This has two major explanations: on one hand players were dragged by games and competition, on the other hand the highlights, that link to the web pages that provide more information, were roughly visible. A couple of teams, who finally succeeded to obtain the highest scores, also used the Google search engine in order to exploit all the potential of the Internet. This is a clear example of how online educational gaming can benefit from the overall Internet infrastructure. Another example that may be suggested in the next ChiKho versions is the use of online translators in order to access pages in local languages and/or enhance interaction among foreign players. While the game management interface (Fig. 3) was designed to support a completely free exploration of the game space, we observed that almost all players played
20
F. Bellotti, E. Ferretti, and A. De Gloria
the trials sequentially from the first city to the last one and, within each city, from the first trial to the last one, in the order proposed in the user interface. This probably shows that players considered the game as a sort of driven educational guided tour. Analyzing the final scores of the teams we observed that the highest scores were realized by lyceum students. This is not because they already knew the addressed items. But because they could exploit already developed cognitive structures and make suitable links (e.g. concerning historical periods) and because they were particularly interested in developing cultural themes. We used a questionnaire to perform a qualitative analysis of user acceptance and satisfaction and an assessment of ChiKho as an education tool. The questionnaire, which generated 54 responses, contains 4 questions for each city, some free comments about drawbacks and praises of ChiKho and 8 6-point scales. We present distinct results for the two nations, since we noticed some significant differences that may be explained by the different attitudes, cultures and economical levels of Bulgaria and Italy. The survey respondents offered favorable assessment of the following key characteristics: o o o
Overall experience – ChiKho was rated 4.18 in a 0-5 scale by Italian players and 4.78 by Bulgarian. Usefulness – ChiKho was rated 4.71 by Italian and 4.72 by Bulgarian. Appeal – ChiKho was defined as amusing/interesting by all respondents, independent of the country.
Fig. 4 shows that quality of multimedia contents was favorably assessed by players, a part from the quality of audios and music. This was because there was only one song for all the games in every city. The new ChiKho versions are increasing the number of songs. Simple music effects (as jingles for successful or unsuccessful events) are also being introduced. Moreover, we are using the concept of “contextualized sounds”, where the sound-track of a trial game faithfully reproduces the “soundscape” of the featured item/area. For example, in the Saint Paul cathedral puzzle, we play the original recorded sound of the bells. 5,00 4,50 4,00 3,50 3,00 Genoa
2,50
Plovdiv
2,00 1,50 1,00 0,50 0,00 Graphics
Texts
Images
Music
Audio
Movies
Fig. 4. Assessment on a 0-5 scale of the multimedia elements ChiKho. Responses are divided by nationality (Italian players in Genoa and Bulgarian in Plovdiv).
According to the players’ free comments, the ChiKho’s most useful aspects consist in the opportunity to discover people, regions and cultures using new, challenging and
Discovering the European Heritage Through the ChiKho Educational Web Game
21
entertaining methods for collecting information and acquiring knowledge. Players regretted that the exploration was limited to just 4 cities. A number of Bulgarian players suggested that ChiKho may be available on a CD Rom, without the need for an Internet connection. Negative comments mainly concerned the quality of the music and the fact that some trial games’ typologies (e.g. Memory Game, VisualQuiz, etc.) were quite repetitive, as they were played in all the different cities (with different contents, of course). This was due the fact that in the prototype version only 8 typologies were available, while we have now added other 3 and we are further developing other ones. A few players complained that introduction and conclusions texts were useful but quite annoying to read. They suggested they should be listened through an actor’s voice. The last part of the ChiKho questionnaire aimed at a simple, quantitative evaluation of the ChiKho’s ability to support knowledge acquisition by asking free-answer questions about the most prominent environmental, historical, economical and cultural aspects of the “visited” cities. For Bulgarian players we obtained a percentage of 73%, 77%, 79% and 85% of good answers. Most of the Italian players (57%) did not respond to these questions. Good answers accounted for 40%, 32%, 38% and 2% of the total. Several Italian players judged the questionnaire to be too long to be completely filled. Anyway, the low percentage of good responses is a sign that the knowledge-acquisition aspect is not automatic at all and should be better enforced.
7 Conclusions The first version of the ChiKho web game is online since June 16th. The launch of ChiKho has been done in four contemporaneous exhibitions. Early results have shown a great interest by players (some of them have also continued playing at their home) and by their teachers as well. Of course, the game is not intended as a substitute for traditional teaching modalities, but exploits habits and technologies which are beloved by youngsters in order to convey high-quality educational contents. We noticed that several participants were fond of interacting with their local heritage in compelling and engaging ways. Moreover, ChiKho gave them the opportunity to discover new countries and cultures with some common roots (for instance, the image of Saint George and the dragoon is a common iconographic element) and some differences (in particular concerning the style of the buildings). The most interested players also used Internet search engines (i.e. Google, for text and images – which are very useful, in particular in the games where the player has to identify the missing or wrong details) to improve their performance. This is very important in order to show the attractiveness of the game and the usefulness of common Internet tools for developing cultural knowledge. The ChiKho project is still ongoing and further developments are foreseen in order to improve the overall game and increase the available contents and trial games. The prototype version has already been upgraded according to the feedback of the tests described in this paper. Another significant step will involve the implementation of a simple development tool through which students will be able to compose their own trial-game and store them in the database, so that they may be played online by the ChiKho players.
22
F. Bellotti, E. Ferretti, and A. De Gloria
Acknowledgments We would like to thank the ChiKho project partners: the CIDA Cultural Industries Development Agency of London, the Plovdiv NF API and the Kedainiai regional museum. We would also like to thank Erik Cambria, Andrea Costigliolo, Anna Carrino and Elisabetta Pisaturo for their precious contribution in implementing the game graphic and local contents, and all the other students and people that have contributed to the ChiKho contents in Lithuania, Bulgaria, England and Italy.
References 1. H. Pillay, J. Brownlee, and L. Wilss, Cognition and Recreational Computer Games: Implications for Educational Technology, Journal of research on computing in education, Vol. 32 No. 1, pp. 203-215, 1999. 2. M. J. Natale, The Effect of a Male-Oriented Computer Gaming Culture on Careers in The Computer Industry, Computers and Society, June 2002, pp. 24-31. 3. http://www.chikho.com/game 4. T. M. Duffy and D. H. Jonassen, Constructivism and the Technology of Instruction: A Conversation, Lawrence Erlbaum, Hillsdale, New Jersey, 1992. 5. F. Bellotti, R. Berta, E. Ferretti, A. DeGloria, and M. Margarone, VeGame: Field Exploration of Art and History in Venice, IEEE Computer, special issue on Handheld Computing, September 2003
Squidball: An Experiment in Large-Scale Motion Capture and Game Design Christoph Bregler, Clothilde Castiglia, Jessica DeVincezo, Roger Luke DuBois, Kevin Feeley, Tom Igoe, Jonathan Meyer, Michael Naimark, Alexandru Postelnicu, Michael Rabinovich, Sally Rosenthal, Katie Salen, Jeremi Sudol, and Bo Wright http://Squidball.net
Abstract. This paper describes Squidball, a new large-scale motion capture based game. It was tested on up to 4000 player audiences last summer at SIGGRAPH 2004. It required the construction of the world’s largest motion capture space at the time, and many other challenges in technology, production, game play, and study of group behavior. Our aim was to entertain the SIGGRAPH Electronic Theater audience with a cooperative and energetic game that is played by the entire audience together, controlling real-time graphics and audio by bouncing and batting multiple large helium-filled balloons across the entire theater space. We detail in this paper the lessons learned.
1 Introduction Squidball is a large-scale, real-time interactive game that uses motion capture technology and computer graphics to create a unique and energetic experience for mass audiences. Using the motion capture volume with participating player audiences of up to 4,000 people, the game debuted on August 12th, 2004, at the Los Angeles Convention Center as pre-show entertainment for the SIGGRAPH Electronic Theater. This paper describes the design criteria and technology behind this venture. It also explores the adventures and challenges that the production team had to overcome and the lessons learned. SIGGRAPH audiences experienced a similar interactive Electronic Theater preshow over a decade ago when the Cinematrix System was introduced in 1991 [2]. Cinematrix was an interactive entertainment system that allowed members in the audience
Fig. 1. Electronic Theater audience playing Squidball M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 23–33, 2005. c Springer-Verlag Berlin Heidelberg 2005
24
C. Bregler et al.
to control an onscreen game using red and green reflective paddles. Other interactive entertainment systems have been tested on audiences in the hundreds to thousands, described in greater detail in section 2. The success of Cinematrix was the original inspiration for our work, and we initiated our project to bring back this style of pre-show entertainment to SIGGRAPH. Although the 2004 Electronic Theater was our first public test, we envision Squidball being deployed in other large audience events for entertainment, social studies, team building exercises and other potential applications. Developing and testing such a system was a very unique and high-risk venture with many challenges. Other games, graphics and interactive systems are usually designed for single user or a small group, and go through several test cycles. However, for the Squidball project, we were dealing with many factors orders of magnitude larger than standard environments, including a gathering of 4000 people, the construction of a system using a 240 × 240 × 40 feet motion capture volume and a huge screen. Furthermore, the system had to work the first time, without the benefit of any full-scale testing. During our initial brain-storming sessions, we decided to create a game that is played by bouncing and batting a number of balls that would be used as wireless joystick/mouse inputs to a game, across the entire audience. We also decided to track the balls using 3D motion capture technology and to use this data to drive real-time graphics and an audio engine. We set out to design a game that: – Requires no explanation of the rules – people must be able to pick it up and start playing; – Is even more fun than just hitting a beach ball around an auditorium (we already know this is fun); – Is fundamentally about motion-capture and takes full advantage of the capabilities of this technology; – Can be played by 4,000 people simultaneously using a small number of input devices; – Can be played by people standing, sitting or holding a beer in one hand (there was a cash bar in the Electronic Theater); and – Involves people hitting balls AND looking at a projection screen.
2 Related Work As mentioned above, the inspiration for Squidball came from the Cinematrix system [2] shown at the SIGGRAPH 1991 Electronic Theater and several other events. With this game, every audience member had red and green reflective paddles that control onscreen games, including a voting system, Pong, and a Flight Simulator. In the voting schema, the system counted how many red vs. green paddles were shown. In Pong, the left side of the audience played against the right side, and the position of the paddle controlled the ratio of red and green paddles on each side of the audience. It was surprising how quickly the audience learned to control the games and to jointly coordinate the mix between red and green paddles. Of course, the yelling and excitement of a large
Squidball: An Experiment in Large-Scale Motion Capture and Game Design
25
audience was also part of the show. Another set of similar interactive techniques was studied at student theater screenings at CMU [4]. Several computer-vision-based input techniques were tested on large audiences. The input technique most closely related to Squidball is the 2D beach ball shadow tracking, where the location of the shadow could be used as a cursor in several 2D games. D’CuCKOO (a music band that uses various kinds of new technological instruments) designed a gigantic beach ball that creates music as the audience bats it around. The MIDI-Ball [1], a wireless 5-foot sphere, converts radio signals into MIDI commands that trigger audio samples and real-time 3-D graphics with every blow. There have been other systems reported, that track small groups of people as they perform interactive music and dance activities [6], or are used for home video games [3] but none of them have been tested on thousands of players.
3 Large-Scale Motion Capture Here we describe the challenges and experiments of building a large-scale motion capture space and how this ties into the Squidball game engine and game testing. In section 4 we describe additional details of the game design. Our target venue was Hall K in the Los Angeles Convention Center, which was converted into a 4000-seat presentation environment to screen the Electronic Theater for the 2004 SIGGRAPH conference. The total space was 240 × 240 feet. We needed to build a motion capture volume that covered the entire seating area and allowed enough height above it to throw the balls up in the air: a capture volume of 190 × 180 × 40 feet. To the best of our knowledge, no motion capture space of this size had been built before. One of the larger reported spaces was built for the Nike commercial by Motion Analysis Corp and Digital Domain [5]. It had dimensions of 50 × 50 × 10 feet. It used 50 cameras to track six football players. One of our design constraints was tracking multiple (up to 20) balls in 3D in realtime. We had a Vicon motion capture system [7] with 22 MCAM2 cameras, and each camera had a field of view of 60 degrees (12.5mm lens) and 1280 × 1024 pixel resolution. In its intended use, the system can track standard motion capture markers (0.5 inch) in a capture distance of up to 25 feet. The markers are made of retro-reflective material. Visible light illuminators placed around the camera lens shine light out, and almost all light energy is reflected back into the camera. This makes the retro-reflective markers appear significantly brighter than any other object in the camera view, and image processing (thresholding and circle fitting) is used to track those markers in each view. Triangulation of multiple camera views results in very accurate and robust 3D marker tracking. (We are currently also considering vision-based techniques on nonretro-reflective objects for future game experiments.) We determined that the only way to utilize the Vicon motion capture system in a significantly larger space with the same number of cameras was to scale up each aspect of the system. The cameras’ view scale up in an approximately a linear fashion; in other words, a marker 100 times larger in diameter and 100 times further away looks the same to the camera. Of course, because light intensity falls off with the square of the distance traveled, much greater illumination is necessary. With experimentation, we found that halogen stage lighting provided sufficient illumination for the Vicon tracker.
26
C. Bregler et al.
Three other challenges in scaling up the system were 1) producing the larger markers, 2) dealing with camera placement constraints, and 3) calibrating the space. All of them appear simple in theory but, in practice, these became critical production issues. 3.1 How to Produce the Balls (Markers) The motion capture system requires spherical markers for effective tracking. Normal marker sizes are 0.5 inch in diameter, which are covered with 3M retroreflective tape. In order to stage our event in a radically larger-than-normal space, with larger-thannormal camera view distances (max. 250 ft.) and using the balls as real-time inputs, we had to increase the size of those markers significantly. We determined that a 16-inch diameter marker was the smallest marker that could be robustly detected at 250 feet. In the final game, for dramatic effect and game-play, we opted for larger markers: 8-foot chloroprene bladders (weather balloons). In order to achieve the right “bounciness”, we under-inflated the balloons.
Fig. 2. The evolution of our retro-reflective balls
Each marker requires a retroreflective coating in order to be tracked by the Vicon motion capture system. Experiments using retroreflective spray-paint failed. The reflective intensity of the sprayed paint was 70% to 90% less than 3M retroreflective tape. After dozens of tests with tapes, and fabrics of varying color, reflectivity and weight, we settled on specific 3M retroreflective fabric (model # 8910). Figure 2 shows the results of these tests. The first test (the ”lemon”), the second iteration (the ”tomato”), and the final version (the ”orange”), which ultimately produced a perfectly round spherical shape. In order to achieve a perfect spherical shape and to spread the force evenly throughout the surface, the fabric was cut on the bias, in panels like those of a beach ball. At this large scale, any of these shapes were adequate for the Vicon system to track them. The advantages of a perfect sphere were both aesthetic and functional. With the force evenly distributed, one spot is no more likely to rip the fabric than any other spot. Similarly, hitting the ball anywhere has the same predictable result. The balls were inflated with helium to reduce their weight. Because the fabric was heavy, they did not float away when filled with helium, fortunately. In future generations of this game, we plan further reduce the weight of the balls in using a different material or smaller sized balls. 3.2 Camera Placement Standard camera placement for a motion capture system is an “iterative refinement process” dependent on several site-specific aspects. For standard motion capture, cam-
Squidball: An Experiment in Large-Scale Motion Capture and Game Design
27
eras are usually placed on a rectangle around the ceiling, all facing into the capture space. Sample motion capture markers are distributed over the capture space, and cameras are adjusted so that each marker is seen by as many as possible cameras from as many as possible directions. Additionally, the tracking software is checked for each camera during placement. In our scaled-up system, camera placement was a significant challenge. We could not afford as many trial-and-error cycles in camera placement that would be possible in standard-sized motion capture labs since our time was limited in the final space and each adjustment took a significant amount of time. Other logistical constraints affecting camera adjustment included: A) cooperating with the Union LACC workers schedule to get access to the ceiling and catwalks, B) coordinating between people on the 40-foot high ceiling and people on the floor up through radio-communication for each re-mount and re-alignment of a camera, C) getting live feedback from the Vicon PC station in the control booth to people on the ceiling so they could see the effects of their adjustments, D) camera view limitations - the 60 degree wide-angle lenses did not actually see a full 60 degree angle of view; even with the extra heavy studio lights mounted next to the cameras the visibility of the weather balloons dropped off after 250 feet in the center (and at even shorter distances at the perimeter of the camera view), and E) scale issues - moving balls on the ground takes much longer because of their large size and the distance to be covered. In a standard mo-cap studio, you pick up a marker and lay it down a few seconds later; in this space, we had to move a shopping cart with a ball or drive an electric car across the hall. 3D Simulation in Maya. In anticipation of all those problems, we designed a 3D model in Maya for all the target spaces, including one for the campus theater (our first test), one for the campus sports center (our second, third and fourth test), and one for Hall K at the LACC (Figure 3), our final show. This final model was derived from blueprints we obtained from the building maintenance team. We also built a 3D model of the “visibility area” to determine the sight lines of the cameras. We determined that the cameras could not “see” of center at distances of 250 feet. The further out we moved the balls, as shorter the visibility became. Based on experimentation, we built a 3D Maya model for the camera visibility volume (Figure 3 green). This volume was then used in our Maya building model to simulate several camera placement alternatives. Our goal was that each point in the capture volume should be seen at least by 3 cameras, given the constraints on the lengths of video cables. The final configuration we used was pretty close to our simulation. We settled on mounting evenly all 22 cameras around the left and right catwalk, and along the back-end catwalk and the center catwalk, but not the frontal catwalk. We didn’t want to mount any cameras and high-intenstiy lights above the screen, so the audience would not be distracted. (Consult our website to see the final camera-placement in detail). Mounting and Networking. We knew we had to set up the system in LACC very quickly, so we ran multiple practice sessions for camera mounting in New York, first for 10 cameras and then 22 cameras. The setup required careful cabling. Vicon sends camera data first through analog wires to a Datastation, which thresholds video frames and compresses the resulting
28
C. Bregler et al.
Fig. 3. This shows our Maya model of Hall K and the visibility cone (green) for one of the cameras
binary images. The Datastation then sends all 22 Video Streams at 120Hz over gigabit Ethernet to the Vicon PC, which does the real-time 3D tracking. In order to have the shortest possible video cable length, the Datastation had to be close to the cameras. In Hall K at LACC, the Datastation was placed 40 feet above the audience on one of the catwalks. The compressed video data was then sent via gigabit Ethernet down to the “control booth” in the back of the audience on the floor. The control booth contained all the workstations, including the real-time 3D tracker and the game system. During camera placement we operated the Vicon PC in the control booth through a wireless laptop and Remote Desktop. This allowed us to walk from camera to camera on the catwalk, and do all adjustments, while remotely monitoring what the camera “sees” and how the tracking software performs. 3.3 Calibrating the Space The final challenge for the motion capture setup was camera calibration. Using the Vicon software, the calibration process in a standard space is done by waving a calibration object throughout the entire capture volume. Usually, this is a T-shaped wand that has 2 or 3 retroreflective markers placed on a straight line. (Figure 4) The 2D-tracking data for the calibration object from each camera is then used to compute the exact 3D locations, directions and lens properties of the cameras. This is called the calibration data, which is crucial for accurate 3D tracking. Of course, the standard T-wand would not be seen by any camera in such a large target space (below pixel resolution). We determined that a 16-inch marker was the smallest marker that could reliably be seen and tracked from 250 feet. To overcome this, we built several “calibration T-wand” versions. Figure 4 shows one version that allowed us to “wave” the calibration object as high as 30 feet. We conducted initial tests on how much time an “exhaustive volume coverage” would take and how physically exhausting it would be using the roof of our lab. In the campus theater space and the campus sports center, we either walked the wand around holding it at several heights or skate-boarded through the space. In the final test at Hall K, we first used a crane and ropes. Ultimately, we ended up using a T-wand constructed out of light weight bamboo sticks lashed together using a traditional Japanese method and then drove that
Squidball: An Experiment in Large-Scale Motion Capture and Game Design
29
around at several heights on an electric car. A calibration run took around 30 minutes. Tensions ran high during calibration in Hall K, but the process was a success (see video on squidball.net) and we had a spot-on reading of Hall K right up to the periphery of the seating areas. In actuality, we were able to track the balls beyond the boundaries of the game “board”.
Fig. 4. Left: The standard sized calibration objects of length 15 inch. Right: Large 15 feet high wand.
Fig. 5. Calibration in Hall K using a crane
4 Software Integration of Motion Capture and Game Engine Before we could start the various game tests, we needed a rapid prototyping environment connected with the real-time motion capture input, one that generated real-time computer graphics and sound effects. The real-time visualization system and game engine were written using the Max/MSP/Jitter development environment distributed by Cycling74. The system consisted of five main components: – A TCP socket communication system, which distributed motion capture data from the Vicon system to the tracking, game-engine and audio subsystem in real-time. – A real-time tracking module that would take the raw motion capture data, filter it using Kalman filters and then extract useful metrics such as object velocity and collision detection. (This was necessary to produce consistent sound effects with the balls flying high up in the air). – A game engine written partly in Java and partly as Max patches, which drove the game simulation. Submodules of this system included components that handled the basic game narrative, the media files, and the interface to the graphics engine. The graphics engine used Jitter to render the game using OpenGL commands. – An audio subsystem resident as a Max patch on a second computer (and receiving forwarded motion capture information as well as scene control from the main Max computer). It produced sound for collision, bounces, flying noises, etc.
The prototyping environment proved robust and fast enough to use in the final show, and we continued to tweak the game until the night before the opening.
30
C. Bregler et al.
In the production mode, we ran two duplicate sets of the game and audio computers for redundancy, with a switch, but fortunately this was never needed. The system required five human operators during the shows one on sound, one on the game system, one monitoring the Vicon PC, one watching the audience, and a master show controller who coordinated the team.
5 Large-Scale Game Design We wanted to design a game that worked well in the poorly-understood dynamics of cooperation and competition of a large-scale group, but we knew we had a limited number of markers that we could track. We also understood that however we used the markers, the game had to run well with no full-scale testing. We wanted everyone to “win the game” as a single body, not as many small groups. This meant we had to discourage degenerate strategies. We decided that the rules of the game should be discovered on the fly as people played, rather than through an instruction sheet. Finally, we wanted to create a game that was more rewarding than simply bouncing balls around in a space. Playtesting was a major challenge we faced. Gathering a 4000-player audiences is difficult and expensive, so we had limited opportunities to test the game at full-scale. This led to a number of creative solutions in testing. Though we were able to use spaces of approximately the same size as the final space for testing, we were not able to test with a full-scale audience. We came up with a number of innovations, including clumping groups in various locations in the testing space and organizing the clumps strategically to make them appear to be a larger audience. However, the first test of the game with a full audience was not until its premiere at SIGGRAPH. After many discussions and iterations with the prototyping system, we settled on the game rules described below.
Fig. 6. Example Screen shot of Squidball game. Please see video (on squidball.net)for game in action.
The Game: The rules for the game were simple, and had to be discovered by participants through gameplay. The twelve weather balloons in physical space were represented within the digital game space as green spheres on the screen. Players moved the weather balloons around the auditorium (whose space corresponded to a 3D space onscreen), in order to destroy changing grids populated by 3D target spheres. The game
Squidball: An Experiment in Large-Scale Motion Capture and Game Design
31
had 3 levels of increasing complexity, and each level could be replayed 3 times before a loss condition was reached. The second level introduced an element of time pressure, so players had to complete the game challenge within the allotted period of time. This was the level that really taught players how the game worked, as most audiences failed to clear the level on the first try. Through repetition and the existence of a loss condition, the players eventually discovered the victory condition, as well as the correspondence between the weather balloons and their representation within the virtual game space. In the third level, players worked to clear special colored spheres that, when activated, revealed a composite image. Players quickly discovered a range of social strategies that emerged from their physical proximity with other players; in each instance of the game (the game was played 6 times over the course of 4 days) the 4,000 or so players came together organically to collaborate in the play of the game as they discovered what the gameplay required of them. It was an interesting first step in designing a kind of game that was extremely simple in its rules and interaction but extremely complex in the forms of social dynamics it spawned.
6 Gameplay in Practice In the control both, we had a control which we could adjust to alter the sensitivity of the game during play. Turning up the sensitivity made the virtual target sizes larger and therefore gameplay easier. Turning down the sensitivity made the virtual target sizes smaller and gameplay harder.
Fig. 7. Squidball Gamers
For good gameplay, we felt it was essential that the players be able to make mistakes and learn from them. So, by default, we set the sensitivity fairly low. However, in some circumstances we increased the sensitivity to temporarily make the gameplay easier, giving the audience a little ”boost”. A person in the control booth was responsible for watching the progress of the game and making these ”group mind” decisions regarding when to adjust gameplay. We believe that similar controls were included in Cinematrix Even with a sensitivity control, there were some issues. One problem was that the audience at the start of show was less than a full house, which we had not anticipated in our game design. Because of this, some of the game levels proved hard to clear, because virtual targets were located in places where few audience members could reach them. This problem could be addressed by creating multiple configurations for different audience sizes.
32
C. Bregler et al.
A second issue was uneven audience distribution. Some people were in sparse sections of the audience, and did not get to participate as much as others. To address this concern, we enlisted student helpers to help move balls around. A third issue was that, during the game, people had to divide their attention between the screen and the balls. Some people decided to only watch the screen. Others ignored the screen and simply pushed the balls towards the center of the room. Initially, a relatively small percentage of ”aware” players actually watched both and drove the gameplay forward. The number of ”aware” players increased dramatically towards the end of the game, demonstrating that the game design principles were working. However, the split attention issue remains a challenge for any game design involving thousands of people, balls and a single screen. One solution might be to place multiple screens on all sides of the audience. However, this introduces another difficulty: coordination. Even with a single screen, players had difficulty coordinating the balls and target locations. The problem is that the player must face one direction, look at a screen over their shoulder, and then punch a ball in a third direction towards a target. Since few people have much practice at this activity, balls were popped left when they should have been popped right, or forward rather than back. Adding more screens would confound this issue. The problem of having to watch the balls and the screen is fundamental. Originally we also considered audio-only games, or games in which balls hitting each other is the point. We plan to reconsider those ideas in future experiments, and more detailed evaluations of the audience interaction.
7 Conclusions After all the hard work to create and setup Squidball for SIGGRAPH 2004, the roar of the crowd at the end of each a level was gratifying validation of our efforts. Since Squidball, we have been discussing possible iterations for future games. One option we have discussed is to use spotlights shone onto the crowd as targets, rather than using targets on a virtual screen. This would address some of the gameplay issues we encountered. The audience would have a physical cue showing where they are trying to get the balls to, rather than a virtual cue shown on a screen over their shoulder. Using spotlights, it would be possible to create roving patterns, enabling the spotlights to be moved in a pattern which ensures that everyone gets a chance to participate, taking into account the audience density and distribution. Of course, spotlights introduce a whole new set of technical challenges, though none that are insurmountable. We are considering this and other game design changes for Squidball 2. Acknowledgments. This work has been partially supported by NYU, NSF, ACM SIGGRAPH, Vicon Motion Systems Ltd, Apple Computer Inc, Advanced Micro Devices Inc, Alienware, Cycling 74, David Rokeby: very nervous systems, NVIDIA Corp., Segway Los Angeles. The audio was supplied for “Insert Coin” and ”Re-Atair” by Skott, “Superkid” by Max Nix, and “Watson Songs” by The Jesse Styles 3000. Furthermore we would like to thank especially our friends from AVW - TELAV, specificially Jim Irwin, Gary Clark, John Kennedy, Tom Popielski, Gerry Lusk, Mark Podany and Mike
Squidball: An Experiment in Large-Scale Motion Capture and Game Design
33
Gilstrap. Without their production efforts this would have been impossible. Also special thanks to Debbi Baum, Robb Bifano, Alyssa Lees, Jared Silver, Lorenzo Torresani, Gene Alexander, Boo Wong, Damon Ciarrelli, Jason Hunter, Gloaria Sed, Toe Morris, Scott Fitzgerald, Ted Warburton, Chris Ross, Eric Zimmerman, Cindy Stark, Brian Mecca, Carl Villanueva, and Pete Wexler for all their great help. And the SIGGRAPH chair Dena Slothower, who let us use their venue for the first big test.
References 1. B LAINE , T. 2000. The outer limits: A survey of unconventional musical input devices. In Electronic Musician. 2. C ARPENTER , L., 1993. Video imaging method and apparatus for audience participation. US Patent #5210604, #5365266. 3. F REEMAN , W. T., TANAKA , K., O HTA , J., AND K.K YUMA. 1996. Computer vision for computer games. In 2nd International Conference on Automatic Face and Gesture Recognition, Killington, VT, USA, IEEE. 4. M AYENES -A MINZADE , D., PAUSCH , R., AND S EITZ , S. 2002. Techniques for interactive audience participation. In IEEE Int. Conf. on Multimodal Interfaces, Pittsburgh, Pennsylvania. 5. M OTIONA NALYSIS S TUDIOS, 2004. Largest mocap volume.http://www.motionan-alysis. com /about mac/50x50volume.html. 6. U LYATE , R., AND B IANCIARDI , D. 2004. The interactive dance club: Avoiding chaos in a multi participant environment. In Int. Conf. on New Interfaces for Musical Expression. 7. V ICON, 2004. Sponsor of siggraph electronic theater. http://www.vicon.com.
Generating Ambient Behaviors in Computer Role-Playing Games Maria Cutumisu1, Duane Szafron1, Jonathan Schaeffer1, Matthew McNaughton1, Thomas Roy1, Curtis Onuczko1, and Mike Carbonaro2 1
Department of Computing Science, University of Alberta, Canada {meric, duane, jonathan, mcnaught, troy, onuczko}@cs.ualberta.ca 2 Department of Educational Psychology, University of Alberta, Canada {mike.carbonaro}@ualberta.ca
Abstract. Many computer games use custom scripts to control the ambient behaviors of non-player characters (NPCs). Therefore, a story writer must write fragments of computer code for the hundreds or thousands of NPCs in the game world. The challenge is to create entertaining and non-repetitive behaviors for the NPCs without investing substantial programming effort to write custom non-trivial scripts for each NPC. Current computer games have simplistic ambient behaviors for NPCs; it is rare for NPCs to interact with each other. In this paper, we describe how generative behavior patterns can be used to quickly and reliably generate ambient behavior scripts that are believable, entertaining and non-repetitive, even for the more difficult case of interacting NPCs. We demonstrate this approach using BioWare's Neverwinter Nights game.
1 Introduction A computer role-playing game (CRPG) is an interactive story where the game player controls an avatar called a player character (PC). Quickly and reliably creating engaging game stories is essential in today’s market. Game companies must create intricate and interesting storylines cost-effectively and realism that goes beyond graphics has become a major product differentiator. Using AI to create non-player characters (NPCs) that exhibit near-realistic ambient behaviors is essential, since a richer background “tapestry” makes the game more entertaining. However, this requirement must be put in context: the storyline comes first. NPCs that are not critical to the plot are often added at the end of the game development cycle, only if development resources are available. Consider the state-of-the-art for ambient behaviors in recent CRPGs. In Fable (Lionhead Studios), the NPCs wake at dawn, walk to work, run errands, go home at night, and make random comments about the disposition and appearance of the PC. However, the behaviors and comments are “canned” and repetitive and NPCs never interact with each other. The Elder Scrolls 3: Morrowind (Bethesda Softworks) has a huge immersive world. However, NPCs either wander around areas on predefined paths or stand still, performing a simple animation, never interacting with each other and ignoring the simulated day. In The Sims 2 (Electronic Arts), players control the NPCs (Sims) by choosing their behaviors. Each Sim chooses its own behaviors M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 34 – 43, 2005. © Springer-Verlag Berlin Heidelberg 2005
Generating Ambient Behaviors in Computer Role-Playing Games
35
using a motivational system if it is not told what to do. The ambient behaviors are impressive, but they hinge on a game model (simulation) that is integral to this game and not easily transferable to other game genres, including CRPGs. Halo 2 (Bungie) is a first person shooter with about 50 behaviors, including support for “joint behaviors” [11][12]. The Halo 2’s general AI model is described, but no model for joint behaviors is given. Façade [7] has an excellent collaborative behavior model for NPCs, but there are only a few NPCs, so it is not clear if it will scale to thousands of ambient NPCs. They also comment about the amount of manual work that must be done by a writer when using their framework. Other research includes planning, PaTNets, sensor-control-action loops [1][16], and automata controlled by a universal stack-based control system [3] for both low-level and high-level animation control, but not in the domain of commercial-scale computer games. However, planning is starting to be used in commercial computer games in the context of Unreal Tournament [5][21]. Crowd control research involves low-level behaviors such as flocking and collisions [14] and has recently been extended to a higher-level behavioral engine [2]. Group behaviors provide a formal way to reason about joint plans, intentions and beliefs [10]. Our approach is dictated by the practical requirements of commercial computer games. The model we describe in this paper is robust, flexible, extendable, and scalable [6] to thousands of ambient NPCs, while requiring minimal CPU resources. Moreover, our generative pattern abstraction is essential to story designers, shielding them from manual scripting and the synchronization issues of collaborative behaviors, and allowing them to concentrate on story construction. In most games, scripts control NPC behaviors. A game engine renders the story world objects, generates events on the objects, dispatches events to scripts and executes the scripts. Different stories can be “played” with the same game engine using story-specific objects and scripts. Programmers create game engines using programming languages such as C or C++. Writers and artists, who are not usually programmers [17], write game stories by creating objects and scripts for each story. The goal of our research is to improve the way game stories, not game engines, are created. A writer may create thousands of game objects for each story. If a game object must interact with the PC or another game object, a script must be written. For example, BioWare Corp.’s popular Neverwinter Nights (NWN) [15] campaign story contains 54,300 game objects of which 29,510 are scripted, including 8,992 objects with custom scripts, while the others share a set of predefined scripts. The scripts consist of 141,267 lines of code in 7,857 script files. Many games have a toolset that allows a writer to create game objects and attach scripts to them. Examples are BioWare’s Aurora toolset that uses NWScript and Epic Game’s UnrealEd that uses UnrealScript. The difficulties of writing manual scripts are well documented [12]. Writers want the ability to create custom scripts without relying on a set of predefined scripts or on a programmer to write custom scripts. However, story creation should be more like writing than programming, so a writer should not have to write scripts either. A tool that facilitates game story writing, one of the most critical components of game creation, should: 1) be usable by non-programmers, 2) support a rich set of non-repetitive interactions, 3) support rapid prototyping, and 4) eliminate most common types of errors. ScriptEase [19] is a publicly available tool for creating game stories using a high-level menu-driven “programming” model. ScriptEase solves the nonprogrammer problem by letting the writer create scenes at the level of “patterns”
36
M. Cutumisu et al.
[9][13]. A writer begins by using BioWare’s NWN Aurora toolset to create the physical layout of a story, without attaching any scripts to objects. The writer then selects appropriate behavior patterns that generate scripting code for NPCs in the story. For example, in a tavern scene, behavior patterns for customers, servers and the owner would be used to generate all the scripting code to make the tavern come alive. We showed that ScriptEase is usable by non-programmers, by integrating it into a Grade 10 English curriculum [4]. The version of ScriptEase that was used had a rich set of patterns for supporting interactions between the PC and inanimate objects such as doors, props and triggers. It also had limited support for plot and dialogue patterns (the subject of on-going work). In this paper, we describe how we have extended the generative pattern approach of ScriptEase to support the ambient behaviors of NPCs. NPC interactions require concurrency control to ensure that neither deadlock nor indefinite postponement can occur, and to ensure that interactions are realistic. We constructed an NPC interaction concurrency model and built generative patterns for it. We used these patterns to generate all of the scripting code for a tavern scene to illustrate how easy it is to use behavior patterns to create complex NPC interactions. The ambient background includes customers, servers and an owner going about their business but, most importantly, interacting with each other in a natural way. In this paper, we describe our novel approach to NPC ambient behaviors. It is the first time patterns have been used to generate behavior scripts for computer games. The research makes three key contributions: 1) rich backgrounds populated with interacting NPCs with realistic ambient behaviors are easy to create with the right model, 2) pattern-based programming is a powerful tool and 3) our model and patterns can be used to generate code for a real game (NWN). We also show that the patterns used for creating the tavern scene can be reused for other types of NPC interactions. Finally, we show how ambient behavior patterns are used to easily and quickly regenerate and improve all of the behaviors of all ambient NPCs in the NWN Prelude.
2 Defining, Using, and Evaluating Ambient Behavior Patterns A standard CRPG tavern scene can be used to demonstrate ambient behavior patterns. We focus on three ambient behavior patterns from this scene: owner, server, and customer. More complex behaviors could be defined, but these three behaviors already generate more complex behavior interactions than most NPCs display in most CRPGs. In this section, we describe the basic behaviors generated for these patterns, how a story writer would use the patterns and how the patterns were evaluated. Each pattern is defined by a set of behaviors and two control models that select the most appropriate behavior at any given time. In general, a behavior can be used proactively (P) in a spontaneous manner or reactively (R) in response to another behavior. Table 1 lists all of the behaviors used in the tavern. Some behaviors are used independently by a single NPC. For example, posing and returning to the original scene location are independent behaviors. This paper addresses only high-level behaviors, since the NWN game engine solves low-level problems. For example, if the original location is occupied by another creature when an NPC tries to return, the game engine moves the NPC as close as possible. Subsequent return behaviors may allow the NPC to return to its original location. Behaviors that involve more than one
Generating Ambient Behaviors in Computer Role-Playing Games
37
NPC are collaborative (joint) behaviors. For example, an offer involves two NPCs, one to make the offer and one to accept/reject it. The first column of Table 1 indicates whether a proactive behavior is independent or collaborative. Note that interactions with the PC are not considered ambient behaviors and they are not discussed in this paper. The most novel and challenging ambient behaviors are the ones that use behaviors collaboratively (interacting NPCs). The second column lists the proactive behaviors. The letters in parentheses indicate which kind of NPC can initiate the proactive behavior. For a collaborative behavior, the kind of collaborator is given as part of the behavior name, e.g., the “approach random C” behavior can be initiated by a server or customer (S, C) and the collaborator is a random customer (C). Table 1. Behaviors in the Server (S), Customer (C), and Owner (O) Patterns
Behavior Type Independent
Collaborative
Proactive Behavior
Reactive Chains
pose (S, C, O) return (C, O) approach bar (S, C) fetch (O) approach random C (S, C) talk to nearest C (C) converse with nearest C (C) ask-fetch nearest S (C, O) ask-give O (C) offer-give to nearest C (O)
pose, done return, done approach, done fetch, done approach, done speak, speak, converse* (speak, speak)+ done speak, fetch, receive, speak, done speak, give, receive, speak, done speak, decide, ask-give*; (accept) speak, decide, speak, done (reject) speak, decide, ask-fetch*; (accept) speak, decide, speak, done (reject)
offer-fetch to nearest C (S)
The third column of Table 1 shows the reactive chains for each proactive behavior. For example, the ask-fetch proactive behavior generates a reactive chain where the initiator speaks (choosing an appropriate one-liner randomly from a conversation file), the collaborator fetches (goes to the supply room while speaking), the initiator receives something, the collaborator speaks and the done behavior terminates the chain. Each reactive chain ends in a done behavior, unless another chain is reused (denoted by an asterisk such as converse* in the talk behavior). Each behavior consists of several actions. For example, a speak behavior consists of facing a partner, pausing, performing a speech animation and uttering the text. A ()+ indicates that the parenthesized behaviors are repeated one or more (random) times. For example, the converse proactive behavior starts a reactive chain with one or more speak behaviors, alternating between two characters. The talk proactive behavior starts a reactive chain with a speak behavior (a greeting) for each interlocutor, followed by a converse behavior. The offer-give (owner offers a drink) and offer-fetch (server offers to fetch a drink) proactive behaviors each have two different reactive chains (shown in Table 1) depending on whether the collaborator decides to accept or reject the offer. The writer uses the Aurora toolset to construct the tavern area, populate it with customers, servers and an owner, and saves the area in a module. The writer then opens the module in ScriptEase and performs three kinds of actions. First, create some instances of the server, customer and owner patterns by selecting the patterns from a
38
M. Cutumisu et al.
menu and then binding each instance to an appropriate NPC. Second, bind the options of each pattern instance to game objects and/or values. Fig. 2 shows how to set options (gray tabs in the left pane) for the server NPC. The center pane shows how the Actor option is bound to a Server NPC, created in the Aurora toolset. The right pane shows how to (optionally) change the relative default proactive behavior chances described in Section 3. The spin chances do not need to add to 100 – they are automatically normalized. Third, select the “Save and Compile” menu command to generate 3,107 lines of NWScript code (for the entire tavern scene) that could be edited in the Aurora toolset if desired. The simplicity of the process hides the fact that a large amount of scripting code is generated to model complex interactive behaviors.
Fig. 1. Using ScriptEase ambient behavior patterns for a tavern scene
The behavior patterns are easy to use - creating and testing a tavern scene in NWN required less than half an hour. The generated code is efficient, producing ambient behaviors that are crisp and responsive, with no perceptible effect on response time for PC movement and actions. The NPCs interacted with each other flawlessly with natural movements. A scene with ten customers, two servers and an owner was left to play for hours without any deadlock, degradation in performance, repetition or indefinite postponement of behaviors for any actor. Since the effectiveness and performance of ambient behaviors is best evaluated visually, we illustrate our approach using a series of movies captured from actual game-play [19]. These patterns were designed for a tavern scene. However, they are general enough to generate scripts for other scenes. For example, in a house scene, the customer pattern can be used for the inhabitants, the server pattern for a butler, and the owner pattern for a cook. The butler interacts with the inhabitants, fetching for them by going to the kitchen. The inhabitants talk amongst themselves and the cook occasionally fetches supplies. Our approach handles group (crowd) behaviors in a natural way. The customers constitute an example of a crowd – a group of characters with the same behavior, but each selecting different behaviors based on local context.
Generating Ambient Behaviors in Computer Role-Playing Games
39
To determine the range of CRPG ambient behaviors that can be accommodated by patterns, we conducted a case study for the Prelude of the NWN official campaign, directed at both independent and collaborative behaviors. The original code used adhoc scripts to simulate collaborative behaviors. We removed all of the manually scripted NPC behaviors and replaced them with behaviors generated from patterns. Six new ambient behavior patterns were identified: Poser, Bystander, Speaker, Duet, Striker, and Expert. These patterns were sufficient to generate all of the NPC ambient behavior scripts. Further evidence of the generality of ambient behavior patterns will require a case study that replaces behaviors in other game genres as well. There is no reason why a soccer or hockey goaltender could not be provided with entertaining ambient behaviors to exhibit when the ball (puck) is in the other end of play, such as standing on one leg, stretching, leaning against a goal post, or trying to quiet the crowd with a gesture. For example, one of the criticisms for EA FIFA 04 was directed to the goalie’s behavior [18] and will be addressed in the announced EA FIFA 06 [8].
3 Creating New Ambient Behavior Patterns To create new behavior patterns or adapt existing behavior patterns, one must look one level below the pattern layer at how the patterns are constructed from basic behaviors. A pattern designer can compose reusable basic behaviors to create a new behavior pattern or add basic behaviors to existing patterns, without writing any scripts. It is easy to mix/combine behaviors. There are two more levels below the pattern construction layer – the concurrency control and the script layers. Each behavior pattern includes a proactive model and a reactive model. The proactive model selects a proactive behavior based on probabilities. This simplest proactive model uses static probabilities assigned by the writer. For example, the server pattern consists of the proactive behaviors approach a random customer, approach the bar, offer-fetch a drink to the nearest customer and pose. In this case, a static probability distribution function [.10, .05, .03, .80] could be used to select one of these behaviors for each proactive event. The left pane of Fig. 2 shows the proactive model for the server. The reactive model specifies a reactive chain for each proactive behavior. For example, the right pane of Fig. 2 shows the reactive chain for the server’s offer-fetch proactive behavior listed in Table 1. Each reactive behavior fires an event that triggers the next reactive behavior until a done behavior signals the end of the reactive chain. The circle identifies the actor that performs the behavior (S, server; C, customer). Other options, such as what is spoken, have been removed from the diagram for clarity. Each of the other three proactive behaviors for the server (approach bar, approach customer, and pose) has a reactive chain that consists of a single behavior followed by a done behavior, as listed in Table 1. A behavior can use selection to choose between multiple possible following behaviors. For example, the decide behavior can fire either one of two speak events based on the customer’s drink wishes. A loop can be formed when a behavior later in the chain fires an event earlier in the chain. Loop termination can result from using selection to exit the loop. In general, the reactive model could be a cyclic graph, providing complete expressive power. For ambient behaviors, loops do not appear to be necessary – reactive chains (with decision points) seem to be sufficient. For non-ambient
40
M. Cutumisu et al.
behaviors, these loops may be necessary. Each proactive behavior that has reactive components serves as an entry point into the reactive model. The simplicity of the reactive model hides a necessarily complex concurrency model underneath (described in Section 4). The basic behaviors we created for the tavern scene (speak, decide, receive etc.) provide sufficient reusable components to create other ambient behavior patterns. However, it is easy to create new reusable basic behaviors as well. A new basic behavior is a series of simple ScriptEase actions, such as move to a location/object or face a direction. If no new basic behaviors are required, a new behavior pattern can be constructed in about an hour. Each new basic behavior could also take about an hour to complete. Once made, basic behaviors can be reused in many behavior patterns and behavior patterns can be reused in many stories. ScriptEase contains a pattern builder that allows a pattern designer to create new encounter patterns. We have added support to it for building basic behaviors and ambient behavior patterns.
Fig. 2. The proactive model and a reactive chain for the Server pattern’s offer-fetch behavior
Our proactive model also supports complex decisions, based on motivation or context so that it can be used for NPCs that are more important to the story. In each case, the probabilities for each proactive behavior are dynamic, based on either the current motivations (state of the NPC) or the context (state of the world). However, in this paper we focus on a static probabilistic proactive model – most NPC “extras” do not need motivational models to control their ambient behaviors.
4 The Concurrency Control Model Concurrency models have been studied extensively for general-purpose computing. A description of the difficulties in building a concurrency model for interacting NPCs is beyond the scope of this paper. However, we raise a few points to indicate the difficulty of this problem. First, synchronization between actors is essential so that an actor completes all of the actions for an event before the next event is fired. For example, the server should not fetch a drink before the customer has decided whether to order a drink or not. Second, deadlock must be avoided so a pair of actors does not
Generating Ambient Behaviors in Computer Role-Playing Games
41
wait forever for each other to perform a behavior in a reactive chain. Third, indefinite postponement must be avoided or some behaviors will not be performed. Our concurrency control mechanism is invisible to the story writer and is only partially visible to the pattern designer. It has proactive and reactive components that use proactive and reactive events respectively (user-defined events are used in NWN). The proactive model has a proactive controller. When the PC enters an area, the controller triggers a register proactive event on each NPC within a range of the PC. There is no need to control ambient behaviors in areas not visible to the user, since doing so slows down game response. In games such as Fable, NPCs uphold their daily routine whether the user can see them or not. Computational shortcuts are needed to minimize the overhead. On each NPC, the registering proactive event triggers a spin behavior that, in turn, fires a single proactive event (for instance, offer-fetch) as a result of a probabilistic choice among all the proactive behaviors that the actor could initiate. The selected proactive event (offer-fetch) fires a single reactive event that corresponds to the first behavior in the reactive chain (speak). To follow the chain properly, each behavior event (proactive or reactive) has one additional string parameter called the context. As its last action, the basic behavior for each event (except decide and done) fires a reactive event with this context as a parameter. The pattern designer creates a reactive chain by providing suitable context values in the correct order for the desired chain. For example, to construct the ask-fetch chain from Table 1, the designer provides the context parameters: “speak”, “fetch”, “receive”, “speak”, “done”. The decide behavior returns its context parameter with either “-yes” or “-no” appended so the reactive event can select the next appropriate event. This reactive control model ensures synchronization in a single chain by preventing an actor from starting a behavior before the previous behavior is done. However, it does not prevent synchronization problems due to multiple chains. For example, suppose the server begins the reactive chain for the offer-fetch proactive behavior shown in Fig. 1 by speaking a drink offer, and suppose the owner starts a proactive ask-fetch behavior to send the server to the supply room. The server will receive events from both its own reactive offer-fetch chain and the owner’s reactive ask-fetch chain in an interleaved manner that violates synchronization. To ensure synchronization, we introduced an eye-contact protocol that ensures both actors agree to participate in a collaborative reactive chain before the chain is started. Actor1 suspends all proactive events and tries to make eye-contact with actor2. If actor2 is involved in a reactive chain, actor2 denies eye-contact by restarting actor1’s proactive events. If actor2 is not involved in a reactive chain, actor2 sends a reactive event to actor1 to start the appropriate reactive chain. This protocol cannot be implemented with events alone, so we use state variables of the actors. We use another mechanism to eliminate deadlock and indefinite postponement. Either of these situations can arise in the following way. First, an eye-contact is established with an actor, so that the proactive controller does not generate another proactive event. Second, at the conclusion of the reactive chain started by the eye-contact, the actor is not re-registered to generate a new proactive event. Not only will this actor wait forever, but the other actor in the collaborative reactive chain can wait forever as well. One way for this situation to occur is for a script to clear all of the actions in an actor’s action queue, including an expected action to fire an event in the reactive chain. In this case, the reactive chain is broken and the proactive controller
42
M. Cutumisu et al.
will never generate another proactive event for the NPC. For example an NPC’s action queue is cleared if the user clicks on an NPC to start a conversation between the PC and the NPC. Our solution uses a heartbeat event to increment a counter for every NPC and to check whether the counter has reached a specific value. The game engine fires a heartbeat event every 6 seconds. If the counter reaches a threshold value, that NPC’s ambient proactive controller is restarted. The counter is reset to zero every time an event is performed by the NPC, so as long as the NPC is performing events (not deadlocked) no restart will occur. Neither the story writer nor the pattern designer need be aware of these transparent concurrency control mechanisms. We have recently added a perceptive model to our system. The perceptive model allows NPCs to be aware of the PC’s presence and act accordingly. When an NPC who is performing its proactive/reactive behaviors perceives the PC, the NPC’s action queue is cleared, proactive behavior generation is suspended, and the NPC performs an appropriate perceptive behavior. After the perceptive behavior is completed, proactive behavior generation resumes. This model allows NPC behaviors to be interrupted and it also supports NPC-PC interactions in addition to NPC-NPC collaborations. The success of this exercise has shown the robustness and flexibility of our proactive and reactive models, and of the underlying concurrency control mechanism.
5 Conclusion We described a model for representing NPC ambient behaviors using generative patterns that solves the difficult problem of interacting NPCs. We implemented this model in the NWN game using ScriptEase generative patterns. We are building a common library of rich ambient behavior patterns for use and reuse across CRPGs. Our next goals are to develop patterns that support NPCs that are more central to the plot of the game and NPCs that act as henchmen for the PC. Each of these goals involves escalating challenges, but we have constructed our ambient behavior model with these challenges in mind. For example, the model supports the non-deterministic selection of behavior actions based on game state. For ambient behaviors this approach can be used with a static probability function to eliminate repetitive behaviors that are boring to the player. For non-ambient behaviors these probabilities can be dynamic and motivation-based for more challenging opponents and allies. We have constructed a synchronization model that is scalable to the more complex interactions that can take place between major NPCs and between these NPCs and the PC. We demonstrated our approach using a real commercial application, BioWare Corp.'s Neverwinter Nights game. However, our model could have a broader application domain that includes other kinds of computer games, synthetic performance, autonomous agents in virtual worlds, and animation of interactive objects. Acknowledgements. This research was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC), the Institute for Robotics and Intelligent Systems (IRIS), and Alberta’s Informatics Circle of Research Excellence (iCORE). We are grateful to our anonymous reviewers for their valuable feedback.
Generating Ambient Behaviors in Computer Role-Playing Games
43
References 1. Badler, N., Webber, B., Becket, W., Geib, C., Moore, M., Pelachaud, C., Reich, B., and Stone, M.: Planning and Parallel Transition Networks: Animation's New Frontiers. In Computer Graphics and Applications: Pacific Graphics '95 (1995) 101-117 2. Caicedo, A., Thalmann, D.: Virtual Humanoids: Let Them Be Autonomous without Losing Control. In the 4th Conference on Computer Graphics and Artificial Intelligence (2000) 3. Capin, T.K., Pandzic, I.S., Noser, H., Thalmann, N. M., and Thalmann, D.: Virtual Human Representation and Communication in VLNET. IEEE Computer Graphics and Applications. 17(2) (1997) 42-53 4. Carbonaro, M., Cutumisu, M., McNaughton, M., Onuczko, C., Roy, T., Schaeffer, J., Szafron, D., Gillis, S., Kratchmer, S.: Interactive Story Writing in the Classroom: Using Computer Games. In Proceedings of the International Digital Games Research Conference (DiGRA 2005). Vancouver, Canada (2005) 323-338 5. Cavazza, M., Charles, F. and Mead, S.J.: Interacting with Virtual Characters in Interactive Storytelling. In ACM Joint Conference on Autonomous Agents and Multi-Agent Systems. Bologna, Italy (2002) 318-325 6. Charles, F. and Cavazza, M.: Exploring the Scalability of Character-based Storytelling. In ACM Joint Conference on Autonomous Agents and Multi-Agent Systems (2004) 872-879 7. Mateas, M. and Stern, A.: Façade: An Experiment in Building a Fully-Realized Interactive Drama. Game Developers Conference (GDC 2003), Game Design Track (2003) 8. GameSpot EA FIFA Soccer 2006: http://www.gamespot.com/xbox360/sports/fifa2006/ preview_6125667.html 9. Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns: Elements of Reusable Object-Oriented Software. Reading, MA, Addison-Wesley (1994) 10. Grosz, B. and Kraus, S.: Collaborative Plans for Complex Group Actions. Artificial Intelligence. 86 (1996) 269 -358 11. Isla, D.: Handling Complexity in the Halo 2 AI. Game Developers Conference (GDC 2005) 12. McNaughton, M., Cutumisu, M., Szafron, D., Schaeffer, J., Redford, J., Parker, D.: ScriptEase: Generative Design Patterns for Computer Role-Playing Games. In Proceedings of the 19th IEEE Conference on Automated Software Engineering (ASE 2004) 88-99 13. McNaughton, M., Redford, J., Schaeffer, J. and Szafron, D.: Pattern-based AI Scripting using ScriptEase. In Proceedings of the 16th Canadian Conference on Artificial Intelligence (AI 2003). Halifax, Canada (2003) 35-49 14. Musse, S. R., Babski, C., Capin, T. K., and Thalmann, D.: Crowd Modelling in Collaborative Virtual Environments. In Proceedings of ACM Symposium on VRST (1998) 115-123 15. Neverwinter Nights: http://nwn.bioware.com 16. Perlin, K. and Goldberg, A.: Improv: A System for Scripting Interactive Actors in Virtual Worlds. In Proceedings of SIGGRAPH 96. New York. 29(3) (1996) 205-216 17. Poiker, F.: Creating Scripting Languages for Non-programmers. AI Game Programming Wisdom. Charles River Media (2002) 520-529 18. Review Amazon, EA FIFA Soccer 2004: http://www.amazon.com/exec/obidos/tg/detail//B00009V3KK/104-2888679-3521549?v=glance 19. ScriptEase (2005): http://www.cs.ualberta.ca/~script/scriptease.html 20. Valdes, R.: In the Mind of the Enemy: The Artificial Intelligence of Halo 2 (2004): http://stuffo.howstuffworks.com/halo2-ai.htm 21. Young, R. M.: An Overview of the Mimesis Architecture: Integrating Intelligent Narrative Control into an Existing Game Environment. In AAAI Spring Symposium on Artificial Intelligence and Interactive Entertainment, USA (2001)
Telepresence Techniques for Controlling Avatar Motion in First Person Games Henning Groenda, Fabian Nowak, Patrick R¨ oßler, and Uwe D. Hanebeck Intelligent Sensor-Actuator-Systems Laboratory, Institute of Computer Science and Engineering, Universit¨ at Karlsruhe (TH), Karlsruhe, Germany {groenda, nowak}@ira.uka.de {patrick.roessler, uwe.hanebeck}@ieee.org
Abstract. First person games are computer games, in which the user experiences the virtual game world from an avatar’s view. This avatar is the user’s alter ego in the game. In this paper, we present a telepresence interface for the first person game Quake III Arena, which gives the user the impression of presence in the game and thus leads to identification with his avatar. This is achieved by tracking the user’s motion and using this motion data as control input for the avatar. As the user is wearing a head-mounted display and he perceives his actions affecting the virtual environment, he fully immerses into the target environment. Without further processing of the user’s motion data, the virtual environment would be limited to the size of the user’s real environment, which is not desirable. The use of Motion Compression, however, allows exploring an arbitrarily large virtual environment while the user is actually moving in an environment of limited size.
1
Introduction
Telepresence usually describes the state of presence in a remote target environment. This can achieved by having a robot gather visual data of the remote environment and present it to the user, who is wearing a head-mounted display. The robot imitates the motion of the user’s head and hands, which are tracked. As a result, the user only perceives the target environment, and his actions affect this environment, the user identifies with the robot, i. e., he is telepresent in the remote environment [1]. Of course, this technique can also be used in virtual reality, where the user controls an avatar instead of a robot. One of the biggest markets for virtual reality is first person games, i. e., games, where the user perceives the environment through the eyes of one of the game’s characters, his avatar. Using telepresence as a user interface to this kind of games, the user experiences a high degree of immersion into the game’s virtual environment and identifies fully with the avatar. Thus, telepresence techniques provide an appropriate interface for intuitive avatar control. M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 44–53, 2005. c Springer-Verlag Berlin Heidelberg 2005
Telepresence Techniques for Controlling Avatar Motion
45
Common input devices for avatar control are keyboards, mice, joysticks and game pads which all lack the ability of controlling the avatar intuitively. Force feedback, a simple kind of haptic feedback, improves the gaming experience, but does not provide more intuitive avatar control. There are several applications, that allow controlling avatars in immersive game environments. ARQuake [2] is an Augmented Reality interface to such a game. That means, in ARQuake the user perceives the real world that is augmented with computer-generated objects through a semi-transparent headmounted display. Another approach is applied in CAVE Quake II [3], which uses the CAVE Environment [4] to give the user realistic impression of the first person game Quake II. In order order to control the avatar, the user moves a device called wand, which resembles a joystick with six degrees of freedom. In other work, the user navigates an avatar through virtual environments by means of a walking-in-place metaphor [5] or by using complex mechanical devices, e. g. an omnidirectional treadmill [6]. Our approach is to combine first person games and extended range telepresence by means of Motion Compression [7]. Motion Compression allows the user in a confined user environment to control the avatar in an arbitrarily large virtual world by natural walking. The virtual world is presented to the user by a non-transparent head-mounted display and thus, the user is unable to perceive the physical world. As the avatar is controlled by normal walking, this system creates a high degree of realism. Additionally, distances and turning angles are kept locally equal in the physical and virtual world, which makes the avatar control intuitive. As the user has both visual and proprioceptive feedback this approach results in better navigation in the virtual environment, than when using common input devices [8, 9, 10]. A basic knowledge of Motion Compression is crucial for understanding how a virtual environment of arbitrary size can be mapped to the physical environment. Hence, the next section gives a brief review of this technique. Several possible solutions for connecting a first person game to an extended range telepresence system are discussed in section 3. The best solution is developed and the final implementation is described in section 4. Section 5 gives an experimental evaluation of our approach.
2
Motion Compression
Motion Compression transforms the user’s path in the user environment into the target environment. The user environment consists of the physical world surrounding the user, while the target environment is, in this application, the virtual environment of the game. Figure 1 illustrates the relations between the different environments. Motion Compression consists of three functional units described below. The path prediction unit predicts the path the user wants the avatar to take in the virtual environment. This path is called target path and is required by the next unit. The path is predicted based on head motion and information about
46
H. Groenda et al.
visual perception
motion data
user
target environment
user environment Motion Compression
4m
7m
Fig. 1. Overview of the different Motion Compression environments
user environment 4m (a)
target environment 8m (b)
Fig. 2. The left picture shows the transformed target path in the user environment. The right picture shows the corresponding target path in the target environment.
the virtual environment. A basic approach only uses the avatar’s current gaze direction for prediction. The path from the current position to a position in a fixed distance in gaze direction is assumed as the target path. This approach can be used for any virtual or remote environment. The path transformation unit maps the target path onto a path fitting into the user environment. Since the user environment is in most cases smaller than the target environment, the target path cannot always be mapped directly. Motion Compression aims at giving users a realistic impression of the virtual environment by keeping distances and turning angles in the user environment and virtual environment locally equal. Thus, only the curvature of the path is changed. A possible target path and its transformation are illustrated in figure 2. To give
Telepresence Techniques for Controlling Avatar Motion
47
the user a realistic impression of controlling the avatar, the resulting curvature deviation is kept at a minimum. In [7], it is shown that users felt comfortable walking in a virtual environment even with a relatively large curvature deviation. When walking, humans continuously check if they are on the direct way to their desired target, and adjust their direction accordingly. This behavior is exploited for user guidance. When walking, the avatar’s gaze direction is rotated slightly. The user compensates for this rotation and thus follows the transformed path in the user environment, while the avatar moves on the desired target path.
3
Possible Implementation Approaches
There are three possible approaches for employing Motion Compression to a first person game. These approaches are discussed below. 3.1
System Driver
A system driver is a piece of software responsible for connecting hardware devices to the operating system, which in return provides applications access to these devices. Game environments usually only accept input from keyboards, joysticks, game pads, and mice. A driver could be implemented which sends simulated key presses or mouse movements to the game depending on the user’s locomotion. First person games are usually controlled by more than one input device at once. Thus, it has to be taken care that the simulated device is equipped with a sufficient number of input keys and analogue inputs for controlling the avatar in all necessary ways. A long period of manual calibration is required in order to find out how long a key press has to be simulated because no position and orientation information from the game environment can be obtained. This calibration procedure needs to be done only once per game, and any game accepting the driver’s simulated input device can be supported. Unfortunately, small deviations between intended and executed movements cannot be prevented because manual calibration is error prone. Furthermore, this solution is confined to a specific operating system. 3.2
Communication Software
The idea of communication software like LIRC [11] is sending signals by use of Inter-Process-Communication to other applications. These signals are system messages which contain information about mouse movements, key presses and releases, and so on. This approach is less involved in the operating system than the driver approach and thus only depends on the operating system family. However, using a technique similar to the system driver, this approach requires a similar calibration effort. 3.3
Modification of the Game
The companies developing first person games often provide parts of the game’s source code, additional tools, and documentation to the fan community. Hence,
48
H. Groenda et al.
the community is able to create modifications which alter game play, design new weapons or power-ups, and create new enemies. Modifying the game’s source code is also the approach chosen for ARQuake. Changes were made in order to display the correct part of the virtual map. A detailed description can be found in [12]. Of course, the modification of an arbitrary game cannot be used for controlling an avatar in any other game but it apparently runs on any platform the game runs on. A modification also offers direct manipulation of the position and orientation of the avatar, rendering calibration unnecessary. This feature can be used for exactly placing the avatar on a specific position with a specific view angle and thus, any deviation between target and virtual environment is prevented. We chose this approach for implementation because of the capabilities for precise manipulation of the avatar’s position and orientation.
4
Implementation
The first person game Quake III Arena [13] was chosen to be modified because it is widespread, up-to-date, and renders fast and effectively. The resulting modification of Quake III Arena is called MCQuake. MCQuake is connected to the hardware/software framework for telepresent game-play, which is presented in [14]. This framework provides two modules. The first of these modules, the user interface, is responsible for tracking the user’s position and orientation as well as presenting the camera images to the user. The second module, MC Server, is a CORBA server that provides an implementation of the Motion Compression algorithm. This implementation calculates the target position, which is used as the position of the avatar in MCQuake, which implements the third module and thus completes the framework. The data flow between the user, MC Server, and MCQuake is shown in figure 3. When MCQuake is started a data connection to MC Server is established. During the game, this connection is used to continuously fetch the target position.The connection is maintained until the game is quit. The target position fetched from the Motion Compression implementation can be used in two different ways for moving the avatar. The first possibility is to set the position of the avatar exactly to the user’s target position. In this case, the collision detection of the game engine is bypassed as the avatar is not moved to the new position, but is directly set there in the virtual environment. Since MC Server has no information about obstacles in the target environment, the avatar can walk through them. However, this behavior is not desirable in most cases. Therefore, an alternative way of moving the avatar was implemented, that uses the game’s collision detection. This is achieved by calculating a motion vector as the difference between the avatar’s current position and the commanded position. This vector is then handed to the game engine, which now checks for collisions.
Telepresence Techniques for Controlling Avatar Motion tracking system
user position
49
MC Server
transformed position
visual feedback
user
MC Quake
Fig. 3. Data flow between MCQuake, the user, the Motion Compression implementation, and the tracking module
In both ways, the height of the target position has also to be mapped onto the avatar’s position in the virtual environment. The virtual environment supports two kinds of height information which are mapped differently as described below. The first kind of height information is the absolute height of the avatar, objects, and the floors in the virtual environment. This height is unrestricted allowing the avatar to fall, climb stairs, and walk on slopes. Since common input devices for first person games control avatar movement by commanding only two-dimensional moving directions, the game engine handles the avatar’s absolute height itself. If, for example, the user maneuvers the avatar over a set of stairs, the avatar’s absolute height changes with the height of the floor beneath him. Of course, changes of the absolute height cannot be simulated in the user environment. Thus, using these manipulation methods in MCQuake allows to move the avatar by normal walking in the user environment without restricting the virtual environment to only one fixed height. The second kind of height information, called view height, is relative to the floor the avatar moves on. In the game, however, it is restricted to only two different values used for crouched and normal movement. In the user environment, the tracking unit also provides the user’s view height relative to the physical floor, but as the user’s view height is not restricted to exactly two values, direct mapping is not possible. Nevertheless, crouched movement can be supported by defining a threshold. As long as the user’s view height is below this threshold, the avatar crouches. Given these mappings, the user is now able to control the avatar in an intuitive way through arbitrarily large virtual environments by normal walking. Of course, MCQuake also supports other kinds of motion, like running and strafing which are very common in first person games. For Motion Compression, there is no difference between those and normal walking as motion is handled as a sequence of position updates.
50
5
H. Groenda et al.
Experimental Evaluation
In the experimental setup, the user interface includes a non-transparent headmounted display with a resolution of 1280 × 1024 Pixels per eye and a field of view of 60◦ . This high quality display ensures a realistic impression of the game environment. The position and orientation of the head-mounted display is tracked with an acoustic tracking system. Fig. 5 shows the hardware setup of the user interface in the user environment. tracking system
tracking system
video displays (a)
(b)
Fig. 4. Schematic view of the user environment equipped with four emitters for the acoustic tracking system (a). The head-mounted display with four receivers attached to it (b).
In order to properly test if the users experienced the virtual environment like the real world, an environment well-known to the users was chosen. Hence the map from MensaQuake [15] was used. This map is a realistic model of the cafeteria of the University of Karlsruhe. An impression of this map is given in figure 5(a). Both, the Quake engine and MC Server, run on a multimedia LinuxPC, which allows a frame rate of approximately 80 images/s. The limiting factor of the setup is currently the tracking system, which gives 15 updates per minute. Development of the tracking system, however, aims at a better accuracy and a higher update rate [16] and was already enhanced with a gyroscope cube for better orientation estimation [1]. Ten users without any experience concerning telepresence control by Motion Compression were precisely observed when exploring the virtual cafeteria and asked about their impressions. Figure 5(b) shows a user playing MCQuake. In the current setup, the head-mounted display is supplied with video data by a strong and inflexible cable, which is, of course, invisible to the user. A second person is needed in order to take care of the cable and thus prevent the user from getting caught by it. This problem, however, can be addressed by building
Telepresence Techniques for Controlling Avatar Motion
(a)
51
(b)
Fig. 5. An impression of the student cafeteria in Quake [15] (a). A user playing MCQuake (b).
a wearable computer system for rendering game graphics, that communicates wirelessly with MC Server. Some users also stated to feel a little bit uncomfortable by the sudden change between normal and crouched view height when passing the threshold responsible for crouching. This is due to users expecting the virtual environment to reflect their own view height and thus proves a high degree of feeling present in the virtual environment. Despite the drawbacks stated above, the users’s impression of being present in the virtual environment did not decrease. It was further observed that after a few cautious steps all users were able to navigate intuitively and identified with the avatar, as was confirmed by the users afterwards. In order to obtain a more quantitative analysis of performance, a second experiment was conducted. This experiment compares the time a user needs to navigate his avatar along a specified path in the virtual cafeteria with MCQuake and with Quake with keyboard and mouse as inputs. In addition the user was asked to walk the same path in the real cafeteria. In order to avoid effects of adaption, a user was chosen for the experiment, who was experienced in both, using Motion Compression and Quake with standard input devices. He was also familiar with the cafeteria. The path was partitioned into three parts (a), (b), and (c), as shown in Fig. 6. Table 1 gives a comparison of the times of completion gathered in this experiment. When using Quake with standard inputs the avatar reaches his goal much faster than in MCQuake. This is a result of unrealistically high walking speed in standard Quake, even when the running mode is turned off. A comparison with the real cafeteria shows, that MCQuake provides a more realistic gaming experience than common game control. For example, turning 360 degrees in half a second is no longer possible.
52
H. Groenda et al.
up
down
(c)
(a) (b) 5m
Fig. 6. User path from the time of completion experiment in the virtual cafeteria
Table 1. Average time for a specified path from three runs (a) (b) (c) standard Quake 4.8 s 4.7 s 4.1 s real 8.9 s 14.0 s 15.4 s MCQuake 15.0 s 15.1 s 14.4 s
6
total 13.4 s 38.2 s 44.5 s
Conclusions and Future Work
Telepresence techniques were designed for controlling robots remotely. Since the remote environment can be easily replaced by a virtual environment, telepresence techniques can also be used to control an avatar in a first person game. Possible implementation approaches for connecting a telepresence system using the Motion Compression algorithm to the game Quake III Arena were evaluated. The resulting possibilities were system driver, communication software, and modification of the game environment. The approach of implementing a modification was chosen due to its exact positioning capabilities and the widest range of possibilities for controlling the avatar. Two different ways for moving the avatar were implemented since they offer orthogonal advantages such as exact positioning and in-game collision detection. In an experimental evaluation, Motion Compression as input for a first person game proved to be very intuitive and succeeded in the user feeling present in the virtual environment leading to a high degree of realism. These results are consistent with the results from a quantitative experiment. This experiment shows that the system presented in this work, results in completion times very similar to real world scenarios. In order to give the users the possibility to experience the virtual environment with all senses, a haptic feedback device, that allows to feel obstacles and weapon recoil, will be implemented. This will lead to an even higher degree of immersion. The authors expect systems like this to become common in gaming halls in the next couple of years. As soon as the hardware is affordable, people will even start installing these systems in their homes.
Telepresence Techniques for Controlling Avatar Motion
53
References 1. R¨ oßler, P., Beutler, F., Hanebeck, U.D., Nitzsche, N.: Motion Compression Applied to Guidance of a Mobile Teleoperator. In: Proceedings of the IEEE Intl. Conference on Intelligent Robots and Systems (IROS’05), Edmonton, AB, Canada (2005) 2. Piekarski, W., Thomas, B.: ARQuake: The Outdoor Augmented Reality Gaming System. ACM Communications 45 (2002) 36–38 3. Rajlich, P.: CAVE Quake II. http://brighton.ncsa.uiuc.edu/ prajlich/ caveQuake (2001) 4. Cruz-Neira, C., Sandin, D.J., DeFanti, T.A.: Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE. In: Proceedings of the 20th ACM Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 1993), Anaheim, CA, USA (1993) 135–142 5. Slater, M., Usoh, M., Steed, A.: Steps and Ladders in Virtual Reality. In: ACM Proceedings of VRST ’94 - Virtual Reality Software and Technology. (1994) 45–54 6. Iwata, H.: The Torus Treadmill: Realizing Locomotion in VEs. IEEE Computer Graphics and Applications 19 (1999) 30–35 7. Nitzsche, N., Hanebeck, U.D., Schmidt, G.: Motion Compression for Telepresent Walking in Large Target Environments. Presence 13 (2004) 44–60 8. Darken, R.P., Allard, T., Achille, L.B.: Spatial Orientation and Wayfinding in Large-Scale Virtual Spaces: An Introduction. Presence 7 (1998) 101–107 9. Peterson, B., Wells, M., Furness III, T.A., Hunt, E.: The Effects of the Interface on Navigation in Virtual Environments. In: Proceedings of Human Factors and Ergonomics Society 1998 Annual Meeting. Volume 5. (1998) 1496–1505 10. Tarr, M.J., Warren, W.H.: Virtual Reality in Behavioral Neuroscience and Beyond. Nature Neuroscience Supplement 5 (2002) 1089–1092 11. Bartelmus, K.S.C.: LIRC—Linux Infrared Remote Control. http://www.lirc.org (1999) 12. Thomas, B., Close, B., Donoghue, J., Squires, J., De Bondi, P., Morris, M., Piekarski, W.: ARQuake: An Outdoor/Indoor Augmented Reality First Person Application. In: Proceedings of 4th Intl. Symposium on Wearable Computers, Atlanta, GA, USA (2000) 139–146 13. id Software: Quake III Arena.http://www.idsoftware.com/games/quake/ quake3-arena (2001) 14. R¨ oßler, P., Beutler, F., Hanebeck, U.D., Nitzsche, N.: A Framework for Telepresent Game-Play in Large Virtual Environments. In: 2nd Intl. Conference on Informatics in Control, Automation and Robotics (ICINCO 2005), Barcelona, Spain (2005) 15. The MensaQuake Project: MensaQuake. http://mensaquake.sourceforge.net (2002) 16. Beutler, F., Hanebeck, U.D.: A New Nonlinear Filtering Technique for Source Localization. In: The 3rd IEEE Conference on Sensors (IEEE Sensors 2004), Vienna, Austria (2004) 413–416
Parallel Presentations for Heterogenous User Groups - An Initial User Study Michael Kruppa1,2 and Ilhan Aslan2 1
Saarland University 2 DFKI GmbH
[email protected],
[email protected] Abstract. Presentations on public information systems, like a large screen in a museum, usually cannot support heterogeneous user groups appropriately, since they offer just a single channel of information. In order to support these groups with mixed interests, a more complex presentation method needs to be used. The method proposed in this paper combines a large stationary presentation system with several Personal Digital Assistants (PDAs), one for each user. The basic idea is to ”overwrite” presentation parts on the large screen, which are of little interest to a particular user, with a personalized presentation on the PDA. We performed an empirical study with adult participants to examine the overall performance of such a system (i.e. How well is the information delivered to the users and how high is the impact of the cognitive load?). The results show, that after an initial phase of getting used to the new presentation method, subjects’ performance during parallel presentations was on par with performance during standard presentations. A crucial moment within these presentations is whenever the user needs to switch his attentional focus from one device to another. We compared two different methods to warn the user of an upcoming device switch (a virtual character ”jumping” from one device to another and an animated symbol) with a version, where we did not warn the users at all. Objective measures did not favour either method. However, subjective measures show a clear preference for the character version.
1
Introduction
In past years, there has been a growing interest in the personalization of information systems, such as in the museum area (see e.g. [1], [2]). The common goal of these projects is to increase the user’s satisfaction with the whole system, by adapting to the special interests of each user. However, all of these systems are based on personal devices to be carried around by the user. These personal devices provide the benefit of portability but can not offer the same multimedia capabilities as a stationary presentation system (as argued in detail in [3]). Thus, it is desirable to build a presentation system which combines the opportunity of adaptation on personal devices, with the advanced multimedia capabilities of modern presentation devices. M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 54–63, 2005. c Springer-Verlag Berlin Heidelberg 2005
Parallel Presentations for Heterogenous User Groups - An Initial User Study
55
A fairly simple way to support groups with mixed interests interacting with a single presentation device has been discussed in [4]. This uses voting, in order to help users to agree on a common topic to be presented on the presentation system. However, this solution is not based on personal adaptation but rather forces users to find some common interests. As long as users have at least some overlapping interests, the system might be adequate. As soon as the users don’t share any common interests, this system will, however, force the users to agree on something they do not really want or in the worst case, it will encourage users to find an unused presentation system, so that they can use it on their own. Instead of forcing users to agree on topics, we propose a system which combines personal devices and stationary presentation systems in order to provide presentations which fit the interests of all users sharing a stationary information system. The idea is to have a ”root presentation” on the stationary presentation system and to fill in personal presentations on the user’s personal device, whenever the actual topic of the ”root-presentation” does not fit the interests of the user (see [5] for further details). However, since the proposed presentation method has the potential of putting a high cognitive load on the user and thus might reduce the user’s recall, we conducted an empirical user study into the effectiveness of such multi-device presentations. In the experiment we performed, we were especially interested in 1) the effect of the parallel presentation method on a recall task and 2) any effect that different methods for guiding the user’s attentional focus in these complex presentations could have on the subjects. We believed the different methods for guiding the user’s focus might possibly: – Influence the cognitive load imposed on the user by the complex combined parallel presentations; – Have an effect on the time users need in order to switch their attention from one device to another; A virtual character, migrating from a personal to a stationary device has been implemented by Sumi and Mase [6]. However, there has been no evaluation into the effectiveness regarding user focus guidance by these characters. Since virtual characters have proven to be very effective in guiding a user’s attentional focus towards virtual objects in a virtual 3D world (see [7] and [8]), we expect the virtual character to successfully guide the user’s attentional focus in our scenario as well. Several different projects have dealt with the combined use of mobile and stationary devices (e.g. [9] and [10]), however the combined use of such devices to form a new presentation method has never been evaluated. Our belief, that subjects should be capable of focusing their attention on the correct device and not be distracted by the other, is based on the fact, that this work is closely related to the well investigated cocktail party effect (see [11], [12]). Experiments have shown that human beings are capable of focusing on a single audio stream, while there may be many other streams (i.e. voices) at the same volume around (see [13]). This capability is referred to as the cocktail party effect. In our experiment, we added a second information media, namely a
56
M. Kruppa and I. Aslan
visual one, in order to find out whether subjects would be capable of focusing on a specific, combined audio/visual information stream among other audio/visual streams.
2
Method
Subjects participating in the experiment were organized in groups of three. Each subject was equipped with a PDA and they were positioned in front of a large LCD panel (see Figure 1). The experiment itself had three phases. In each phase, the subjects watched a movie clip on the LCD panel. After a varying period of time, each subject was given one of three signals (a virtual character moving from the LCD panel to the PDA, or an animated Symbol on the PDA, or no signal at all) telling subjects to focus their attention on the PDA from that moment on. On the PDA, the subjects had to follow a short presentation, while the presentation on the LCD panel was continuing. As soon as the presentations on the PDAs came to an end, subjects were signalled to focus their attention back on the LCD panel(i.e. the character moving back to the LCD Panel, an animated signal on the LCD panel, or no signal at all). Afterwards, subjects would continue to watch the still running movie on the LCD panel until it would finish.
Fig. 1. Exemplary physical setup during the experiment
In order to figure out whether the combined presentation mode had an influence on the recall performance of subjects, a questionnaire had to be filled in after each presentation. The questions related both to content presented on the LCD panel (standard presentation mode) and on the PDAs (combined presentation mode). In a final questionnaire we asked for comments regarding the new presentation technology. Figure 2 illustrates the whole procedure of the experiment.
Parallel PDA3
Parallel PDA1 Parallel PDA2 Parallel PDA3
Panel Presentation 3 Parallel PDA1 Parallel PDA2 Parallel PDA3
Questionnaire
Parallel PDA2
Panel Presentation 2
57 Questionnaire
Parallel PDA1
Questionnaire
Panel Presentation 1
Questionnaire
Introduction
Questionnaire
Parallel Presentations for Heterogenous User Groups - An Initial User Study
Fig. 2. Procedure of the experiment
2.1
Subjects and Design
The subjects were 19 males and 23 females, all native speakers of German, recruited from the university campus and, on average, 28 years old. The subjects were paid 5 Euros for participation. The experiment lasted for approximately 40 minutes and was conducted in German. Subjects were organized in groups of three. The independent variables were Signal Method (character, symbol or none) and Presentation Method (standard, running on the LCD panel or parallel, running simultaneously on the PDA and the LCD panel). We defined the subjects’ recall test performance on an assessment questionnaire and the estimated time for user focus shifts as dependent variables. Both independent variables were manipulated within-subjects. Table 1 illustrates how the variable Signal Method was manipulated during the experiment (i.e. each subject would get a different signal in each of the three phases of the experiment). Table 1. Device switch signal method distribution throughout the experiment Presentation part
Subj.1 signal Subj.2 signal Subj.3 signal
One: Ballard
Character
Symbol
None
Two: Carter/Byrd
None
Character
Symbol
None
Character
Three: Leakey/Godall/Fossey Symbol
2.2
Materials
Presentations. The content presented to the subjects during the experiment was taken from a National Geographic Society publication, a DVD entitled: The Great Explorers and Discoverers. The content consisted of short video clips of approximately 4 minutes length. Each movie clip focused on the work and life of up to three explorers or discoverers. We carefully tried to select content that presented information not commonly known in our culture area. In order to do so, we had pre-tests with 7 subjects. In these pre-tests, we tried to determine the amount of previous knowledge regarding the information delivered during presentations. Based on this data, we selected the following clips on explorers and discoverers:
58
M. Kruppa and I. Aslan
– George Brass and Robert Ballard – Howard Carter and Richard Byrd – Louis Leakey, Jane Goodall and Diane Fossey In addition to these video clips, the DVD also featured in depth information on each explorer and discoverer in the form of spoken text and photographs. We took this material, to generate three different presentations with additional information on Robert Ballard, Richard Byrd and Louis Leakey. These additional presentations were shown to the subjects in the ”parallel mode” on the PDA, while the main presentation on the stationary information system was going on. Tests. Subjects were tested on their recall of the information presented with our parallel presentation system. The questions related both to visual- (i.e. the question showed a photo of a person, object or an event and asked subjects to describe what they saw) and audio-information (for example dates and names mentioned during presentations). Both types of questions were open-ended. Subjects were presented with an initial questionnaire asking for demographic information as well as for previous experiences with PDAs. After each presentation, subjects were given a questionnaire regarding the information delivered during that particular presentation. At the end of the third presentation, subjects were given a second questionnaire with subjective questions asking for a signalling preference as well as for general comments regarding the whole presentation system. Operationalization of the Independent Variables. The Signal Method did not influence the way the content was presented to the users, however, the character signal and the animated symbol worked in a different way: The character disappeared on the ”active device” (i.e. the one the user focuses on at the moment) and reappeared on the ”target device” (i.e. the device, the user should focus his attention on next). The symbol animation on the other hand simply occurred on the ”target device”, so users had to constantly monitor the ”passive device” in order not to miss the signal. The second independent variable Presentation Method was manipulated during each presentation. Apparatus. The hardware setup for the experiment consisted of a large, wallmounted LCD panel, a spatial audio system and 3 Hewlett Packard iPAQs with integrated wireless LAN capability. A standard Windows PC was used to run the presentations on the LCD panel and to render the sound to the spatial audio system. The presentations for both mobile and the stationary devices were realised with Macromedia Flash MX. Each Flash Movie connected via a socket connection to a server implemented in Java. The server was run on a separate Windows machine and controlled the timing of the whole experiment. The server allowed for simple, text based control by the experimenter, in order to start each presentation part after questionnaires were completed. The character engine had
Parallel Presentations for Heterogenous User Groups - An Initial User Study
59
been developed within the scope of the PEACH project1 and adapted for our purposes. The experiment was run in the intelligent environment at the Artificial Intelligence Lab of Professor Wahlster at the Saarland University, Germany. During each experimental session, an experimenter was present in order to answer any possible questions and to control the flow of the whole session. The subjects were told that the goal of the experiment was to evaluate the parallel presentation method. They were informed about the technical setup and about the way information would be presented to them. They were told that all information presented (i.e. images as well as spoken information) should be memorized as good as possible. In addition, they were told that after each of the presentations they would be tested on their knowledge regarding the information delivered during the precedent presentation. Just before the experimenter started the first presentation, he/she positioned the subjects in front of the LCD panel, assigned a PDA to each of them (together with a ”one-ear-headphone” connected to the PDA). The experimenter also informed the subjects that the questionnaire might include questions, regarding information they could not know, since they were expected to focus on another device, while that information was presented.
3
Results
In the following analysis an α level of .05 is used 2 . 3.1
The Visually Enhanced Cocktail Party Effect
The answers to the tests regarding the information delivered during presentations were scored by the experimenter. Each completely correct answer was awarded two points while partially correct answers were awarded one point. For each subject, separate scores were calculated: three scores (one for each presentation/experiment phase) for the average performance in answering questions related to information presented on the PDA (i.e. parallel presentation mode) and another three for the average performance in answering questions related to information presented on the LCD panel (i.e. standard presentation mode). To evaluate the performance during parallel presentations, we subtracted the average means for the standard presentations from those for the parallel presentations. Figure 3 shows the average results of this calculation for each presentation. The graph indicates that there was a strong improvement in the performance related to parallel presentations from the first to the last presentation. 1 2
Personal Experiences with Active Cultural Heritage(PEACH) - http://peach.itc.it Which means that if we observe an effect with p < .05, we may conclude that the probability that the sample means would have occurred by chance if the populations means are equal is less than .05.
0%
10%
No signal
20%
Symbol
30%
Character
20%
Presentation 3
40%
40%
Presentation 2
60%
M. Kruppa and I. Aslan
Presentation 1
60
0%
Fig. 3. left:Difference between recall performance related to PDA and panel presentations, right: recall comparison between different signals during parallel presentation
These data were then subjected to a paired samples t-test. The t-test testing the difference between presentation 1 and presentation 2 just very slightly missed the significance level (t(42) = −1.757;p = .086), however the t-test on the difference between presentation 2 and presentation 3 showed a highly significant result (t(42) = −4.168;p = .000). Also, the t-test testing the difference between presentation 1 and presentation 3 showed a highly significant result (t(42) = −7.581;p = .000). Thus, the analysis showed a positive learning effect among subjects during the experiment regarding the parallel presentations. After only two presentation runs with the new presentation method, subjects learned to efficiently focus their attention on the device which was presenting relevant information to them. In the third presentation run, the subjects recall performance regarding information delivered during parallel presentations nearly reached the performance level during standard presentations. In order to find out whether the user focus guidance worked throughout the experiment in general, two different scores were calculated for each subject: – The overall recall performance related to information delivered on the device users should focus their attention on – The overall recall performance related to irrelevant information (i.e. information presented on the device, users should not focus their attention on) A two-tailed t-test comparing the two values revealed a highly significant difference between the two values (t(42) = −16.902;p = .000). From this result we conclude that the subjects acceptance of the system’s active attentional focus guidance was very high. 3.2
User Focus Guidance Methods
In order to analyze the impact of Signal Method on the recall performance of subjects during parallel presentations, average scores for each standard presentation were calculated for each subject and subtracted from the average scores corresponding parallel presentations (in this way we make sure, that the result
Parallel Presentations for Heterogenous User Groups - An Initial User Study
61
is not influenced by the complexity of the different material presented). These scores were than ordered according to the signal method used while achieving the score. Figure 3 shows the result of this calculation. The graph indicates that subjects performed best when the Character was used to guide their attention. The performance during parallel presentations with no signal method was only slightly worse. The data was subjected to paired samples t-test. However, none of the t-tests comparing the different signal methods revealed any significance: – Character vs. Symbol: t(42) = 1.150;p = .257 – Character vs. no Signal: t(42) = .193;p = .848 – Symbol vs. no Signal: t(42) = −.933;p = .356 Even though the t-tests did not reveal any significance of these results, we believe that we can base the following assumption on the data: When using a signal method which forces the subjects to split their attention between devices (as with the animated Symbol, which obliged subjects to monitor the ”inactive” device, in order not to miss the signal for an upcoming device switch), the subjects overall recall performance may be decreased. 3.3
Subjective Assessment
The final questionnaire allowed subjects to state their preference for either one of the three signalling methods. They were also asked to give a reason for their decision. The numbers showed, that subjects had a clear preference for the virtual character (57% preferred the character, 24% preferred no signal at all, 7% preferred the animated symbol and 12% had no preference at all).
4
Discussion and Conclusions
The data supports our hypothesis that subjects would be capable of focussing on a single multimedia stream within a number of multimedia streams. The results clearly indicate that subjects were capable of concentrating on one device (during parallel presentations) and ignoring the presentation on the second device. We noticed a significant improvement in the subjects recall performance with respect to parallel presentations during the experiment. Based on these observations, we believe that in order to build public information systems capable of supporting heterogeneous user groups, our approach presented in this paper is very promising. Due to the strong learning effect regarding parallel presentations, we would suggest supporting users with a ”training phase” prior to presenting critical content. In this way users could get used to the new presentation method without missing important information. The subjective assessment shows a clear preference for the virtual character as signalling method. Opinions stated by subjects in the questionnaire indicate, that the virtual character was putting less stress on the subjects than the other methods.
62
M. Kruppa and I. Aslan
The data showed no statistically significant impact of the variable Signal Method on the recall performance of subjects during parallel presentations. However results indicated again that the animated Symbol showed the weakest performance. The performance of the character condition and the one without a signal were almost identical. Nevertheless, when considering this together with the fact that the majority of subjects stated in the final questionnaire, that they preferred the character as the signal method, we may conclude that a virtual character ”jumping” between devices may help users to follow complex, multi-device presentations, without putting additional cognitive load on the users. Although generalization of these results should only be done very carefully, they indicate some important implications for software development aiming at supporting heterogeneous user groups sharing a single public presentation system: – By combining a public multimedia information system with private personal digital assistants, it is possible to generate presentations which should support all the different interests of the users sharing the public device – Parallel multimedia presentations offer, after an initial training-phase, the same potential to deliver information as a standard, single device multimedia presentation – In order to allow users of the system to follow multi-device presentations without putting too much stress on them, an appropriate way of guiding the users attentional focus needs to be used. A virtual character migrating between the devices has a high potential to fulfil this task
5
Future Work
In a next step, we will concentrate not only on the presentation of information on public displays combined with personal displays but also on the interaction with such displays through personal devices (as proposed in [14]). We have already implemented the pilot architecture for such a project. Proposing interaction with a large public display through a private device, the interaction method is of utmost importance, considering privacy matters. Therefore, we would like to provide additional privacy measures by supporting multi modal interaction methods (i.e. writing and tabbing on a private display in contrast to speech and tabbing on a public display). Wasinger[15] investigated different combinations of multi modal interaction methods depending on their usability in a shopping scenario, where people had to interact via extra- (touching the real shopping articles) and intra gesture (tabbing on a virtual presentation on a small private display of the shopping articles)with shopping articles. We will adopt those methods to fit into our multi-device presentation scenario. Furthermore we would like to include multiple and also smaller public devices in our research (e.g. smart door displays).
Parallel Presentations for Heterogenous User Groups - An Initial User Study
63
Acknowledgements The work presented is supported by the International Post-Graduate College ”Language Technology and Cognitive Systems” of the German Research Foundation (DFG) and partly funded by the PEACH project of ITC-irst.
References 1. Rocchi, C., Stock, O., Zancanaro, M., Kruppa, M., Kr¨ uger, A.: The museum visit: generating seamless personalized presentations on multiple devices. In: Proc. of IUI Conference 04, Madeira, ACM Press (2004) pp 316–318 2. Not, E., Petrelli, D., Sarini, M., Stock, O., Strapparava, C., Zancanaro, M.: Hypernavigation in the physical space: adapting presentations to the user and to the situational context. In: The New Review of Hypermedia and Multimedia 4, London, Taylor Graham Publishing (1998) pp 33–46 3. Kruppa, M., Kr¨ uger, A.: Concepts for a combined use of Personal Digital Assistants and large remote displays. In: Proc. of Simulation and Visualization Conference (SimVis03), Magdeburg (2003) pp 349–361 4. Kruppa, M.: The better remote control - Multiuser interaction with public displays. In: Proc. of the MU3I workshop at IUI Conference 04, Madeira (2004)pp 1–6 5. Kr¨ uger, A., Kruppa, M., M¨ uller, C., Wasinger, R.: Readapting Multimodal Presentations to Heterogenous User Groups. In: Notes of the AAAI-Workshop on Intelligent and Situation-Aware Media and Presentations, AAAI Press (2002) pp 46–54 6. Sumi, Y., Mase, K.: Interface agents that facilitate knowledge interactions between community members. In: Life-Like Characters – Tools, Affective Functions, and Applications, Springer-Verlag (Cognitive Technologies series) (2004) pp 405–427 7. Lester, J.C., Towns, S.G., Callaway, C.B., Voerman, J.L., FitzGerald, P.J.: Deictic and emotive communication in animated pedagogical agents. Embodied conversational agents (2000) pp 123–154 8. Towns, S.G., Vorman, J.L., Callaway, C.B., Lester, J.C.: Coherent gestures, locomotion, and speech in life-like pedagogical agents. In: Proc. of IUI Conference 97, Florida (1997) pp 123–154 9. Myers, B.: Using Hand-Held Devices and PCs Together. In: Communications of the ACM. Volume 44, Issue 11. (2001) pp 34–41 10. Pham, T., Schneider, G., Goose, S.: A Situated Computing Framework for Mobile and Ubiquitous Multimedia Access using Small Screen and Composite Devices. In: Proc. of the 8th International ACM Conference on (Multimedia-00), Marina del Rey, California (2000) pp 323–331 11. Handel, S.: Listening: An Introduction to the Perception of Auditory Events, Cambridge, Massachusetts, MIT Press (1989) 12. Arons, B.: A Review of The Cocktail Party Effect. In: Journal of the American Voice I/O Society 12, Cambridge, Massachusetts (1992) pp 35–50 13. Stifelmann, L.J.: The Cocktail Party Effect in Auditory Interfaces: A Study of Simultaneous Presentation. In: MIT Media Laboratory Technical Report. (1994) 14. Kortuem, G.: Mixed initiative interaction: A model for bringing public and personal artefacts togehter. In: Dissappearing Computer Workshop. (2003) 15. Wasinger, R., Kr¨ uger, A., Jacobs, O.: Integrating Intra and Extra Gestures into a Mobile and Multimodal Shopping Assistant. In: Proc. of Pervasive Computing Conference 05, M¨ unchen (2005) pp 297–314
Performing Physical Object References with Migrating Virtual Characters Michael Kruppa1,2 and Antonio Kr¨ uger2 1
Saarland University 2 DFKI GmbH
[email protected],
[email protected] Abstract. In this paper we address the problem of performing references to wall mounted physical objects. The concept behind our solution is based on virtual characters. These characters are capable of performing reasonable combinations of motion, gestures and speech in order to disambiguate references to real world objects. The new idea of our work is to allow characters to migrate between displays to find an optimal position for the reference task. We have developed a rule-based system that, depending on the individual situation in which the reference is performed, determines the most appropriate reference method and technology from a number of different alternatives. The described technology has been integrated in a museum guide prototype combining mobile- and stationary devices.
1
Introduction
The integration of computing technology into the physical environment is rapidly moving forward. The main benefit of this development is, among the fact that the technology becomes accessible almost everywhere, the possibility to build services which are embedded into the physical environment and that directly use parts of the environment as an interface. In these mixed reality scenarios virtual information layers are superimposed on the physical world, which in turn make it easier for an instrumented space to guide the users attention to relevant physical objects in their vicinity (e.g. in a museum). This object reference task can be accomplished in several technological ways. Objects can be highlighted directly by a spotlight [1], can be referenced by an auditory spatial cue [2] or described by verbal messages [3]. In this paper we will explore another particular solution for this problem: object references by virtual characters. We believe that virtual characters have a special potential to perform this task that is grounded on their anthropomorphic nature very familiar to humans. Virtual characters have proven to successfully disambiguate references in virtual 3D worlds [4], these characters seem to promise similar results when performing references in the physical world. The Migrating Character concept described in this paper allows virtual characters to relocate themselves in physical space. Furthermore, the Migrating Characters are capable of performing many different types of references, depending on the available technology. Based on an ontology, M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 64–73, 2005. c Springer-Verlag Berlin Heidelberg 2005
Performing Physical Object References with Migrating Virtual Characters
65
representing both the world knowledge and a user model including the user’s actual context and preferences, we have developed a rule-based system that determines an optimal referencing solution in an arbitrary situation.
2
Related Work
This work has been inspired by Sumi’s and Mase’s AgentSalon [5]. Mase developed the idea of migrating agents that are able to move from the user’s personal device to a large screen, where the user’s character could meet with characters of other users and start a discussion on various topics. The focus in AgentSalon was on stimulating discussions between participants of an exhibition or conference. For this purpose it was sufficient to use the migrating agents in a rather narrow way (i.e. moving from a PDA to a large screen). Our work differs and extends these concepts in two ways. Firstly, we are generalizing the concept of migrating characters to make use of any possible projection and display capability in the environment. Secondly, we use the migrating characters mainly as a reference mechanism to physical objects in the user’s environment and not only as a representation of the user’s interest as originally intended in AgentSalon. Virtual Characters are not the only way to provide references to physical objects in instrumented spaces. Pinhanez original work on the Everywhere Display[1] includes also several ideas on how to reference physical objects in the environment, mainly by annotating them with arrows and labels. In contrast to this approach our work makes use of one single metaphor to reference physical objects, that of an anthropomorphic character. Our assumption is, that this character, familiar to the user, will enhance consistency and coherency of presentations. The virtual character Cosmo, developed by Lester et al. [4] has proven to be very effective in disambiguating references within a 3D virtual world. The character is able to relocate itself within the virtual space, it can perform gestures, mimics and spoken utterances. The goal of the Migrating Characters discussed in this paper, is to transfer the concept of deictic believability[4] of virtual characters in virtual 3D worlds to the physical world, by allowing a virtual character to ”freely” move within the physical space.
3
Migrating Character Concept - Overview
The Migrating Character concept is based on three major elements: mobility, sensitivity and adaptivity. The mobility aspect allows the character to relocate itself in physical space. Physical character locomotion, in combination with gestures and speech, is used to disambiguate physical object references. In order to allow the character to be helpful in the physical world, it must be capable of sensing the world around it. It must be capable of determining/knowing its own and its user’s location. Furthermore, it should have a certain amount of world knowledge in order to successfully determine appropriate referencing solutions.
66
M. Kruppa and A. Kr¨ uger
Finally, adaptation to user preferences will raise the overall satisfaction of users with the Migrating Characters. In the following subsections, we will have a closer look at each of these three main aspects of our concept. Character locomotion - Mobility. Character locomotion is the key element within our concept. It allows virtual characters not only to accompany users while exploring the physical world, but also to assist users by means of deictic gestures. We distinguish between active and passive character locomotion. In the active locomotion category we find all the methods allowing the character to relocate itself, regardless of the user’s movements, for example when the character ”jumps” from one device to another (similar to [6] and[7]) or moves along a projection surface. Whenever the character depends on the user in order to relocate itself, we refer to this movement as passive locomotion (for example, when the character is located on a mobile device carried around by the user). Both categories yield different advantages and implications: – Using active locomotion, the character is capable of positioning itself freely in the environment. Active locomotion can be the result of an explicit user request or an action performed by the character itself. – Using passive locomotion, the character is sure to be close to the user, however it depends on the user in order to reach a certain position. – Depending on the chosen locomotion method, the character uses either the same (passive locomotion) or a different (active locomotion) frame of reference of the user and must change its gestures/utterances accordingly. These observations demand different character behaviors depending on the actual/available locomotion method. We assume that the character is always driven by a certain objective. In case it is necessary to move to another location in order to fulfill a specific goal, the character could either move to the new location actively (in this case it should however ensure that the user is following) or it could try to convince the user to move to the new location and hence move the character passively. In either way, the character needs to be aware of the users movements/actions in order to react appropriately. Physical Context - Sensitivity. In order to determine an optimal solution to a given referencing task within the physical world, a virtual character needs to be aware of the physical objects around it (i.e. the objects around the user) as well as of the physical context of the user (i.e. position and orientation). Furthermore, the character will need detailed knowledge on the object to be referenced (e.g. What is it?;How big is it?;Which other objects are close to it?). Depending on the position, size, proximity and similarity of physical objects, different strategies need to be chosen in order to disambiguate references. While the user position and orientation may limit the choice of appropriate devices to be used by the character due to physical restrictions like visibility, it may also indicate that a user dislocation is necessary prior to performing an object reference (the influence of target object size and distance between target object and
Performing Physical Object References with Migrating Virtual Characters
67
referrer on the chosen reference strategy is supported in literature, for example in [8]). References to physical objects performed by the character are based on the frame of reference of the user and the (possibly different) frame of reference of the character and the location of the physical objects. Personal Context - Adaptivity. In order to maximize the user’s satisfaction with the Migrating Character, it is necessary to adapt to the user’s specific preferences. Preferences may be related to specific devices (e.g. a user may prefer to limit the use of the PDA screen to a minimum, even though he might have to move to another room in order to do so). In addition, users should be given the opportunity to prevent the use of certain devices or reference strategies in specific situations, for example a user might dislike the character to use a spatial audio system in a public situation, while the user might have a different preference in a private situation.
4
Physical Object References
We have developed several different techniques which will allow a Migrating Character to perform a unique object reference. These different techniques, together with a rule-based system determining the most suitable referencing solution in a specific situation, will be described in the following subsections.
A
B
C
Fig. 1. Examples of Migrating Characters performing physical object references
Referencing Methods. The simplest reference method to be performed by a Migrating Character is a sole utterance with no accompanying gesture or movement. However, this type of reference may only be performed, if the resulting utterance produces a unique reference. A more complex, but also more precise reference is achieved when the Migrating Character performs a combination of a gesture and an utterance on the PDA. In order to refer to a physical object, this method demands a photo or abstract visual representation of the physical object, which is to be shown on the screen of the PDA (see figure 1 A). If there is a wall mounted display next to the physical object to be referenced, another method is to let the Migrating Character disappear from the users PDA and reappear on the display (see figure 1 B). The last reference method is the most
68
M. Kruppa and A. Kr¨ uger
precise one, but also the technically most demanding. Using this reference type, the Migrating Character disappears from the PDA and reappears as a projection on the wall right next to the physical object which is to be referenced (see figure 1 C). Determining An Optimal Reference Strategy. Whenever a physical object reference is necessary, the rule-based physical reference system is instantiated. The data fed into the system is taken from an online ontology and transformed
Perform possibly ambiguous reference
Reference goal
no
Pas.Locomotion Lead user to the right room
no
User in right room?
Object picture available?
yes
Do reference on picture on PDA
yes no Ask user to turn around
no
Can place MC* next to object?
User looking at right wall?
yes
Act.Locomotion Place MC* on wall or display and perform reference
no
yes no Ask user to move closer to the wall
no
User close to wall?
Object picture available?
yes yes
User preference OK?
yes
Perform spoken (back)reference
yes no Perform spoken reference
no
Object close to similar object?
yes
Object in focus history?
* Migrating Character
Fig. 2. Schematic overview on the reference method determination process
into facts which are then asserted and evaluated by the rules defined in the rule-based system. These facts describe the room in which the user is located at the moment, the objects located in that room, the reference goal and the actual situative context of the user. Whenever instantiated, the rule-based system will determine the next step which is necessary in order to reach the final goal of a unique physical object reference. Based on the result produced by the rule-based system, the Migrating Character will perform the according actions and then (in case this wasn’t the final step of the reference task), the rule-based system will be instantiated again with the updated world model and user context. This process is illustrated in figure 2. Starting from the initial reference goal, the first three rules determine whether a user dislocation is necessary. If any one of these three rules is activated, the result determined by the rule-based system will not be a reference instruction for the Migrating Character but instead it
Performing Physical Object References with Migrating Virtual Characters
69
will be an instruction for a necessary physical user context change. Based on this instruction, the user is asked to relocate herself accordingly. Once, none of the first three rules are activated due to the user’s physical context, the result determined by the rule-based system will be a reference instruction for the Migrating Character. In case the object to be referenced is not close to a similar object (this is calculated based on the size and type of the target object and surrounding objects), a simple spoken reference will be sufficient. Otherwise, based on the user’s focus history (i.e. last two objects which have been referenced, see section 5.3), the availability of a picture of the target object, the availability of the necessary hardware and physical space for a character dislocation right next to the object, and the users preferences, different strategies are chosen to disambiguate the physical object reference. If all strategies fail, the worst case is a possibly ambiguous spoken reference by the Migrating Character.
5
Exemplary Migrating Character Implementation
Based on the Migrating Character concept, we have realized a system utilizing virtual characters in order to produce unique references to physical objects. Starting with an initial reference goal (e.g. a particular detail of a fresco), several hardware and software components are put into use, in order to determine an optimal referencing solution to be performed by the Migrating Character. The components forming our system will be discussed in the following subsections. 5.1
Character Engine
The Migrating Characters are realized (i.e drawn and programmed) in Macromedia Flash MX 1 . For each character, we developed two different layouts, one for devices with little display space (i.e. a PDA or tablet PC) and one for larger displays. The Migrating Character animations are controlled remotely over a socket connection. 5.2
Character Locomotion
We implemented and evaluated different techniques which allow the Migrating Characters to be relocated in physical and virtual space (i.e. relocate the visual representation and audio source of a Migrating Character). As mentioned in the discussion of the Migrating Character Concept, these methods may be split into two categories, namely passive and active locomotion. Passive Locomotion. To allow for passive character locomotion, we make use of the (visually) smaller character versions mentioned above(see figure 3). The character animations are integrated in a full screen user interface on the mobile device. This user interface is also implemented in Flash MX and 1
http://www.macromedia.com
70
M. Kruppa and A. Kr¨ uger
Fig. 3. Exemplary character gestures (small screen version)
offers the opportunity of connecting to the Migrating Character Server (MCS) via a simple socket connection over wireless lan. The whole application running on the mobile device is remotely controlled by the server which calculates the user position (based on the data the mobile device is receiving from the infrared beacons and the Radio Frequency ID tags, as described in 5.3) and generates xml formatted scripts which are then interpreted by the character animation on the mobile device. Active Locomotion. Active character locomotion is realized by two different methods. The simple one is a character movement on the screen of a stationary device. Similar to the mobile application, a full screen user interface is running on the stationary screens. The character is loaded on top of that and can be easily located anywhere on the screen, in order to perform references on images shown somewhere on the screen. The second method allows a virtual character to appear on an arbitrary surface of a room. A steerable projector-unit is used to visualize the virtual character. In our scenario the unit is mounted on the ceiling in the center of the room allowing to use any suitable walls and desk surfaces for projection. If the MCS decides to use the Migrating Character as a projection on the wall next to the target object, it requests access to the steerable projector and the spatial audio system from the device manager controlling the devices. Given access to these devices is granted, the MCS controls the devices via Java Remote Method Invocation. As the projection surfaces are usually not perpendicular to the projector beam, the resulting image appears distorted. In order to correct this distortion we apply the method described in [1]. We make also use of a spatial audio system that can be accessed trough a service within the room and allows applications to concurrently spatialize arbitrary sounds [2]. If appropriate, the MCS sends MP3 files (generated by a speech synthesis) and the coordinates of the current location of the character to the spatial audio system, which positions the sounds accordingly. The anthropomorphic interface of the Migrating Character obviously appears more natural with the speech being perceived from the same direction as the projection is seen. Character transition between displays. The transition from one display to another is underlined by sounds and an animation. The character disappears on the active device and reappears on the target advice. The sound accompanying this animation also starts at the active device and is continued on the target
Performing Physical Object References with Migrating Virtual Characters
71
device. In this way, the sound is an additional guidance method towards the new location of the Migrating Character.
5.3
The Situative Context of the User
The situative context of the user is a combination of sensory data, resulting in an estimated user position and orientation, knowledge on the actual physical surroundings of the user and a user specific reference history kept by the system. User Location And Orientation. We use two kinds of senders to detect user positions: Infrared beacons2 (IR beacons) and active Radio Frequency Identification tags3 (RFID tags). The beacons and tags are mounted at the ceiling and the user carries a PDA with integrated sensors (the standard infrared port of the PDA and a PCMCIA active RFID reader card). A received (or sensed) tag or beacon indicates that a user is near the location of the respective sender. Each IR beacon is combined with one RFID tag that provides the coordinates of the beacon and of the tag itself. When a tag or beacon is sensed by the PDA a Geo Referenced Dynamic Bayesian Network[9] (geoDBN) is instantiated and associated with the coordinate that is stored in its memory. This method allows us to track users indoors with a precision of 1-2 meters. In addition, due to the fact that the infrared beacons demand a direct line of sight between sender and receiver, they provide good estimate on the user’s orientation. World Knowledge Ontology. We store the necessary world knowledge in an ontology which is online accessible [10]. An online editor is used to set up the ontology and to put initial data into the ontology. Once the world knowledge is described, it may be accessed and modified via simple http requests. The result of such an request is an XML formatted document holding the desired information. At the moment, we have three different categories in the ontology, namely rooms, objects and users. Rooms and objects are defined by their size and location. In addition, objects have further information on their type, which is important when determining possible ambiguities while planning an object reference (see section 4). In case, objects are technical devices that can be used to display a Migrating Character, the ontology also allows to mark them as ”in use” or ”vacant”. Objects are always associated with rooms in the ontology and their positions are relative within the room coordinates. Information on users is stored in the ontology along with their preferences, location, orientation and history of references. In this way, the objects that are currently relevant at the users position, may be easily determined. The ontology structure also allows to easily find out, whether a user needs to relocate herself in order to be in reach of a certain object. 2 3
Produced by Eyeled. Produced by identec solutions.
72
M. Kruppa and A. Kr¨ uger
Reference History. A history of the last two objects referenced is kept for each user. This history may help to simplify references to objects, which have recently been in the focus of the user. In some cases, when referencing an object which is represented in this history, the Migrating Character may perform a unique spoken reference, which would have otherwise been ambiguous. If there are two objects of the same kind in the history, the spoken reference would have to be very precise, like for example: ”The last but one image we have seen”. If the reference history holds only a single object, or two objects of different kinds, the spoken reference may be reduced, for example: ”Back to the previous painting”.
6
Experiences
As a test bed for the Migrating Characters, we have chosen a museum scenario. The characters have been integrated into a previously developed museum guide system [6] based on PDAs and wall mounted displays, which we developed within the scope of the PEACH project4 . We have extended the behavior of these characters in order to allow them to disambiguate spoken object references. As additional hardware, we added a steerable projector and a spatial audio system. While the museum guide system features its own planning mechanism to determine appropriate content for a user, based on preferences and assumptions on user interests determined by observing the user’s interaction with the system, the rule-based physical reference system has been integrated in the presentation planning process. The resulting museum guide system offers the following new benefits: – Removing or displacing display devices will not result in system failure, but will only change the strategy chosen when referring to physical objects – In case a technical device is occupied, the system will simply choose a different reference strategy, instead of waiting for the device to become available – New physical and technical objects can be easily integrated by simply adding them to the knowledge ontology Furthermore, even though the system can’t guarantee for a unique reference in all possible situations, it is still very effective in disambiguating physical references in many situations, thus improving the comprehensibility of information delivered by our museum guide system regarding physical objects in a museum.
7
Conclusions and Future Work
In this paper we have presented an extended concept of how to use Migrating Characters for referencing physical objects in instrumented environments. We have implemented a prototype in a museum scenario to gain field experiences and provide further evidences for the technical feasibility of our approach. To 4
Personal Experiences with Active Cultural Heritage (PEACH) - http://peach.itc.it/
Performing Physical Object References with Migrating Virtual Characters
73
test the domain-independence of our approach, we are currently planning to apply the presented system and methods also to a shopping domain. So far we have only investigated single-user scenarios. We are now extending our ideas to multi-user scenario and are currently addressing complications that arise if multiple Migrating Characters are present in the same environment.
Acknowledgements The work presented is supported by the International Post-Graduate College ”Language Technology and Cognitive Systems” of the German Research Foundation (DFG) and partly funded by the PEACH project of ITC-irst.
References 1. Pinhanez, C.: The everywhere displays projector: A device to create ubiquitous graphical interfaces. In: Proc. of Ubicomp-2001, Atlanta, USA (2001) 315–331 2. Schmitz, M.: Safir: A spatial audio framework for instrumented rooms. In: Workshop on Invisible and Transparent Interfaces, in conj. with AVI-2004, Gallipoli, Italy (2004) 3. Kray, C.: Situated Interaction on Spatial Topics. DISKI series vol. 274, AKA Verlag, Berlin, Germany (2003) 4. Lester, J.C., Towns, S.G., Callaway, C.B., Voerman, J.L., FitzGerald, P.J.: Deictic and emotive communication in animated pedagogical agents. Embodied conversational agents (2000) 123–154 5. Sumi, Y., Mase, K.: Agentsalon: Facilitating face-to-face knowledge exchange through conversations among personal agents. In: Proceedings of Agents-2001, Montreal, Canada (2001) 393–400 6. Kruppa, M., Kr¨ uger, A., Rocchi, C., Stock, O., Zancanaro, M.: Seamless personalized TV-like presentations on mobile and stationary devices in a museum. In: Proceedings of ICHIM-2003, Paris, France (2003) 7. Sumi, Y., Mase, K.: Interface agents that facilitate knowledge interactions between community members. In: Life-Like Characters – Tools, Affective Functions, and Applications, Springer-Verlag (Cognitive Technologies series) (2004) 405–427 8. van der Sluis, I., Krahmer, E.: The Influence of Target Size and Distance on the Production of Speech and Gesture in Multimodal Expressions. In: Proceedings of ICSLP-2004, Jeju ,Korea (2004) 9. Brandherm, B., Schwartz, T.: Geo referenced dynamic Bayesian networks for user positioning on mobile systems. In: International Workshop on Location- and Context-Awareness in conj. with Pervasive-2005, Munich, Germany (2005) 223–234 10. Heckmann, D., Schwartz, T., Brandherm, B., Schmitz, M., von WilamowitzMoellendorff, M.: Gumo - the general user model ontology. In: Proceedings of UM-2005, Edinburgh, UK (2005) 428–432
AI-Mediated Interaction in Virtual Reality Art Jean-luc Lugrin1, Marc Cavazza1, Mark Palmer2 , and Sean Crooks1 1
School of Computing, University of Teesside, TS1 3BA Middlesbrough, United Kingdom {j-l.Lugrin, m.o.cavazza, s.crooks}@tees.ac.uk 2 University of the West of England, Bristol, United Kingdom
[email protected] Abstract. In this paper, we introduce a novel approach to the use of AI technologies to support user experience in Virtual Reality Art installations. The underlying idea is to use semantic representations for interaction events, so as to modify the course of actions to create specific impressions in the user. The system is based on a game engine ported to a CAVE-like immersive display, and uses the engine’s event system to integrate AI-based simulation into the user real-time interaction loop. The combination of a set of transformation operators and heuristic search provides a powerful mechanism to generate chain of events. The work is illustrated by the development of an actual VR Art installation inspired from Lewis Caroll’s work, and we illustrate the system performance on several examples from the actual installation.
1 Introduction In entertainment applications, Artificial Intelligence techniques have most often been used to implement embodied agents, or decision making for “system” agents. One more recent development concerns the use of AI to support user experience through new AI-based interactivity techniques. This is especially of interest for the development of artistic installations based on interactive 3D worlds [2][3][4][5][9]. One of the major difficulties in developing such installations is to properly translate the artistic intention into actual elements of interactivity, which in turn determine the user experience. The starting point of this research was to facilitate the description of highlevel behaviours for virtual worlds that would form part of VR art installations. Our underlying hypothesis has been that AI representations, inspired from planning formalisms [10], and AI-based simulation derived from these, could constitute the basis for virtual world behaviour, while facilitating the description from the artist of complex behaviours accounting for the user experience. In this paper, we introduce a new approach to interactivity, in which the consequences of user interactions can be dynamically computed so as to produce chain of consequences eliciting a specific kind of user experience. An important aspect is that this chain of events is computed from first principles embedding elements of the artistic brief. In other words, AI techniques are used for their ability to represent actions and computing analogical transformation on action representations using applicationdependent characterisation of local action outcomes (“naturalness”, “surprise”, “level of disruption”, etc.). M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 74 – 83, 2005. © Springer-Verlag Berlin Heidelberg 2005
AI-Mediated Interaction in Virtual Reality Art
75
After a brief introduction to the system architecture and its underlying principles, we introduce the artistic installation whose development supports our discussion throughout the paper. We then describe the mechanisms by which user interaction can trigger chains of events, whose logic is controlled by semantic representations of actions and objects. After presenting a detailed example of system operation, we conclude on the potential and generality of the approach.
2 System Overview and Architecture The system presents itself as an immersive virtual environment using a PC-based CAVE™-like device, the SAS Cube™. It is developed on top of the Unreal Tournament 2003™ game engine, which supports visualisation and basic interaction mechanisms [7]. The Unreal engine has also been ported to immersive displays using the latest version of the CaveUT system [5][6], which supports stereoscopic visualisation as well as head tracking. The rationale for using AI techniques to simulate behaviour in graphical worlds is to exploit a specific feature of game engines, namely the fact that they rely on event-based systems to represent all kinds of interaction. Event systems originated from the need to discretise physical interactions to simplify physical calculations: while the dynamics of moving objects would be subject to numerical simulation, the consequences of certain physical interactions (e.g. shattering following collisions) could be determined in a discretised system without having to resort to simulation of the collision itself. Our overall approach relies on the recognition of potential high-level actions from low-level physical events, to generate semantic representations of the virtual world’s events. In other words, from a low-level set of events such as collisions and contacts between objects we derive high-level actions such as “pushing”, “breaking”, “tilting”, etc., which represent the actions potentially observed by the user. Direct alteration of the course of these actions (through modification of their representation) will in turn affect the user experience, which is the essence of our application. The system operates by continuously parsing low-level events into high-level action representations that constitute an ontology of the main type of actions characterising
Fig. 1. System Overview (with head tracking)
76
J.-l. Lugrin et al.
a given environment. The behavioural system, represented by the AI Module, and the UT 2003 visualisation engine are integrated via an Event Interception System (EIS). This event system is a specific layer developed in UT 2003 that constantly intercepts low-level events and passes them to the behavioural layer where high-level actions are recognised and modified prior to their re-activation (see Fig. 1). The technologies reported in this paper have been used to support interaction in VR Art installations. We illustrate the paper discussion by presenting the first prototype of a VR installation conceived by one of the authors: Mark Palmer. Gyre and Gimble is a VR artistic project based upon Carroll’s Alice stories. Although Gyre and Gimble draws upon the Alice stories the intention was never to reproduce their narratives, but to explore the disruption to perception offered within them. In fact Carroll’s stories provide a natural starting point for anybody interested in the consequences of logic and the creation of alternative realities. The way that we often encounter this in his stories is through the mixing, collision and invention of games as well as the transformation of their rules. A playfulness that he also directs towards language presenting us with paradoxes1 arising out of the situations and conversation that Alice finds herself. This is why his books are always far more than just the presentation of a plot, they are events that unfold involving the reader in their own logic. Reacting to this interaction becomes the primary driver of Gyre and Gimble deliberately distancing itself from narrative. Rather than becoming a vehicle for retelling portions of the Alice story, like the stories themselves it becomes an ‘event’ and the occasion that involves users in this disruptive process. Here the challenge was to make a technology based upon gaming as effective as Carroll’s creative subversion of games. The joint decision to draw from the scene in Through the Looking Glass, where Alice discovers that, try as she might to look at things in a shop they evade her gaze, was to provide the opportunity to use spectacle itself as a means of interaction. Using a combination of a viewer’s distance and centre of focus it became possible to employ attention as a means of interaction. The collision of objects that then occurred as a result of an object’s desire to escape constitute, from the system perspective, the starting point for the computation of chain of events (and consequences). The brief’s environment is a 3D world reflecting the aesthetics of the original Tenniels’ illustration (using 3D objects with non-photorealistic rendering,). The user, evolving in the environment as Alice, in first person mode is a witness to various objects behaviour, which she can also affect by her presence. One room surrounded by several cabinets and a table represents the Gyre and Gimble world. The place is composed of ten different types of objects, a total of 90 interactive objects such as candles, holder, clock, and book, dispersed on shelves, cabinet and table (see screen shot below on Fig. 2). 1
“When we were little,” the Mock Turtle went on at last, more calmly, though still sobbing now and then, “We went to school in the sea. The master was an old Turtle – we used to call him Tortoise-”“Why did you call him Tortoise, if he wasn’t one?” Alice asked. “We called him Tortoise because he taught us”, said the Mock Turtle angrily. “Really you are very dull!” p.83 Alice’s Adventures in Wonderland and Through the Looking Glass, Penguin Books, London 1998.
AI-Mediated Interaction in Virtual Reality Art
77
Fig. 2. The "Gyre and Gimble" Environment2 and Example of Interactive Objects. Nonphotorealistic rendering has been preferred, inspired from the original Tenniel’s illustrations.
3 User Interactions The User interacts with the VR Art installation through navigation but most essentially via their attitude towards the world objects. First, the user triggers object spontaneous movement by the simple means of her gaze3, reflecting the most natural form of interaction with the scene. World objects are associated a “native” behaviour, by which they will evade the user’s gaze, and escape towards other objects, depending on the object categories. As a result, the user witnesses a stream of object behaviours, prompted by his interaction but whose logic is not directly accessible. The user’s attitude is actually reflected on those behaviours through complex, global mechanisms involving degrees of “perturbation” and “surprise”, as detailed in the following section. The spontaneous objects motion is provoking collisions between objects, which are the starting point for the generation of an event chain by the AI module and whose perception by the user constitutes a central aspect of her interactive experience. As explained in the next sections, the amplitude of the transformations computed are based on semantic properties and analogies between objects and depends on how the user engages with the environment. This type of computation can provide a measure of concepts such as “surprise” [8].
4 Event Chain Generation Principles The two main principles for the creation of chains of events responding to user interaction are the modification of the default consequences of certain events and/or the addition of new effects. Our system uses an ontology of actions, which explicitly represent a set of possible actions taking place in the virtual world. From these action 2 3
The visual content presented here represents a first version of the environment. Actually measured from the head vector, with fair approximation considering the average distance from the screens.
78
J.-l. Lugrin et al.
representations, it is possible to generate alternative consequences using semantic properties of the action descriptions. These action representations are inspired from planning formalisms (namely the SIPE formalism [10]) in the fact that they associate within the same representation an intervention (the “causes”) and its consequences (the “effects”). We have thus termed these representations CE (for Cause-Effect). This Cause-Effect formalism represents a fundamental layer upon which the above transformations can be exerted, resulting in various chains of events. As was previously described, CE representations are continuously produced by “parsing” the lowlevel system events describing collisions, etc. into CE, using the semantic properties of the objects taking part in those actions. Fig. 3 shows the CE representation for a projecting action Project-Object (?obj1, ?obj2). Its “trigger” part corresponds to the event initiating the action (Hit (?obj1, ?obj2)) and its “effect” part to its effect (that fact that the object is projected upon the collision impact). The “condition” field corresponds to physical properties characterising objects taking part in that action (i.e. moveable, non-breakable), which are used when instantiating a CE representation.
Fig. 3. The Cause-and-Effect (CE) Event and Semantic Object Representation
The generation an event chain is based on the real-time transformation of CE instances while their effects are being “frozen”. From a formal perspective these transformations should modify the contents of a CE instance to produce new types of events upon reactivation of the CE’s effects. In order to do so, it relies on several features, which correspond to i) a semantics for the CE representation and the world objects, ii) specific procedures for transformation which alter the contents of the CE instance and iii) a control mechanism for the set of transformations over the CE. The first aspect derives from an underlying ontology, in which objects are classified according to different dimensions characterising their physical properties (movable, breakable, flammable, etc.). These semantic dimensions in turn determine the applicability of some actions (as implemented in the “condition” part of a CE). For instance, only breakable objects can be the targets of shatter-on-impact CEs, or only certain types of objects can be affected by a tilt-object CE. The second aspect is implemented through the notion of Macro-Operator or MOp. If we consider that action representations will have to be transformed (i.e. their internal variables will have to be substituted, or additional elements added to their “effect” field) we need specific procedures for performing such expression substitutions. Finally, a
AI-Mediated Interaction in Virtual Reality Art
79
control mechanism should determine which CE should be modified, and which modification is most appropriate from a semantic perspective. The control mechanism will select a set (or a sequence) of MOp to be applied on the candidate CE, and the criteria for such selection must be semantic ones. As described in a forthcoming section, this control mechanism is based on a heuristic search algorithm [1]. MOps are described in terms of the transformations classes they operate on a CE’s effects. Examples of MOp classes include: •
Change-Object, which substitutes new objects to those originally affected by the CE’s effects.
•
change-effect, which modify the effects (consequences) of a CE
• •
Propagate-effect extend the CE’s effects to other semantically compatible objects in the environment Link-Effect, which relate one CE’s effect to another one’s
MOps are also knowledge operators, in the sense that they evaluate the semantic consistency of a transformation, e.g. whether the substituted object or effect satisfies the semantic restrictions of the original action. The AI module applies sequences of MOp to generate the space of possible transformations from the CE under consideration. The number of transformations considered depends of the nature of the MOP. For instance Change-Effect operators, will generate five candidate transformations if the object supportes five different states. Fig. 4 below illustrates the application of this Macro-operator performed by the AI Module on a given CE. In the example considered, the book hits the candle while trying to escape the user’s gaze. Upon
Fig. 4. A Macro-Operator Application Example
80
J.-l. Lugrin et al.
impact the candle should normally be projected forward. However, the AI module intercepts the action and modifies its default consequences using MOp (Fig. 4). In our example the Change-Effect MOp replaces the consequences of the collision with new ones, provided certain semantic criteria are satisfied (e.g. they should generate a state compatible with the action’s object). 4.1 Event Processing The basic principle for the dynamic creation of event chains relies on degrees of analogy between recognised events and modified ones. The modifications are derived from an initial event, using a set of Macro-Operators, reinforced by spatial distribution considerations. Thus, the AI module generates a set of possible transformation from which it selects the “most appropriate” one, in terms of semantic compatibility and spatial constraints. The system can choose one or multiple transformation according to the “level of disruption” chosen. The latter concept is one which determines the nature of user experience. The overall algorithm explores the space of transformations using heuristic search. The three basic steps of this search are: Modification Generation, Evaluation and Selection. 1 - Generation: The first operation consists of generating the list of possible transformation from the given interaction. The process of MOp-Application generates a number of modified CEs which extend the search space for the selection algorithm. 2 - Evaluation: Each modified CE is associated a value reflecting its degree of analogy with the CE initially recognised. For those modifications involving objects other than the default one, their spatial distribution score is also computed. These scores are aggregated into a degree of plausibility for each transformation, which is represented by a value normalised between zero and one. This value is used as a heuristic function is the determination of the best transformation. 3 - Selection: During this operation, the algorithm uses the heuristic value computed in the previous step to select the most appropriate transformation, based on a pre-set “level of disruption” (Fig. 5). The level of disruption can be considered as a heuristic threshold used to determine the extent of transformations, hence affecting the user experience.
The combination of knowledge-based operators (MOp) and heuristic search based on analogy provides an original approach that allows a systematic exploration of a transformation space. The user studies carried out with 60 subjects have mainly validated the ability of our system to induce causal perception for the alternative event chain (70 % of causal relations established from alternative effect generated). Conjointly, the use of analogy measures can be related to the user experience, as they encompass notions such as “naturalness”, “surprise”, etc.
5 User Influences and Experiences As previously explained, the Level of Disruption corresponds to the search threshold for the use of heuristic values. According to the value of this threshold the transformation produced goes from the more natural (0) to the more “surprising” (1). Our system discretises this value into five disruption levels: NULL / LOW / MEDIUM / HIGH /
AI-Mediated Interaction in Virtual Reality Art
81
VERY-HIGH. In the Gyre and Gymble world, the Level of Disruption is dynamically updated in relation to the User-Objects-Proximity and User-Activity
•
The User-Object-Proximity is weighted between [0-1] and corresponds to the average distance of the user to objects present in this field of view. This metric reflects a level of engagement of the users with the objects which, depending on the artistic brief, can be interpreted as interest or threat.
•
The User-Activity represents an appreciation of the user frequency movement, expressed by a weight between [0-1] (a value of 0 meaning that the user is immobile).
The level of disruption is then frequently updated using a simple matrix (see fig. 5). Increasing or decreasing the plausibility and amplitude of the transformations create different user experiences in terms of emotions reflected in, and by, the world itself. The user thus indirectly influences the transformation amplitude through values for his behaviour. This constitutes another example of the generation of more sophisticated user experience through AI technologies.
Fig. 5. User Behaviour and Current Level of Disruption
The system provides simple mechanisms to achieve a vast number of alternative behaviours based on simple modification of the level of disruption. As depicted in figure 6, in the Gyre a Gimble environment, a simple interaction, such as a candle being hit by a moving book, can result in a large number of possible consequences even when only considering two MOp (Change-Effect and Propagate-Effect). A low value for the level of disruption parameter (close to 0.25) tends to result in minor changes. Indeed, they are often related to the propagation of a normal consequence to spatially close and/or same-type objects. For instance, the book-candle collision will also project one of the closest similar candles. However, a medium level of disruption (around 0.5) usually extends or substitutes default consequences to different objects, as when the book is projected with the candles around, instead of the original candle (Fig. 6 label 2). Higher levels of disruption (close to 1.0) affect the type of effects generated and the entire population of objects situated in the user’s field of view. At this level, the consequence of an interaction becomes hardly predict-
82
J.-l. Lugrin et al.
able, as it depends of the local context of the environment (i.e.: the type, state and distance of objects surrounding the initial event). Here, such a level triggers the opening of the book while some candles start burning or tilting (Fig. 6 label 3). The essential advantages of this approach consist in being able to control the consequences of user interaction, at different levels, using concepts that can be related to artistic intentions. Most importantly, these principles also support generative aspects, where the system enriches the creative process. The authoring process leading to generation of non deterministic behaviour is composed of three main steps. In the first step, the artist defines deterministic behaviour using our CE action-formalism. Then, he determines the nature of transformation possible by proposing a combination of MOp. Lastly, the artist relates the transformation amplitude to user behaviour by adjusting the level of disruption matrix (as previously shown in Fig. 5).
Fig. 6. Level of Disruption and User Experiences
6 Conclusion Traditional interactive systems rely on direct associations between interaction events and their scripted consequences. This has some limitations for Digital Arts application, forcing the specification of all low-level events when implementing and artistic brief. Such an approach has also limited flexibility when it comes to eliciting more complex user experience, such as reacting to the user’s global behaviour. An AI perspective brings two major advances. The first one consists in an explicit representation layer for high-level actions which supports principled modifications of actions’ consequences, these principles being derived from artistic intentions. Another advantage is the use of the generative properties of the AI approach to enrich the user experience, thus simplifying the authoring process. As VR Art develops [4][9], the requirements on advanced interactivity will become more demanding, and mediating interaction through AI representations seems a promising research direction, as suggested by several installations we have been supporting [3].
AI-Mediated Interaction in Virtual Reality Art
83
Acknowledgments This research has been funded in part by the European Commission through the ALTERNE (IST-38575) project. Marc Le Renard and Jeffrey Jacobson are thanked for their participation on the SAS Cube and CaveUT developments.
References 1. Bonet, B. and Geffner, H.. Planning as heuristic search. Artificial Intelligence: Special Issue on Heuristic Search, (2001),129:5-33 2. Cavazza, M., Lugrin, JL., Crooks, S., Nandi, A., Palmer, M., , and Le Renard, M.: Causality and Virtual Reality Art, Fifth ACM International Conference on Creativity and Cognition, Goldsmiths College, London, (2005). 3. Cavazza, M., Lugrin, JL., Hartley, S., Libardi,P., Barnes, M., Le Bras, M., Le Renard, M., Bec, L., and Nandi, A.: New Ways of Worldmaking: the Alterne Platform for VR Art. ACM International Conference on Multimedia tools, end-system and applications, New York, (2004). 4. Grau, O.,Virtual Art : From Illusion to Immersion ,Cambridge (Massachussets), MIT press. ISBN: 0262072416 (2002) 5. Jacobson, J. and Hwang, Z. Unreal Tournament for Immersive Interactive Theater. Communications of the ACM, Vol. 45, 1, pp. 39-42, 2002 6. Jacobson, J., Le Renard, M., Lugrin, JL., and Cavazza M.: The CaveUT System: Immersive Entertainment Based on a Game Engine. Second ACM Conference on Advances in Computing Entertainment, ACE 2005, (2005). 7. Lewis, M and Jacobson, J.: Games Engines in Scientific Research. Communications of ACM, Vol. 45, No. I, January 2002, (2002) 27-31. 8. Macedo, L, Cardoso, A.: Modelling Forms of Surprise in an Artificial Agent, in Proc. of the 23rd Annual Conference of the Cognitive Science Society, 23rd Annual Conference of the Cognitive Science Society, Edinburgh, (2001) 588-593 9. Moser, M.A. (Ed.), Immersed in Technology: Art and Virtual Environments, Cambridge (Massachussets), MIT Press, (1996). 10. Wilkins, D. E.: Causal reasoning in planning. Computational Intelligence, vol. 4, no. 4, (1988) 373-380.
Laughter Abounds in the Mouths of Computers: Investigations in Automatic Humor Recognition Rada Mihalcea1 and Carlo Strapparava2 1
University of North Texas, Department of Computer Science, Denton, Texas, 76203, USA
[email protected] 2 ITC-irst, Istituto per la Ricerca Scientifica e Tecnologica, I-38050, Trento, Italy
[email protected] Abstract. Humor is an aspect of human behavior considered essential for inter-personal communication. Despite this fact, research in humancomputer interaction has almost completely neglected aspects concerned with the automatic recognition or generation of humor. In this paper, we investigate the problem of humor recognition, and bring empirical evidence that computational approaches can be successfully applied to this task. Through experiments performed on very large data sets, we show that automatic classification techniques can be effectively used to distinguish between humorous and non-humorous texts, with significant improvements observed over apriori known baselines.
1
Introduction
The creative genres of natural language have been traditionally considered outside the scope of any computational modeling. In particular humor, because of its puzzling nature, has received little attention from computational linguists. However, given the importance of humor in our everyday life, and the increasing importance of computers in our work and entertainment, we believe that studies related to computational humor will become increasingly important in fields such as human-computer interaction, intelligent interactive entertainment, and computer-assisted education. Previous work in computational humor has focused mainly on the task of humor generation [1,2], and very few attempts have been made to develop systems for automatic humor recognition [3]. This is not surprising, since, from a computational perspective, humor recognition appears to be significantly more subtle and difficult than humor generation. In this paper, we explore the applicability of computational approaches to the recognition of verbally expressed humor. In particular, we investigate whether automatic classification techniques are a viable approach to distinguish between humorous and non-humorous text, and we bring empirical evidence in support of this hypothesis through experiments performed on very large data sets. Since a deep comprehension of humor in all of its aspects is probably too ambitious and beyond the existing computational capabilities, we chose to restrict our investigation to the type of humor found in one-liners. A one-liner is a M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 84–93, 2005. c Springer-Verlag Berlin Heidelberg 2005
Laughter Abounds in the Mouths of Computers
85
short sentence with comic effects and an interesting linguistic structure: simple syntax, deliberate use of rhetoric devices (e.g. alliteration, rhyme), and frequent use of creative language constructions meant to attract the readers attention. While longer jokes can have a relatively complex narrative structure, a one-liner must produce the humorous effect “in one shot”, with very few words. These characteristics make this type of humor particularly suitable for use in an automatic learning setting, as the humor-producing features are guaranteed to be present in the first (and only) sentence. We attempt to formulate the humor-recognition problem as a traditional classification task, and feed positive (humorous) and negative (non-humorous) examples to an automatic classifier. The humorous data set consists of one-liners collected from the Web using an automatic bootstrapping process. The nonhumorous data is selected such that it is structurally and stylistically similar to the one-liners. Specifically, we use four different negative data sets: (1) Reuters news titles; (2) proverbs; (3) sentences from the British National Corpus (BNC); and (4) commonsense statements from the Open Mind Common Sense (OMCS) corpus. The classification results are encouraging, with accuracy figures ranging from 79.15% (One-liners/BNC) to 96.95% (One-liners/Reuters). Regardless of the non-humorous data set playing the role of negative examples, the performance of the automatically learned humor-recognizer is always significantly better than apriori known baselines. The remainder of the paper is organized as follows. We first describe the humorous and non-humorous data sets, and provide details on the Web-based bootstrapping process used to build a very large collection of one-liners. We then show experimental results obtained on these data sets using several heuristics and two different text classifiers. Finally, we conclude with a discussion and directions for future work.
2
Humorous and Non-humorous Data Sets
To test our hypothesis that automatic classification techniques represent a viable approach to humor recognition, we needed in the first place a data set consisting of both humorous (positive) and non-humorous (negative) examples. Such data sets can be used to automatically learn computational models for humor recognition, and at the same time evaluate the performance of such models. 2.1
Humorous Data
Large amounts of training data have the potential of improving the accuracy of the learning process, and at the same time provide insights into how increasingly larger data sets can affect the classification precision. The manual construction of a very large one-liner data set may be however problematic, since most Web sites or mailing lists that make available such jokes do not usually list more than 50–100 one-liners. To tackle this problem, we implemented a Web-based bootstrapping algorithm able to automatically collect a large number of oneliners starting with a short seed list, consisting of a few one-liners manually identified.
86
R. Mihalcea and C. Strapparava
Table 1. Sample examples of one-liners, Reuters titles, proverbs, OMC and BNC sentences One-liners Take my advice; I don’t use it anyway. I get enough exercise just pushing my luck. Beauty is in the eye of the beer holder. Reuters titles Proverbs Trocadero expects tripling of revenues. Creativity is more important than knowledge. Silver fixes at two-month high, but gold lags. Beauty is in the eye of the beholder. Oil prices slip as refiners shop for bargains. I believe no tales from an enemy’s tongue. OMCS sentences BNC sentences Humans generally want to eat at least once a day. They were like spirits, and I loved them. A file is used for keeping documents. I wonder if there is some contradiction here. A present is a gift, something you give to someone. The train arrives three minutes early.
Starting with the seed set, the algorithm automatically identifies a list of webpages that include at least one of the seed one-liners, via a simple search performed using a Web search engine. Next, the webpages found in this way are HTML parsed, and additional one-liners are automatically identified and added to the seed set. The process is repeated until enough one-liners are collected. See [4] for details about this bootstrapping algorithm. Two iterations of the bootstrapping process, started with a small seed set of ten one-liners, resulted in a large set of about 24,000 one-liners. After removing the duplicates using a measure of string similarity based on the longest common subsequence metric, we were left with a final set of approximately 16,000 one-liners, which are used in the humor-recognition experiments. The one-liners humor style is illustrated in Table 1. Manual verification of a randomly selected sample of 200 one-liners indicates an average of 9% potential noise in the data set, which is within reasonable limits, as it does not appear to significantly impact the quality of the learning.
2.2
Non-humorous Data
To construct the set of negative examples, we tried to identify collections of sentences that were non-humorous, but similar in structure and composition to the one-liners. Structural similarity was enforced by requiring that each example in the non-humorous data set follows the same length restriction as the one-liners: one sentence with an average length of 10–15 words. Compositional similarity is sought by trying to identify examples similar to the one-liners with respect to their creativity and intent. We tested four different sets of negative examples, with three examples from each data set illustrated in Table 1: 1. Reuters titles, extracted from news articles published in the Reuters newswire over a period of one year (8/20/1996 – 8/19/1997) [5]. The titles consist of short sentences with simple syntax, and are often phrased to catch the readers attention (an effect similar to the one rendered by one-liners).
Laughter Abounds in the Mouths of Computers
87
2. Proverbs extracted from an online proverb collection. Proverbs are sayings that transmit, usually in one short sentence, important facts or experiences that are considered true by many people. Their property of being condensed, but memorable sayings make them very similar to the one-liners. In fact, some one-liners attempt to reproduce proverbs, with a comic effect, as in e.g. “Beauty is in the eye of the beer holder”, derived from “Beauty is in the eye of the beholder”. 3. Open Mind Common Sense (OMCS) sentences. OMCS is a collection of about 800,000 commonsense assertions in English as contributed by volunteers over the Web. It consists mostly of simple single sentences, which tend to be explanations and assertions similar to glosses of a dictionary, but phrased in a more common language. For example, the collection includes such assertions as “keys are used to unlock doors”, and “pressing a typewriter key makes a letter”. Since the comic effect of jokes is often based on statements that break our commonsensic understanding of the world, we believe that such commonsense sentences can make an interesting collection of “negative” examples for humor recognition. For details on the OMCS data and how it has been collected, see [6]. From this repository we use the first 16,000 sentences1 . 4. British National Corpus (BNC) sentences, extracted from BNC – a balanced corpus covering different styles, genres and domains. The sentences were selected such that they were similar in content with the one-liners: we used the Smart system2 to identify the BNC sentence most similar to each of the 16,000 one-liners. Although the BNC sentences have typically no added creativity and no specific intent, we decided to add this set of negative examples to our experimental setting, in order to observe the level of difficulty of a humor-recognition task when performed with respect to simple text.
3
Automatic Humor Recognition
We experiment with automatic classification techniques using: (a) heuristics based on humor-specific stylistic features (alliteration, antonymy, slang); (b) content-based features, within a learning framework formulated as a typical text classification task; and (c) combined stylistic and content-based features, integrated in a stacked machine learning framework. 3.1
Humor-Specific Stylistic Features
Linguistic theories of humor (e.g. [7]) have suggested many stylistic features that characterize humorous texts. We tried to identify a set of features that were both significant and feasible to implement using existing machine readable resources. Specifically, we focus on alliteration, antonymy, and adult slang, previously suggested as potentially good indicators of humor [8,9]. 1
2
The first sentences in this corpus are considered to be “cleaner”, as they were contributed by trusted users (Push Singh, p.c.). Available at ftp.cs.cornell.edu/pub/smart.
88
R. Mihalcea and C. Strapparava
Alliteration. Some studies on humor appreciation [8] show that structural and phonetic properties of jokes are at least as important as their content. In fact oneliners often rely on the reader awareness of attention-catching sounds, through linguistic phenomena such as alliteration, word repetition and rhyme, which produce a comic effect even if the jokes are not necessarily meant to be read aloud. Note that similar rhetorical devices play an important role in wordplay jokes, and are often used in newspaper headlines and in advertisement. The following one-liners are examples of jokes that include alliteration chains: Veni, Vidi, Visa: I came, I saw, I did a little shopping. Infants don’t enjoy infancy like adults do adultery.
To extract this feature, we identify and count the number of alliteration/rhyme chains in each example in our data set. The chains are automatically extracted using an index created on top of the CMU pronunciation dictionary3 . Antonymy. Humor often relies on some type of incongruity, opposition or other forms of apparent contradiction. While an accurate identification of all these properties is probably difficult to accomplish, it is relatively easy to identify the presence of antonyms in a sentence. For instance, the comic effect produced by the following one-liners is partly due to the presence of antonyms: A clean desk is a sign of a cluttered desk drawer. Always try to be modest and be proud of it!
The lexical resource we use to identify antonyms is WordNet [10], and in particular the antonymy relation among nouns, verbs, adjectives and adverbs. For adjectives we also consider an indirect antonymy via the similar-to relation among adjective synsets. Despite the relatively large number of antonymy relations defined in WordNet, its coverage is far from complete, and thus the antonymy feature cannot always be identified. A deeper semantic analysis of the text, such as word sense or domain disambiguation, could probably help detecting other types of semantic opposition, and we plan to exploit these techniques in future work. Adult slang. Humor based on adult slang is very popular. Therefore, a possible feature for humor-recognition is the detection of sexual-oriented lexicon in the sentence. The following represent examples of one-liners that include such slang: The sex was so good that even the neighbors had a cigarette. Artificial Insemination: procreation without recreation.
To form a lexicon required for the identification of this feature, we extract from WordNet Domains4 all the synsets labeled with the domain Sexuality. The list is further processed by removing all words with high polysemy (≥ 4). Next, we check for the presence of the words in this lexicon in each sentence in the corpus, and annotate them accordingly. Note that, as in the case of antonymy, 3 4
Available at http://www.speech.cs.cmu.edu/cgi-bin/cmudict. WordNet Domains assigns each synset in WordNet with one or more “domain” labels, such as Sport, Medicine, Economy. See http://wndomains.itc.it.
Laughter Abounds in the Mouths of Computers
89
WordNet coverage is not complete, and the adult slang feature cannot always be identified. Finally, in some cases, all three features (alliteration, antonymy, adult slang) are present in the same sentence, as for instance the following one-liner: Behind every greatal manant is a greatal womanant , and behind every greatal womanant is some guy staring at her behindsl !
3.2
Content-Based Learning
In addition to stylistic features, we also experimented with content-based features, through experiments where the humor-recognition task is formulated as a traditional text classification problem. Specifically, we compare results obtained with two frequently used text classifiers, Na¨ıve Bayes and Support Vector Machines, selected based on their performance in previously reported work, and for their diversity of learning methodologies. Na¨ıve Bayes. The main idea in a Na¨ıve Bayes text classifier is to estimate the probability of a category given a document using joint probabilities of words and documents. Na¨ıve Bayes classifiers assume word independence, but despite this simplification, they were shown to perform well on text classification. While there are several versions of Na¨ıve Bayes classifiers (multinomial and multivariate Bernoulli), we use the multinomial model, shown to be more effective [11]. Support Vector Machines. Support Vector Machines (SVM) are binary classifiers that seek to find the hyperplane that best separates a set of positive examples from a set of negative examples, with maximum margin. Applications of SVM classifiers to text categorization led to some of the best results reported in the literature [12].
4
Experimental Results
Several experiments were conducted to gain insights into various aspects related to an automatic humor recognition task: classification accuracy using stylistic and content-based features, learning rates, impact of the type of negative data, impact of the classification methodology. All evaluations are performed using stratified ten-fold cross validations, for accurate estimates. The baseline for all the experiments is 50%, which represents the classification accuracy obtained if a label of “humorous” (or “non-humorous”) would be assigned by default to all the examples in the data set. 4.1
Heuristics Using Humor-Specific Features
In a first set of experiments, we evaluated the classification accuracy using stylistic humor-specific features: alliteration, antonymy, and adult slang. These are numerical features that act as heuristics, and the only parameter required for their application is a threshold indicating the minimum value admitted for a statement to be classified as humorous (or non-humorous). These thresholds are learned
90
R. Mihalcea and C. Strapparava Classification learning curves 100
90
90
Classification accuracy (%)
Classification accuracy (%)
Classification learning curves 100
80 70 60 50
80 70 60 50
Naive Bayes SVM
40 0
20
40 60 Fraction of data (%)
80
Naive Bayes SVM
40 100
0
20
40 60 Fraction of data (%)
(a)
100
(b)
Classification learning curves
Classification learning curves
100
100
90
90
Classification accuracy (%)
Classification accuracy (%)
80
80 70 60 50
80 70 60 50
Naive Bayes SVM
40 0
20
40 60 Fraction of data (%)
80
Naive Bayes SVM
40 100
0
20
(c)
40 60 Fraction of data (%)
80
100
(d)
Fig. 1. Learning curves for humor-recognition using text classification techniques, with respect to four different sets of negative examples: (a) Reuters; (b) BNC; (c) Proverbs; (d) OMCS
Table 2. Humor-recognition accuracy using alliteration, antonymy, and adult slang One-liners One-liners One-liners Heuristic Reuters BNC Proverbs Alliteration 74.31% 59.34% 53.30% Antonymy 55.65% 51.40% 50.51% Adult slang 52.74% 52.39% 50.74% All 76.73% 60.63% 53.71%
One-liners OMCS 55.57% 51.84% 51.34% 56.16%
automatically using a decision tree applied on a small subset of humorous/nonhumorous examples (1000 examples). The evaluation is performed on the remaining 15,000 examples, with results shown in Table 25 . Considering the fact that these features represent stylistic indicators, the style of Reuters titles turns out to be the most different with respect to one5
We also experimented with decision trees learned from a larger number of examples, but the results were similar, which confirms our hypothesis that these features are heuristics, rather than learnable properties that improve their accuracy with additional training data.
Laughter Abounds in the Mouths of Computers
91
liners, while the style of proverbs is the most similar. Note that for all data sets the alliteration feature appears to be the most useful indicator of humor, which is in agreement with previous linguistic findings [8]. 4.2
Text Classification with Content Features
The second set of experiments was concerned with the evaluation of contentbased features for humor recognition. Table 3 shows results obtained using the four different sets of negative examples, with the Na¨ıve Bayes and SVM classifiers. Learning curves are plotted in Figure 1. Table 3. Humor-recognition accuracy using Na¨ıve Bayes and SVM text classifiers One-liners One-liners One-liners One-liners Classifier Reuters BNC Proverbs OMCS Na¨ıve Bayes 96.67% 73.22% 84.81% 82.39% SVM 96.09% 77.51% 84.48% 81.86%
Once again, the content of Reuters titles appears to be the most different with respect to one-liners, while the BNC sentences represent the most similar data set. This suggests that joke content tends to be very similar to regular text, although a reasonably accurate distinction can still be made using text classification techniques. Interestingly, proverbs can be distinguished from oneliners using content-based features, which indicates that despite their stylistic similarity (see Table 2), proverbs and one-liners deal with different topics. 4.3
Combining Stylistic and Content Features
Encouraged by the results obtained in the first two experiments, we designed a third experiment that attempts to jointly exploit stylistic and content features for humor recognition. The feature combination is performed using a stacked learner, which takes the output of the text classifier, joins it with the three humor-specific features (alliteration, antonymy, adult slang), and feeds the newly created feature vectors to a machine learning tool. Given the relatively large gap between the performance achieved with content-based features (text classification) and stylistic features (humor-specific heuristics), we decided to do the meta-learning using a rule based learner, so that low-performance features are not eliminated in the favor of the more accurate ones. We use the Timbl memory based learner [13], and evaluate the classification using a stratified ten-fold cross validation. Table 4 shows the results obtained for the four data sets. Combining classifiers results in a statistically significant improvement (p < 0.0005, paired t-test) with respect to the best individual classifier for the Oneliners/Reuters and One-liners/BNC data sets, with relative error rate reductions of 8.9% and 7.3% respectively. No improvement is observed for the Oneliners/Proverbs and One-liners/OMCS data sets, which is not surprising since, as shown in Table 2, proverbs and commonsense statements cannot be clearly differentiated using stylistic features from the one-liners, and thus the addition of these features to content-based features is not likely to result in an improvement.
92
R. Mihalcea and C. Strapparava
Table 4. Humor-recognition accuracy for combined learning based on stylistic and content features One-liners One-liners One-liners One-liners Reuters BNC Proverbs OMCS 96.95% 79.15% 84.82% 82.37%
4.4
Discussion
The results obtained in the automatic classification experiments reveal the fact that computational approaches represent a viable solution for the task of humorrecognition, and good performance can be achieved using classification techniques based on stylistic and content features. Despite our initial intuition that one-liners are most similar to other creative texts (e.g. Reuters titles, or the sometimes almost identical proverbs), and thus the learning task would be more difficult in relation to these data sets, comparative experimental results show that in fact it is more difficult to distinguish humor with respect to regular text (e.g. BNC sentences). Note however that even in this case the combined classifier leads to a classification accuracy that improves significantly over the previously known baseline. In addition to classification accuracy, we were also interested in the variation of classification performance with respect to data size, which is an aspect particularly relevant for directing future research. Depending on the shape of the learning curves, one could decide to concentrate future work either on the acquisition of larger data sets, or toward the identification of more sophisticated features. Figure 1 shows that regardless of the type of negative data or classification methodology, there is significant learning only until about 60% of the data (i.e. about 10,000 positive examples, and the same number of negative examples). The rather steep ascent of the curve, especially in the first part of the learning, suggests that humorous and non-humorous texts represent well distinguishable types of data. An interesting effect can be noticed toward the end of the learning, where for both classifiers the curve becomes completely flat (Oneliners/Reuters, One-liners/Proverbs, One-liners/OMCS), or it even has a slight drop (One-liners/BNC). This is probably due to the presence of noise in the data set, which starts to become visible for very large data sets6 . This plateau is also suggesting that more data is not likely to help improve the quality of an automatic humor-recognizer, and more sophisticated features are probably required.
5
Conclusion
In this paper, we showed that automatic classification techniques can be successfully applied to the task of humor-recognition. Experimental results obtained on very large data sets showed that computational approaches can be efficiently 6
We also like to think of this behavior as if the computer is losing its sense of humor after an overwhelming number of jokes, in a way similar to humans when they get bored and stop appreciating humor after hearing too many jokes.
Laughter Abounds in the Mouths of Computers
93
used to distinguish between humorous and non-humorous texts, with significant improvements observed over apriori known baselines. To our knowledge, this is the first result of this kind reported in the literature, as we are not aware of any previous work investigating the interaction between humor and techniques for automatic classification. Finally, through the analysis of learning curves plotting the classification performance with respect to data size, we showed that the accuracy of the automatic humor-recognizer stops improving after a certain number of examples. Given that automatic humor-recognition is a rather understudied problem, we believe that this is an important result, as it provides insights into potentially productive directions for future work. The flattened shape of the curves toward the end of the learning process suggests that rather than focusing on gathering more data, future work should concentrate on identifying more sophisticated humor-specific features, e.g. semantic oppositions, ambiguity, and others. We plan to address these aspects in future work.
References 1. O. Stock and C. Strapparava. Getting serious about the development of computational humour. In Proceedings of the 8th International Joint Conference on Artificial Intelligence (IJCAI-03), Acapulco, Mexico, August 2003. 2. K. Binsted and G. Ritchie. Computational rules for punning riddles. Humor, 10(1), 1997. 3. J. Taylor and L. Mazlack. Computationally recognizing wordplay in jokes. In Proceedings of CogSci 2004, Chicago, August 2004. 4. R. Mihalcea and C. Strapparava. Bootstrapping for fun: Web-based construction of large data sets for humor recognition. In Proceedings of the Workshop on Negotiation, Behaviour and Language (FINEXIN 2005), Ottawa, Canada, 2005. 5. D. Lewis, Y. Yang, T. Rose, and F. Li. RCV1: A new benchmark collection for text categorization research. The Journal of Machine Learning Research, 5:361– 397, December 2004. 6. P. Singh. The public acquisition of commonsense knowledge. In Proceedings of AAAI Spring Symposium: Acquiring (and Using) Linguistic (and World) Knowledge for Information Access., Palo Alto, CA, 2002. 7. S. Attardo. Linguistic Theory of Humor. Mouton de Gruyter, Berlin, 1994. 8. W. Ruch. Computers with a personality? lessons to be learned from studies of the psychology of humor. In Proceedings of the The April Fools Day Workshop on Computational Humour, 2002. 9. C. Bucaria. Lexical and syntactic ambiguity as a source of humor. Humor, 17(3), 2004. 10. G. Miller. Wordnet: A lexical database. Communications of the ACM, 38(11):39– 41, 1995. 11. A. McCallum and K. Nigam. A comparison of event models for Naive Bayes text classification. In Proceedings of AAAI-98 Workshop on Learning for Text Categorization, 1998. 12. T. Joachims. Text categorization with Support Vector Machines: learning with many relevant features. In Proceedings of the European Conference on Machine Learning, 1998. 13. W. Daelemans, J. Zavrel, K. van der Sloot, and A. van den Bosch. Timbl: Tilburg memory based learner, version 4.0, reference guide. Technical report, University of Antwerp, 2001.
AmbientBrowser: Web Browser for Everyday Enrichment Mitsuru Minakuchi1 , Satoshi Nakamura1 , and Katsumi Tanaka1,2 1
National Institute of Information and Communications Technology, Japan {mmina, gon}@nict.go.jp 2 Kyoto University
[email protected] Abstract. We propose an AmbientBrowser system that helps people acquire knowledge during everyday activities. It continuously searches Web pages using both system-defined and user-defined keywords, and displays summarized text obtained from pages found by searches. The system’s sensors detect users’ and environmental conditions and control the system’s behavior such as knowledge selection or a style of presentation. Thus, the user can encounter a wide variety of knowledge without active operations. A pilot study showed that peripherally displayed knowledge could be read and could engage a user’s interest. We implemented the system using a random information retrieval mechanism, an automatic kinetic typography composer, and easy methods of interaction using sensors.
1
Introduction
In the words of Bertrand Russell, “There is much pleasure to be gained from useless knowledge.” Undoubtedly, people enjoy acquiring knowledge. People read books or watch TV to find out about trivia, although these facts are of no practical use in everyday life. They may also compete in quizzes or parade their knowledge in conversation. This desire for knowledge can be considered to correspond to the needs of esteem and being described in Maslow’s hierarchy of needs [1]. Thus, we believe that intellectual stimulation is essential for human well being and new knowledge is a source of pleasure. There are various media available to convey knowledge, for example, books, magazines, newspapers, posters, TV programs, and Web pages and people can select their preferred media. We roughly classify the selection of media for knowledge acquisition into two types: pull and push types. In the pull-type pattern, people have a clear intention and specific targets. Portable media, such as books, magazines, and newspapers, are suited to this type because people are explicitly motivated to take these media with them and read them. In the push-type pattern, people are fed knowledge without having formed a specific intention. For example, they often encounter and obtain information M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 94–103, 2005. c Springer-Verlag Berlin Heidelberg 2005
AmbientBrowser: Web Browser for Everyday Enrichment
95
from posters, billboards, a TV left on, etc. Stationary media play an important role in this pattern because they are continuously available and present information constantly. We consider the push-type is more suitable for daily acquisition of knowledge because people are less likely to actively seek knowledge when they are relaxed or involved in the numerous activities that are part of everyday life. In addition, a wider variety of knowledge is probably acquired by push-type acquisition in comparison to pull-type because the former comprises all the information within the user’s range of vision and finding new knowledge is in itself enjoyable. Furthermore, interesting information may inspire individuals to explore a subject more deeply or search for related knowledge. Computers have not been fully utilized for such everyday enrichment because they are mostly used interactively and require direct manipulation, which is appropriate for explicit tasks. Various push technologies have been tried, but with little success. However, the ubiquitous computing environment is becoming a reality and many applications for everyday use have been proposed. We therefore propose the AmbientBrowser system, which provides people with a variety of enriching knowledge in the form of peripheral information during their everyday activities. In this paper, we describe the design and implementation of the AmbientBrowser.
2
Design
The basic concept of the AmbientBrowser is to give people the opportunity to encounter a wide variety of knowledge by continuously presenting information on ubiquitous displays. Fig. 1 shows an example of the AmbientBrowser in use.
Fig. 1. AmbientBrowser being used in living room
Various methods for peripheral presentation of information have been proposed in related work. However, in some of these studies, what are called ambient display, information is expressed in an abstract format, for example, using trembling, rotation speed, or figures in a picture [2] [3] [4]. Although these methods are less intrusive, they provide little information. In some cases, presentation of
96
M. Minakuchi, S. Nakamura, and K. Tanaka
detailed information on peripheral displays have been proposed [5] [6], but these proposals did not consider any enriching effects. We used Web pages as a knowledge source because there are a vast number of valuable pages on the World Wide Web. Currently, the main methods of accessing Web pages are via linking and search engines. Both are pull-type methods, i.e., the user actively accesses information with a specific aim. We thus have to provide an automatic information retrieval mechanism. Ubiquitous displays can be regarded as displays that ‘decorate’ the environment with an atmosphere of information. An example is an electronic billboard that dynamically provides news to people who are waiting in a queue, for example, enabling them to acquire information naturally and help pass otherwise idle time. We assume an environment surrounded by ubiquitous displays in locations such as kitchens, bathrooms, bedrooms, studies, offices, and streets. Computers connected to the Internet control these displays and provide the peripheral information. However, there are several questions relating to the use of ubiquitous displays; for example, what is the most appropriate method of presenting information? Since most Web pages are designed to be read actively, they may be difficult to read on ubiquitous displays if they are rendered in the same way as on a PC. Another question relates to how people can interact with displays. The AmbientBrowser is based on viewing only, but a simple and intuitive form of interaction would make it more useful. In summary, we need consider three main issues: – Information retrieval mechanism: how to select suitable topics. – Presentation of information: must be able to be read at a glance. – Method of interaction: must fit in with daily activities. We discuss these issues in the following subsections. 2.1
Information Retrieval Mechanism
We believe that ubiquitous displays should have their own distinctive characteristics and roles because people may be interested in, and naturally acquire, knowledge if the information on a ubiquitous display relates to their context, i.e. location or space. For example, a visitor to an aquarium could learn about the ecology and characteristics of various fish species if there was a ubiquitous display showing this information. Ubiquitous displays may also need to reflect a user’s preferences. For example, news or trivia about football, basketball, or baseball is likely to engage the interest of sports lovers. To fulfill these requirements, the AmbientBrowser selects target Web pages via Web search engines such as Google, using preset keywords. Keywords are categorized into two types as follows: – Keywords for ubiquitous displays — each display has keywords that represent its role in relation to its location. For example, a display in a kitchen has
AmbientBrowser: Web Browser for Everyday Enrichment
97
keywords such as “kitchen”, “food”, “cooking”, “water”, “drink”, and so on. The owner sets these keywords. Location-aware network services may set keywords according to the location of the display, which is convenient for portable displays. – Keywords reflecting user’s preferences — each user has keywords that portray his or her interests. The user tells the display what these keywords are while looking at it. Unlike many of the previous studies on context-aware computing, our goal is not just to notify users of significant information in a specific situation. Rather, the AmbientBrowser is aimed at also delivering unexpected knowledge because people enjoy chance encounters with interesting knowledge. Roger Caillois, for example, lists “alea” (i.e. chance) as one of the four universal categories of play [7]. Hence, we introduced randomness to the method of selecting target Web pages. Specifically, the system first randomly decides the number of keywords for the Web search within a preset range. It then randomly selects keywords from the ubiquitous display’s own keywords and user-preference keywords. It posts the selected keywords to the search engine and gets an URL list of search results. Finally, it randomly selects a URL of a target Web page, which is used as an information source, from the list. However, only preset keywords may not be sufficient to select a wide variety of knowledge because these keywords are static and limited. Thus, we introduced a casual navigation mechanism; the system sometimes automatically jumps to a Web page linked from the current target page. This mechanism is useful to find associative but unpredictable knowledge. 2.2
Presentation of Information
Typically, browsing the Web requires active reading. In other words, the reader has to focus on following the text and must manipulate the browser to update the pages being displayed by using the scroll bar or clicking on various links. This active style of browsing, however, is unsuitable for the AmbientBrowser because most of our daily activities do not involve sitting in front of displays. It is therefore difficult to browse Web pages using traditional user interfaces, such as graphical user interfaces, that require direct manipulation. In addition, users may find these operations tedious when they are relaxing. Thus, we need to convert original Web pages into an appropriate style of presentation. Given the characteristics of the AmbientBrowser, three factors are important in displaying information: readability, intuitiveness, and passiveness. Readability means that information should be easy to read from a distance or an angle because the user will not always be standing in front of the display. The content displayed should also be able to be understood at a glance, i.e. intuitively, because the user will not always be looking at the display. Passive reading is
98
M. Minakuchi, S. Nakamura, and K. Tanaka
casual, and is suitable for times when the user has little intention of reading, such as when he or she is relaxed. One of the simplest ways to improve readability is to use large letters. Assigning a CSS [8] to the ubiquitous display of original Web pages may work well. However, with regard to intuitiveness, enlargement of characters alone is inadequate. On the one hand, too many characters on the display may overwhelm the user, while on the other, too few characters may not convey enough knowledge to arouse the user’s interest. Converting Web pages into a TV-program-like format [9], i.e. transforming text into audio, avatar motion, and camera work, may be useful for passiveness. Because a TV program is time-based content, that is, its representation proceeds automatically as time elapses, users can watch content without having to do anything. However, the use of sound is unsuitable for ubiquitous displays because it is sometimes too noisy. It is also a time-consuming way of understanding content. Images and movies are appealing and easy to see, but they are less suitable for conveying advanced knowledge. We thus consider that text is the essential medium for the AmbientBrowser. Possible approaches to text presentation that satisfy the requirements above include using dynamic presentation [10] such as automatic scrolling and rapid serial visual presentation [11], because they can use large letters and do not require scanning eye movements to follow the text. One of the prospective presentation methods for the AmbientBrowser is kinetic typography – text that uses movement or other temporal change [12]. It is not only attractive but also useful to express additional meaning such as tone of voice, implication, and features of the content. Our ongoing investigation on readability of text on ubiquitous display suggests that kinetic typography is sufficiently readable from a distance while subjects are working on another task. In addition, motion patterns are distinguishable from each other without reading the text. This feature is helpful to let users notice features of shown information at a glance. And, more importantly, all subjects felt kinetic typography is more enjoyable than other presentation styles such as static text and simple scrolling like a ticker. 2.3
Method of Interaction
While looking at the AmbientBrowser, the user may want to interact with it. For example, someone who is interested in the information that has been shown may want to find related information, or may want to adjust the display’s update period to his or her preference. Thus, it is important to provide methods of interaction that accommodate users’ varying interests. However, because it is a premise of the AmbientBrowser that users are not always facing it when using it, they should not be required to look at the display or use input devices. Sensors fulfill these requirements, similar to multi-modal interfaces and context-aware computing. For example, if a range sensor detects that the user is coming closer, the system assumes that the user wants to read more, and it then displays detailed or related information.
AmbientBrowser: Web Browser for Everyday Enrichment
99
The system also controls the timing or speed of information presentation according to the sensor data. For example, the AmbientBrowser updates information according to brightness in the room, flow of the water, user’s motion, etc. These physical controls, i.e. user’s daily activities affects environmental artifacts, is fresh and enjoyable.
3
Pilot Study on Peripherally Displayed Knowledge
As discussed in Section 2.2, a few words or dozens of words presented in largesize letters can satisfy requirements for both readability and intuitiveness. This technique may also satisfy the requirement for passiveness because the user does not need to follow the words. The words can be extracted from headlines in Web pages and text summarization techniques can also be utilized. More simply, we can use RDF Site Summary (RSS) data. Two questions then arise: how do users notice information on a ubiquitous display? How many words are needed to stimulate a user’s interest? In an attempt to answer these questions, we experimentally investigated the use of the AmbientBrowser. We used displays that continuously showed text data (Fig. 2-a). Text data consisted of 100 tanka (thirty-one syllable poem), 333 maxims, and 1000 sayings by great men and women. Each discrete piece of text contained, at most, 100 characters. We also prepared three target sentences that instructed the subject to record a specific condition, for example, background color, for each type of data. The displays randomly selected a piece of text with probabilities that were defined for each type of data (the proportion was 10:3:1), and displayed it for 10 seconds. After 10 seconds, they selected the next piece of text and updated the display. Three subjects, one male researcher and two female office workers, participated in the experiment. We placed two 12” displays and a 5” display on each of the subjects’ desks in positions where they did not obstruct their work (Fig. 2-b). In addition, we set up a 21” display in a place where it could be seen by the subjects from anywhere in the room. The displays were not synchronized and independently displayed different data. Although the displays varied in size, there was no problem with visibility because the characters were large enough to be seen. The subjects were told to work as usual and not to be conscious of the system. They were also asked to record the number of times each background color occurred when he or she noticed at the target sentence on any display. After using the system for half a day to become accustomed to it, the displays were made available to the subjects during their working hours (about eight hours per day) for three days. The results are summarized in Table 1. Displays A, B, and C were placed on the desks of Subjects a, b, and c, respectively, and Display D was the 21” display visible to all of them. It was not known which display they noticed the target sentences.
100
M. Minakuchi, S. Nakamura, and K. Tanaka
(a)
(b)
Fig. 2. (a) Example of display. Target sentence staets: Please record the background color when you find this. (b) Example of experimental setting (5” display). Table 1. Actual presentation times and recorded times of target sentence Recorded times
Type
Presentation times
b
c
A
B
C
D
a
tanka
18
25
24
17
4
6
17
maxim
3
7
9
7
4
3
3
saying
3
2
1
6
1
1
1
The results indicated that the subjects noticed the target text on at least once of the several occasions it was presented. All the subjects reported that they were not distracted by the system. We did not examine the use of the system outside working hours, but we assume that there would be at least the same or even a higher possibility that users would notice the displayed information because they would be able to view the displays more casually. This result demonstrated that the users had adequate awareness of text on ubiquitous displays during everyday activities. The subjects reported that they enjoyed the displayed information and remembered some of it. This suggests that the AmbientBrowser is an effective means of enrichment. The subjects also reported that they thought about the experimenter when they looked at the target sentence because the experimenter’s name was written on the targets. This suggests that brief messages are sufficient to recall related information or actions.
4
Implementation
Fig. 3 shows the architecture of the AmbientBrowser. Possible keywords for Web searches are stored in the keyword database. As discussed in Section 2.1, there are two types of keywords: those for the display and those for the user. We used RFID cards to authenticate users. The system uses the user keywords of authenticated users. The keyword selector, URL list
AmbientBrowser: Web Browser for Everyday Enrichment
Owner
User
edit
RFID Reader
Keyword DB
URL List Fetcher Target URL Selector
Sensors
Keywords for User Preference
display Rendering Engine
Search Engine
Keyword Selector
History DB check
Ubiquitous Display
RFID Card
Keyword Editor Keywords for Ubiquitous Display
101
post results
WWW WWW
Motion Composer
Motion Library
Natural Language Processor
Concept Dictionary
Content Fetcher
Fig. 3. Architecture of AmbientBrowser
fetcher, and target URL selector randomly select a target Web page in the way described in Section 2.1. The target URL selector sometimes select a link in the current page instead of search results. It also checks the history database to avoid selecting the same page within a short period. Keywords may contain URLs, for example, ones for RSS feeds. If the keyword selector selects a URL, the keyword selector and target URL selector are skipped, and the system uses the URL. The content fetcher accesses the target URL and generates text data to be displayed. Several techniques, for example, extracting headlines, finding topic words, and text summarization, may be utilized. It also extracts pictures from the Web page. The system then attaches motion to the text data. For this, we used an automatic kinetic typography composer [13]. The motion library contains predefined motion patterns with index words. The natural language processor finds the most semantically related index words for each word in the given text. The motion composer combines the found motion patterns to make the final animation data. The rendering engine renders it on the ubiquitous display with the extracted pictures if any. Fig. 4 shows an example of automatically generated moving text. The system’s sensors detect users and environmental conditions, and control the AmbientBrowser, for example, by narrowing keywords to a specific topic or adjusting the animation speed for better readability. The current implementation uses brightness, temperature, humidity, airflow, and motion sensors.
5
Discussion
We use the AmbientBrowser in our laboratory. Although we have not yet evaluated it quantitatively, we find that people enjoy it, with many of them looking at it when they are taking a break or passing by. They also report that they
102
M. Minakuchi, S. Nakamura, and K. Tanaka
Fig. 4. Example of moving text (sequential snapshots)
are sometimes interested in the content and talk about it. The AmbientBrowser may thus have potential for initiating communication. The variation of keywords has limitations in keeping up with a user’s changing interests because the AmbientBrowser uses only preset, i.e. static, keywords. A user may use more than one RFID card to set keywords that match his/her interests, for example, cards for news, sports, or shopping. However, setting keywords is relatively difficult and we are therefore planning to introduce a dynamic keyword collection mechanism based on context data and social recommendation. For example, the system may select topic words in the presented text that the user takes a close look at. It could also use keywords that appear numerous times in blogs or bulletin boards, assuming that they relate to hot topics. User keywords might not be useful for personalizing content selection when multiple users are present. We think user keywords are applicable in situations when only several users are present because unexpected knowledge can be fun. In addition, it may also promote communication among users; for example, presented knowledge may result in a discussion among users. Although personalization will be difficult in crowded environments, various technologies such as location detection and face recognition can be utilized to find out who is looking at the display. Kinetic typography is generally favored because it is attractive. However, it occasionally attracts too much attention and is too intrusive when in one’s direct line of vision. An interruptibility prediction method [14] may solve this problem, i.e., the system could switch to static presentation when the user is busy. Information overload may overwhelm users and make them tired. We think the application design is important. The system should be so easily ignorable that users casually look at the display as if they only glance at leaflets. Actually, the pilot study showed that continuous presentation of information did not disturb daily work.
AmbientBrowser: Web Browser for Everyday Enrichment
6
103
Conclusion
We designed and implemented an AmbientBrowser system for everyday enrichment. Observation of the system in practical use showed that people enjoy using the AmbientBrowser and that it has potential to help them acquire knowledge during everyday activities. In future work, we will improve the system in terms of context awareness and advanced presentation of text. We also want to investigate its effects on enrichment and in encouraging communication through long-term observation.
References 1. Maslow, A. H.: Motivation and Personality. Harper & Row, New York (1970) 2. Weiser, M. and Brown, S.: Designing Calm Technology.http://www.ubiq.com/ weiser/calmtech/calmtech.htm (1995) 3. Dahley, A., Wisneski, C., Ishii, H.: Water Lamp and Pinwheels: Ambient Projection of Digital Information into Architectural Space. in CHI’98 Conference Summary on Human Factors in Computing Systems (1998) 269–270 4. Stasko, J., Miller, T., Pousman, Z. Plaue, C., Ullah, O.: Personalized Peripheral Information Awareness Through Information Art. in Proceedings of the 6th International Conference on Ubiquitous Computing (2004) 18–35. 5. Heiner, J. M., Hudson, S. E., Tanaka, K.: The Information Percolator: Ambient Information Display in a Decorative Object. in Proceedings of the 12th Annual ACM Symposium on User Interface Software and Technology (1999) 141–148 6. McCarthy, J., Costa, T. and Liongosari, E.: UniCast, OutCast & GroupCast: Three Steps Toward Ubiquitous, Peripheral Displays. in Proceedings of the 3rd International Conference on Ubiquitous Computing (2001) 332–345 7. Caillois, R.: Les jeux et les hommes. Gallimard (1958) 8. Cascading Style Sheets, level 1. http://www.w3.org/TR/CSS1/ (1999) 9. Tanaka, K., Nadamoto, A., Kusahara, M., Hattori, T., Kondo, H., Sumiya, K.: Back to the TV: Information Visualization Interfaces Based on TV-program Metaphors. in Proceedings of IEEE International Conference on Multimedia and Expo 2000 (2000) 1229–1232 10. Mills, C., Weldon, L.: Reading Text From Computer Screens. ACM Computing Surveys, Vol. 19, No. 4 (1984) 329–358 11. Potter, M.: Rapid Serial Visual Presentation (RSVP): A Method for Studying Language Processing. New Methods in Reading Comprehension Research (1984) 91–118 12. Lee, J.C., Forlizzi, J., Hudson, S. E.: The Kinetic Typography Engine: An Extensible System for Animating Expressive Text. in Proceedings of the 15th Annual ACM Symposium on User Interface Software and Technology (2002) 81–90 13. Minakuchi, M., Tanaka, K.: Automatic Kinetic Typography Composer. in Proceedings of the ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (2005) 14. Fogarty, J., Hudson, S. E., Atkeson, C. G., Avrahami, D., Forlizzi, J., Kiesler, S., Lee, J. C., Yang, J.: Predicting Human Interruptibility with Sensors. ACM Transaction on Computer-Human Interaction, Vol. 12, No. 1 (2005) 119–146
Ambient Intelligence in Edutainment: Tangible Interaction with Life-Like Exhibit Guides Alassane Ndiaye, Patrick Gebhard, Michael Kipp, Martin Klesen, Michael Schneider, and Wolfgang Wahlster German Research Center for Artificial Intelligence, DFKI GmbH, Stuhlsatzenhausweg 3, D-66123 Saarbrücken
[email protected] Abstract. We present COHIBIT, an edutainment exhibit for theme parks in an ambient intelligence environment. It combines ultimate robustness and simplicity with creativity and fun. The visitors can use instrumented 3D puzzle pieces to assemble a car. The key idea of our edutainment framework is that all actions of a visitor are tracked and commented by two life-like guides. Visitors get the feeling that the anthropomorphic characters observe, follow and understand their actions and provide guidance and motivation for them. Our mixed-reality installation provides a tangible, (via the graspable car pieces), multimodal, (via the coordinated speech, gestures and body language of the virtual character team) and immersive (via the large-size projection of the life-like characters) experience for a single visitor or a group of visitors. The paper describes the context-aware behavior of the virtual guides, the domain modeling and context classification as well as the event recognition in the instrumented environment.
1 Introduction Ambient Intelligence (AmI) refers to instrumented environments that are sensitive and responsive to the presence of people. Edutainment installations in theme parks that are visited by millions of people with diverse backgrounds, interests, and skills must be easy to use, simple to experience, and robust to handle. We created COHIBIT (COnversational Helpers in an Immersive exhiBIt with a Tangible interface), an AmI environment as an edutainment exhibit for theme parks that combines ultimate robustness and simplicity with creativity and fun. The visitors find a set of instrumented 3D puzzle pieces serving as affordances. An affordance is a cue to act and since these pieces are easily identified as car parts, visitors are being motivated to assemble a car from the parts found in the exhibit space. The key idea of our edutainment framework is that all actions of a visitor, who is trying to assemble a car, are tracked and two life-like characters comment on the visitor’s activities. Visitors get the feeling that the anthropomorphic characters observe, follow and understand their actions and provide guidance and motivation for them. We have augmented the 3D puzzle pieces invisibly with passive RFID tags linking these items to digital representations of the same (cf. [9]). Since we infer particular assembly actions of the visitors indirectly from the change of location of the instrumented car parts, the realtime action tracking is extremely simplified and robust comM. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 104 – 113, 2005. © Springer-Verlag Berlin Heidelberg 2005
Ambient Intelligence in Edutainment
105
pared with vision-based approaches observing the behavior of the visitors (see also [10]). The COHIBIT installation provides a tangible, (via the graspable car pieces), multimodal, (via the coordinated speech, gestures and body language of the virtual character team) and immersive (via the large-size projection of the life-like characters) experience for a single visitor or a group of visitors. Our installation offers ultimate robustness, since visitors can always complete their physical assembly task, even if the RFID-based tracking of their actions or the virtual characters are stalled. As Cohen and McGee put it, such hybrid systems “support more robust operation, since physical objects and computer systems have different failure modes” (cf. [1], p. 44). Thus, the ambient intelligence provided by the situation-aware virtual guides can be seen as an exciting augmentation of a traditional hands-on exhibit, in which the visitors do not see any computer equipment or input devices. The fact that visitors in our AmI environment are not confronted with any computing device contrasts the work described here with previous work. Like our system, PEACH (cf. [8]) and Magic Land (cf. [6]) both use mixed reality approaches for edutainment in museums, but the PEACH user must carry a PDA and the Magic Land user must wear a HMD to experience the mixed reality installation. The remainder of the paper is organized as follows. First, we present an overview of the instrumented edutainment installation. Then we focus on the distinguishing features of our approach: Section 3 concentrates on the context-aware behavior of the virtual guides, while Section 4 focuses on the domain modeling and context classification and Section 5 deals with the event recognition in the instrumented environment. Finally, Section 6 concludes the paper.
2 The Instrumented Edutainment Installation The COHIBIT installation creates a tangible exploration experience guided by a team of embodied conversational agents (cf. [5]). While the visitor can fully engage in the manipulation of tangible, real world objects (car pieces), the agents must remain in the background, commenting on visitor activities, assisting where necessary, explaining interesting issues of car technology at suitable moments and motivating the visitor if s/he pauses for too long. Since turn-taking signals (speech, gesture, posture) from the visitor cannot be sensed by our AmI environment, the agents must infer from the sensor data (car pieces movement) and from the context (current car configuration constellation, current state of the dialog) when it is a good time to take the turn/initiative and when to yield a turn, probably interrupting themselves. The exhibit requires interaction modalities that allow us to design a natural, unobtrusive and robust interaction with our intelligent virtual agents. We use RFID devices for this purpose. This technology allows us to determine the position and the orientation of the car pieces wirelessly in the instrumented environment. The RFID-tagged car pieces bridge the gap between the real and the virtual world. By using tangible objects for the car assembly task, visitors can influence the behavior of the two virtual characters without necessarily being aware of it. The RFID technology does not restrict the interaction with the exhibit to a single visitor. Many visitors can move car pieces simultaneously making the car assembly task a group experience. The technical set-up of the AmI environment (see Figure 1) consists of:
106
A. Ndiaye et al.
• Ten tangible objects that are instrumented with passive RFID tags and represent car-model pieces on the scale 1:5. There are four categories of pieces: (a) two front ends; (b) one driver’s cab, including the cockpit as well as the driver and passenger seats; (c) two middle parts providing extra cabin space, e.g. for a stretch limousine; (d) five rear ends for the following body types: convertible, sedan/coupé, compact car, van and SUV. • A workbench and shelves: The workbench has five adjacent areas where car pieces can be placed. Each area can hold exactly one element and the elements can be placed in either direction, i.e. the visitor can build the car with the front on the left and the rear on the right hand side or the other way around. • A high-quality screen onto which the virtual characters are projected in life-size. (We use state-of-the-art characters and CharaVirldTM, the 3D-Player of Charamel.) We project also a virtual screen which is used to display technical background information in the form of graphics, images, and short video sequences. • Three cameras facing to visitors and mounted on the ceiling for realtime detection of visitor presence. • A sound system for audio output in the form of synthesized speech feedback. (The audio output is provided by a unit selection based Text-To-Speech system.)
Fig. 1. Overview of the interactive installation prototype
The installation runs in two modes. The OFF mode is assumed when nobody is near the construction table. A visitor approaching the table is being detected by the ceiling camera and lets the system switch to ON mode. The idea behind these two modes is based on our experiences with CrossTalk – a self-explaining virtual character exhibition for public spaces [7]. In the OFF mode the virtual characters try to capture the attention of potential visitors. They talk to each other about their jobs and hobbies while making occasional references to the situational context (i.e., the current time of day, the outside weather, upcoming events). Making references to current real world conditions creates an illusion of life and establishes common bonds with the visitor’s world. When the visitors enter the installation, the system recognizes their presence and switches to ON mode. The virtual characters welcome them and explain what the
Ambient Intelligence in Edutainment
107
purpose of the exhibit is and encourage the visitors to start with the car assembly task. In the following construction phase visitors can remove pieces from the shelves and put them on the workbench. They can also modify a construction by rearranging pieces or by replacing them with other elements. In the ON mode the virtual characters play various roles, e.g., as guides, by giving context-sensitive hints how the visitor can get the construction completed; commentators and expert critics, by accompanying the assembly process through personalized comments on the visitors’ actions and on the current task state; motivators, by encouraging the visitors to continue playing; tutors, by evaluating the current state of the task and providing additional background information about car assembly. Figure 2 illustrates the visitor’s experience in our AmI exhibition space by showing the three basic phases of the ON modus (welcome, construction and completion).
Fig. 2. Examples of visitor actions and corresponding character behavior
108
A. Ndiaye et al.
3 Creating Context-Aware Behavior of the Virtual Guides In the COHIBIT installation, the characters’ verbal and non-verbal behavior is defined as a function of the visitors’ actions and the current state of the construction. The characters should be able to react instantly (e.g., by interrupting their current talk), adapting their conversation to the current situation. At the same time, we must avoid that their comments and explanations become too fragmented and incoherent as they react and interrupt each other. Commenting every move would be boring or even irritating. Instead, the system has to pick significant events like a soccer commentator who does not comment on every move of the many players on the field but selects those events that are most significant for the audience. Our design guidelines for the mixed-initiative interactive dialogue were developed by aiming at three major goals: (1) believability, (2) assistance, and (3) edutainment. Believability means to strengthen the illusion that the agents are alive. One means to achieve this is to let the agents live beyond the actual interaction in what we call the OFF mode. If no visitors are present, the two agents are still active, engaged in private conversation. Thus, from a distance, passers-by take them for being alive and, as an important side-effect, they might be attracted to enter the installation [4]. In ON mode, believability is created by making the agents react intelligently to visitor actions, i.e. being aware of the construction process and knowing possible next actions to complete the car assembly. The second goal is that of assistance concerning the car assembly task. Although it is not a difficult task, observation of naïve visitors showed that the agents’ assistance can speed up the task by identifying relevant car pieces, giving suggestions where to place pieces and how to rebuild configuration constellations that can never result in a valid car model. The third goal is to convey information in an entertaining fashion on two domains: cars and virtual characters. Both topics arise naturally in our AmI environment. The car pieces serve as a starting point for explanations on topics such as air conditioning or safety features of cars. Virtual character technology is explained mainly in the OFF mode, during smalltalk, but also in the ON mode. For example, if the same situation has occurred so often that all prescripted dialog chunks have been played, the characters react by apologizing for using only a limited number of pre-scripted dialog contributions. 3.1 Scenes and Sceneflow The installation’s behavior is defined in terms of structure (sceneflow) and content (scenes) [2]. The sceneflow is modeled as a finite state machine (currently consisting of 94 nodes and 157 transitions) and determines the order of scenes at runtime using logical and temporal conditions as well as randomization. Each node represents a state, either in the real world (a certain car construction exists on the workbench) or in the system (a certain dialog has been played). Every transition represents a change of these states: e.g., a visitor places another piece on the workbench or a certain time interval has passed. Each node and each transition can have a playable scene attached to it. A scene is a pre-scripted piece of dialog that the two agents perform. Our 418 scenes are defined separately from the sceneflow definition in the form of a multimodal script which contains the agents’ dialog contributions, gestures, facial expressions
Ambient Intelligence in Edutainment
109
and optional system commands (e.g., displaying additional information on the virtual screen). Scenes can be edited by an author with standard text processing software. The major challenge when using pre-scripted dialog segments is variation. For the sake of believability the characters must not repeat themselves. Therefore, the multimodal script allows for context memory access (identifier of the relocated car piece, session time, current date and weather situation) and making scenes conditional (play a scene only on a Wednesday or only between 9:00 and 11:00 am). The contextual information is verbalized at runtime. In addition, we increase variation by blacklisting: already played scenes are blocked for a certain period of time (e.g., 5 minutes), and, variations of these scenes are selected instead. For each scene, we have 2-9 variations that make up a so-called scene group. Scene groups that need a high degree of variation are identified by analyzing automatic transcripts of real-life test runs for high density of scene groups. In case that all scenes of a scene group are played, a generic diversion scene is triggered that can be played in many contexts. Various sets of such scenes exist where the agents comment on the current car piece, the number of today’s visitors, the current weather etc. As a third strategy for variation, long scenes are decomposed into smaller sections that are assembled at runtime based on the current conditions and blacklisting constraints. 3.2 Design Optimization Based on User Studies In an iterative development approach, the system was tested repeatedly with 15 naïve visitors at two different stages of development to identify necessary changes. In these informal tests, an evaluation panel, including research staff and members of the theme park management, observed the visitors (including groups of up to three) who interacted with the system to check on system performance and robustness under realistic conditions. A number of visitors were interviewed by the panel after the test. Critical discussion of our observations and the interviews led to essential changes in the prototype’s interaction design: • Visitors are grateful for instructions and assistance. Some visitors seem to be afraid of doing “wrong” in presence of fellow visitors → Focus on the assembly task first (concise comments), assist where necessary, tell about more complex issues later. • Visitors move multiple pieces at once. Multiple visitors even more so. The system is bombarded with many simultaneous events → Introduce lazy event processing as a form of wait-and-see strategy (see Section 5). • Visitors memorize only few facts presented as background information being too busy with the assembly task. → Give in-depth information at points of “natural rest” (car is completed), and visualize the information using multimedia. • Visitors may be more interested in the agents than in the car. They try to find out how the agents react to various situations like “wrong” car piece combinations. → Include “meta-dialogs” about virtual character technology to satisfy these interests and cover a large range of “error cases” to give the visitor space for exploration. The user studies led to strategies for deciding when the agents should start speaking and for how long, based only on the knowledge about the current visitor action and the state of the dialog. A piece being placed on the workbench is a possible time to start talking because the visitor (1) may expect the agents to talk (many visitors
110
A. Ndiaye et al.
placed a piece on the table and looked at the agents expectantly) or (2) may need assistance on how to continue. However, starting to speak each time a piece is placed could irritate visitors focused on the assembly task. Consequently, (1) depending on the visitor’s assembly speed (number of actions per time unit) we give a lengthy, short or no comment and (2) we let the agents interrupt themselves as new events arrive. To make the interruption “smooth” we insert transitory scenes, which depend on the interrupted speaker. For example, if agent A is being interrupted, agent B would say: “Hold on for a second.” Alternatively, agent A could say: “But I can talk about that later.” An even more important time to initiate talking by the agent is when a car is completed. Depending on the context, (a) the completed car is described as a “reward” for correct assembly or (b) a certain aspect of the car is explained in-depth or (c) a recommendation for another exhibit is given. As our system is based on probabilistic state transitions, the density of information given at the various stages of the interaction can be easily adapted according to user studies.
4 Domain Modeling and Context Classification As mentioned in Section 2, there are ten instrumented pieces that the visitors can use to build a car. Elements can be placed on each of the five positions on the workbench in either direction. This leads to a large number of possible configurations, totaling 802,370 different combinations! The RFID readers provide the AmI environment with a high-dimensional context space. It is obvious that we cannot address each configuration individually when planning the character behavior. On the other hand we need to be careful not to over-generalize. If the virtual exhibit guides would just point out “This configuration is invalid.” without being able to explain why or without giving a hint how this could be rectified, their believability as domain experts would be diminished, if not destroyed. We use a classification scheme for complete and partial constructions that consists of the following five categories: 1. Car completed: a car has been completed (30 valid solutions). 2. Valid construction: the construction can be completed by adding elements. 3. Invalid configuration: an invalid combination of elements (e.g., driver’s cab behind rear element). 4. Completion impossible: the construction cannot be completed without backtracking (e.g., driver’s cab is placed on an outermost workbench position where there is no possibility to add the rear) 5. Wrong direction: the last piece was placed in the opposite direction with respect to the remaining elements. An important concept in our classification scheme is the orientation of the current construction (car pointing to left or right). The orientation is determined by the majority of elements pointing in the same direction. If, for example, two elements point to the left but only one element points to the right then the current overall orientation is “left”. The current orientation can change during construction. Workbench configurations that only differ in orientation are considered equivalent. Using the concept of orientation the agents can assist if pieces point in the “wrong” direction by saying:
Ambient Intelligence in Edutainment
111
“Sorry, but we recommend flipping this new piece so it points in the same direction as the other pieces.”. In many cases only the category of an element needs to be considered and not the instance. A category is denoted by F for front element, C for cockpit, M for middle element, and R for rear element. Each configuration context is represented by a construction code that has a length between one and five. Empty positions on the left and on the right hand side of the placed pieces are omitted and empty positions between pieces are marked with a hash symbol. The construction code F#M, for example, describes a state in which a front element and a middle element are placed somewhere on the workbench with an empty position between them. If the user remains inactive in this situation the agents will take initiative: “May we propose to put the cockpit in the gap between the front end and the middle piece.”. The construction code is orientation-independent. Configurations on the workbench are treated as equivalent if they have the same construction code. The car type is defined by the construction code and the rear element used. A roadster, for example, has the construction code FCR and the rear of a convertible. Using the available pieces, visitors can build 30 different cars. If the visitor produces erroneous configurations we only evaluate the local context of the configuration, which comprises the neighboring positions of the currently placed piece. The error code FR#, for example, describes a situation in which a rear element has been placed behind a front element. This configuration is invalid since a front element must be followed by a driver’s cab. Here, the agents will help: “We are sorry, but you have to put first the cabin behind the front, before placing a rear piece.”. The evaluation of the local context can also result in an error situation, where a completion is impossible. The three concepts current orientation, construction code, and local context enable us to reduce the combinatorial complexity by classifying each configuration context into five distinct categories. The construction code and the error code can be seen as succinct descriptions of the current situation on the workbench. They provide enough context information to react intelligently to the current state of the construction.
5 Recognizing Events in the Instrumented Environment The system’s sensors consist of cameras and RFID readers. Sensory data is interpreted in terms of visitor actions (RFID readers) and visitor arrival/departure (cameras). Visitor actions consist of placing and removing car pieces to/from the workbench. Since these actions eventually control the way the characters behave, the major challenge is how to process the raw sensory data, especially if many actions happen at the same time (multiple visitors moving multiple pieces). The processing of the sensory data has 2 phases: (1) mapping of the visitors’ actions onto internal transition events, and (2) updating the representation of the current car construction. In terms of transition events, we distinguish four types that are ranked in a specific priority scheme: 1. Visitor appeared: visitor has entered the installation. Visitor disappeared: visitor has left the installation. 2. Car completed: visitor has completed the car assembly. Car disassembled: visitor has disassembled a car by removing a piece.
112
A. Ndiaye et al.
3. Piece taken: visitor has removed a piece from the workbench. Piece placed: visitor has placed a piece on the workbench. 4. Piece upheld: visitor is holding a piece above the workbench. The events visitor appeared and visitor disappeared have the highest priority since they trigger ON and OFF mode. The events car completed and car disassembled have the second highest priority since they trigger the transitions between the construction phase and the completion phase in the sceneflow. The other three events are used in the construction phase to trigger scenes in which the characters comment on the current state of the construction, give in-depth information about the used pieces and hints on how to continue. We use this priority scheme to decide which transition events are considered for further processing, e.g., if two visitors move pieces simultaneously. If the first visitor completes the construction of a car and the second visitor places another piece on the workbench shortly afterwards, the characters start commenting on the completed car instead of the single unused piece. The generation of transition events is followed by the classification of the current state of the construction using the five categories in our domain model (cf. Section 4). In addition, context information like the category, instance, and orientation of the currently placed piece, the number of pieces on the workbench, the overall creation time, the number of errors so far is updated. This context information along with the transition event is stored in an action frame. In a last step, the action frame that contains the transition event with the highest priority is selected for further processing. Transition events are used to branch in the sceneflow graph and context information is used in the selected scenes. The decision when to handle the next event is an integral part of our dialog/interaction model and explicitly modeled in the sceneflow (cf. [3] for more details). This enables us to keep the comments and explanations of the characters consistent and to balance reactivity and continuity in their interactive behavior.
6 Conclusions We have presented COHIBIT, an AmI edutainment installation that guides and motivates visitors, comment on their actions and provides additional background information while assembling a car from instrumented 3D puzzle pieces. The fact that visitors in our AmI environment are not confronted with any computing devices contrasts the work described here with previous work in which visitors have to deal with additional hardware to experience the mixed reality installation. The system is currently fully implemented and will be deployed in a theme park at the beginning of 2006, using professionally modeled car pieces and exhibit design. Although we have already conducted various informal user studies during the development of the various prototypes, the actual deployment will give use the opportunity for a large-scale empirical evaluation. In future work we intend to exploit the full potential of the vision module by recognizing visitors approaching or in the near of the exhibit to invite and encourage them to visit the installation. The computer vision module could also be used to determine whether the visitors are looking at the characters or are engaged in the car assembly task, in order not to interrupt. A further issue we plan to investigate is the appropriate dealing with groups of individuals interacting with the system. A prereq-
Ambient Intelligence in Edutainment
113
uisite for this project is the ability to track many visitors during the whole time they are on the exhibit. Acknowledgments. We are indebted to our colleagues Gernot Gebhard and Thomas Schleiff for their contributions to the system. We thank our partners of Charamel (www.charamel.de) for providing us with the 3D-Player and the virtual characters and our colleagues of the department “Multimedia Concepts and their Applications” at the University of Augsburg for the realtime video-based presence detection system. Parts of the reported research work have been developed within VirtualHuman, a project funded by the German Ministry of Education and Research (BMBF) under grant 01 IMB 01.
References [1] Cohen, P. R., McGee D. R.: Tangible multimodal interfaces for safety-critical applications. In: Communications of the ACM 47(1), 2004, pp. 41-46. [2] Gebhard, P., Kipp, M., Klesen, M., Rist, T.: Authoring scenes for adaptive, interactive performances. In: Proc. of the Second International Joint Conference on Autonomous Agents and Multi-Agent Systems, ACM Press, New York, 2003, pp. 725-732. [3] Gebhard, P., Klesen, M.: Using Real Objects to Communicate with Virtual Characters. In: Proc. of the 5th International Working Conference on Intelligent Virtual Agents (IVA'05), Kos, Greece, 2005, pp. 48-56. [4] Klesen, M., Kipp, M., Gebhard, P., Rist, T.: Staging exhibitions: methods and tools for modelling narrative structure to produce interactive performances with virtual actors. Virtual Reality, Vol. 7(1), Springer, 2003, pp. 17-29. [5] Prendinger, H. and Ishizuka, M. (eds.) Life-like Characters: Tools, Affective Functions and Applications, Springer, 2004. [6] Nguyen, T., Qui, T., Cheok, A., Teo, S., Xu, K., Zhou, Z., Mallawaarachchi, A., Lee, S. Liu, W., Teo, H., Thang, L., Li, Y., Kato, H.: Real-Time 3D Human Capture System for Mixed-Reality Art and Entertainment. In: IEEE Transactions on Visualization and Computer Graphics, vol. 11 (6), November/December 2005, pp. 706-721. [7] Rist, T., Baldes, S., Gebhard, P., Kipp, M., Klesen, M., Rist, P., Schmitt, M.: Crosstalk: An interactive installation with animated presentation agents. In: Proc. of the Second Conference on Computational Semiotics for Games and New Media, Augsburg, September 2-4, 2002, pp. 61-67. [8] Rocchi, C., Stock, O., Zancanaro, M., Kruppa, M., Krüger, A.: The museum visit: generating seamless personalized presentations on multiple devices. In: Nunes, N. J., Rich, Ch. (ed.): International Conference on Intelligent User Interfaces 2004. January 13-16, 2004, Funchal, Madeira, Portugal. pp. 316-318. [9] Ullmer, B. and Ishii, H.: Emerging Frameworks for Tangible User Interfaces. In: Carroll, J.M., (ed.): “Human-Computer Interaction in the New Millennium”, Addison-Wesley, 2001, pp. 579-601. [10] Wasinger, R., Wahlster, W.: The Anthropomorphized Product Shelf: Symmetric Multimodal Interaction with Instrumented Environments. To appear in: Aarts, E., Encarnação, J. (eds.): True Visions: The Emergence of Ambient Intelligence, Springer, 2005.
Drawings as Input for Handheld Game Computers Mannes Poel, Job Zwiers, Anton Nijholt, Rudy de Jong, and Edward Krooman University of Twente, Dept. Computer Science, P.O. Box 217, 7500 AE Enschede, The Netherlands {mpoel, zwiers, anijholt}@cs.utwente.nl
TM
Abstract. The Nintendo DS is a hand held game computer that includes a small sketch pad as one of it input modalities. We discuss the possibilities for recognition of simple line drawing on this device, with focus of attention on robustness and real-time behavior. The results of our experiments show that with devices that are now becoming available in the consumer market, effective image recognition is possible, provided a clear application domain is selected. In our case, this domain was the usage of simple images as input modality for computer games that are typical for small hand held devices.
1
Introduction
Game user interfaces convert the actions of the user to game actions. In a standard PC setting game players mainly use keyboard and mouse. It does not necessarily mean that there are straightforward mappings from the usual keyboard and mouse actions (pointing, clicking, selecting, dragging and scrolling) to similar actions in the game. On the contrary, there are mappings to menu and button actions, to route and region selection on maps, to tasks and objects and to actor activity. The cursor can obtain context-dependent functionality, allowing for example pointing, grabbing, catching, caressing and slapping. In games the computer may also ask the player to select or draw paths and regions on a map. Drawing interfaces for games are available. For example, several games have been designed where in a multi-user setting a player enters a drawing that best expresses a word or phrase that has been assigned to him and other players have to guess the word. Hence, the interaction is based on a drawing, but the drawing is just mediated to the other players and there is no attempt to make the computer interpret the drawing and make this interpretation part of a game. A recent example of a game computer that allows game developers to include player made drawings in a game is the handheld game computer Nintendo DS. Its touch screen can be used as a sketch pad that allows for simple drawings to be made. Engine-Software 1 company is a small company that investigates TM
1
Nintendo DS is a trademark of Nintendo of America Inc. Engine-Software: http://www.engine-software.nl
M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 114–123, 2005. c Springer-Verlag Berlin Heidelberg 2005
Drawings as Input for Handheld Game Computers
115
the possibilities of this feature in the development of interactive games, where pen input replaces the more traditional modalities such as buttons and mouse. The type of application (interactive games) results in hard real-time constraints, and requires a fair amount of robustness: game players do not want to wait for image recognition processes, nor do they want to have an editing or correction stage. Since the device has also only limited computational power, the selection of algorithms for recognition of pen input becomes challenging. In Fig. 1 we have displayed the instruction screen of the game, which can be consulted before the game starts, while on the right, during the game the gamer has drawn an object that helps the hero to deal with a particular problem. Here a trampoline is drawn that can be used to jump over an obstacle. We have implemented and
Fig. 1. Instruction screen (left) and sketch pad (right)
tested a possible setup for recognition of a set of simple line drawings that could play a role in games. Experimental results have been obtained for drawings of a set of 22 different classes of objects, see Fig. 2 for examples. After an (offline) training phase the classification returned correct result in 90 %, whereas the overall recognition rate was over 86 %. We do not compare this approach with respect to playing performance with the approaches using more classical input modalities (keyboard and mouse actions such as typing, clicking, dragging, etc) for supporting game interactions. The focus is on new interaction modalities for interactive games. In the remainder, we first describe the global structure of the recognition process, consisting of line recognition, feature extraction, and template matching. Thereafter, we discuss the results of testing the performance of our classification process.
2
Global Structure
A basic assumption that is made is that users of the device will produce simple line drawings, according to a predefined set of “objects” that play a role in the game. Since users should be able to learn to draw very easily, it was decided that the drawings are built up from straight lines and circles only. The global image recognition process is built up like a pipeline:
116
M. Poel et al.
– The first stage tries to simplify curves that are being drawn. Lines and circles that are drawn tend to be imperfect, partly due to difficulties with drawing straight lines on a slippery surface, and partly due to inaccurate drawing. This stage tries to reduce curves to either lines or circles, and tries to combine series of small line segments into larger segments. – The second stage is concerned with extracting a set of features from lines and circles. Typical features like number of lines and circles, number of horizontal lines, number of intersections etc. – The final stage tries to classify a drawing based upon features. We used a decision tree approach, with a tree that was created off line, from experimental data.
(a) Hammer
(b) Harry
(c) Marbles
(d) Suitcase
(e) Ladder
(f) bulb
(g) Mobile
(h) tendo
(i) Umbrella
(j) Plank
Light
Nin-
(k) Pole
(l) Sleigh
(m) Key
(n) Detonator
(o) Trampoline
(p) Television
(q) Seesaw
(r) Pedal car
(s) Broom
(t) Bomb
(u) CD
(v) Monocycle
Fig. 2. Some examples of drawings and the corresponding classes
Drawings as Input for Handheld Game Computers
2.1
117
Line Recognition and Simplification
Image recognition for devices like the Nintendo DS, and the typical application that one has in mind for such devices can be characterized as follows: – The image quality itself is not very high, due to the limited resolution of digitizing process that is built into the device. Also, the surface of the sketch pad is slippery, which makes it difficult, for instance, to draw perfectly straight lines. – “Drawing” is used as an input modality, used to control a game in real-time, rather than to input complex drawings. The image recognition process should take this into account. Consequently, many advanced techniques have been rules out in favor of simple, fast and robust recognition techniques. For instance, we have assumed that images consist of line drawings, consisting of simple curves like straight line segment or circles. In our case after some comparisons with other algorithms, we chose the algorithm by Douglas and Peucker [1]. This is a global approach where a curve is subdivided into two parts, where the division point is chosen as the point that deviates most from the straight line from start point to end point. This subdivision process is repeated recursively, until the deviation from a straight line becomes smaller than a predefined threshold. The threshold depends on the distance between start and end point in order to retain small scale features. The algorithms were tested on a set of 385 drawings, from 21 categories. For each drawing, the ”correct” set of lines was established manually, based upon human judgment. The results of the three line simplification algorithms have been compared to these correct lines. It turned out that the Douglas and Peucker algorithm was able to classify correctly in 95% of all cases, outperforming the other algorithms
3
Feature Extraction
After line simplification has been applied, the segments and circles are reduced to a small set of features that are suitable for the classification phase, discussed in Sect. 4. In the literature one can find many features used for the recognition of drawings. A short and definitely not complete list of static features is given below. – – – – – –
Chain code features [2]. Line features, such as horizontal, vertical and diagonal [3]. Characteristic points such as vertices, intersections, endpoints [4]. Loops [4]. Geometric forms such as circles, triangles, squares [5,6]. Ratio between height and width [7].
In the next subsections we introduce the selected features, and consider the computational effort to calculate them. The latter is important, since we are focussing on applications that must run in real-time on low end devices.
118
3.1
M. Poel et al.
Directions of Line Segments and Length Ratios
The direction of line segments turned out to be a sound basis for a number of features. On the one hand side, determining such features is computationally cheap. On the other side, the class of drawings consists mainly of pictures where lines are either horizontally, vertically, or diagonally. The direction of drawing a line was deemed insignificant: for instance, a horizontal line can be drawn from left to right or vice versa, but we would not classify the picture differently in these two cases. So that leaves us basically with four different directions, and simply counting the number of segments in each of these four categories resulted in useful features. A slightly different approach is not to count, but rather calculate the ratio of the total (i.e. accumulated) length of all segments in a certain direction to the total accumulated length of all segments.
3.2
Line Crossings and Corners
The number of line crossings and the number of corners have also been used as features. Due to the line simplification preprocessing phase, such features can be calculated in an efficient and robust way, simply by solving linear equations and classifying the intersection points: if the line intersection point is near the end points of both segments, it is a corner point, if the line intersection point is on both segments but not in the neighborhood of these end points, it is an ordinary intersection of the two segments etcetera. This rather simple approach is possible only because we assume drawings to consist of lines. For instance, in [4] intersections are calculated for character recognition in handwriting, and there, intersections have to be determined on the level of relationship between neighboring pixels.
3.3
Detecting Circles
The number of circles is an obvious feature, given the set of drawings that we are interested in. The question is mainly how such circles can be detected in a computationally cheap way. For instance, detection based on the Hough transform, that would be a good approach otherwise, is ruled out on the basis of computational cost. Moreover, it was experimentally observed that users had great difficulty in drawing “nice” circles on the screen of the Nintendo device. Usually, the result could be described best as a “polygon with a large number of edges, with shallow angles between consecutive edges”. This informal description, together with the requirement that end point of the drawing stroke should be within the neighborhood of the start point, resulting in a very simple yet effective detection algorithm for circles. As before the success here depends heavily on the constraints on drawings: pictures should consist of straight line segments and circles only. For example, drawing a shape in the form of the number eight will result in incorrectly detecting a circle shape.
Drawings as Input for Handheld Game Computers
3.4
119
The Complete Feature Vector
Summarizing, we have used the following set of robust features, cf. Table 1. These features have a high discriminative power and can be computed efficiently. Table 1. The list of selected features for recognizing the drawings. The value for these features is given for a typical example of the class “Umbrella” and “Key”. Feature Umbrella Key Number of horizontal lines 3 1 Number of vertical lines 1 2 Number of up diagonals 1 0 Number of down diagonals 1 0 Number of vertices 4 1 Number of intersections 1 1 Number of circles 0 1 Ratio horizontal/total length 0.68 0.75 Ratio vertical/total length 0.10 0.25 Ratio up diagonal/total length 0.13 0 Ratio down diagonal/total length 0.09 0 Ratio height/width 0.91 0.40
4
Classifying Drawings
The final phase in the recognition process is the classification of the drawing based on the features discussed in the previous section, Sect. 3. The approach taken is to use machine learning techniques to train a classifier based on a training set of drawings. The requirements for the classifier are that it should run on the low-end device, hence it should take a minimum of processing power, it should be fast! Moreover the classification procedure should be transparent for by humans and have a high performance. Given this requirements an obvious candidate is decision trees, [8,9]. But decision trees will classify every drawing to one of the a priori determined classes and hence it will also assign a class to drawings, for instance a drawing of a car, which are completely out of the domain. In order to determine that a drawing is out of the domain we use template matching for determining if a drawing actually belongs to the assigned class. In order to construct the decision tree and determine the templates for template matching a set of around 940 drawings was gathered. These drawings where generated by 35 persons, for each class there are approximately 30 example drawings. This set was split in two parts: a test set of 314 drawings (chosen at random) and a training set consisting of the remaining drawings.
120
4.1
M. Poel et al.
Decision Trees for Classifying Drawings
We use Quinlan’s C4.5 algorithm [9] for learning a decision tree from the available training data. It is well known in the theory of decision trees that pruning a decision tree improves the performance on new unseen data. This pruning can be done with several confidence levels, c.f. [9]. A K-fold cross validation experiment was performed on the training set to determine the best confidence level. It turned out that pruning improved the performance but there was no statistical significant difference between the different confidence levels. Hence a confidence level of 0.1 was taken and afterwards a decision tree was learned from the training data using C4.5. For attribute selection the “information gain” criteria was taking, and afterwards the tree was pruned with confidence level 0.1. The test results can be found in the next section, Sect. 5 4.2
Template Matching for Recognizing Out of Domain Drawings
The next step in the classification procedure is to recognize out of domain drawings. A decision tree assigns to each new drawing one of the a priori defined classes, also when the drawing is completely out the range of allowed drawings. This is not to reject drawings. One solution is to define a “reject” class and in corporate this class in the learning of the decision tree. But then we should have examples of all possible classes of drawings which should be rejected. This means that there should be almost an infinity number of drawings for this rejection class in order to learn all the drawings which should be rejected. Hence this approach does not work. A workable solution is as follows. For a new drawing first the drawing is classified by the decision tree, say as class C. Afterwards the drawing is matched against a template constructed for class C. If the drawing differs to much from the template, i.e the difference is above a certain predetermined threshold, the drawing is rejected, i.e. considered out of domain. The template for each class C is constructed as follows. A template T of grid size n × n is taken. For each example e of the class under consideration a bounding box around the drawing is determined, afterwards for each grid (i, j) in the template T the number of lines crossing this grid is determined. This is the value of Te (i, j). After calculating Te ’s for each example of the class in the training set the resulting Te ’s are averaged over the number of elements of class c in the training set, this is the template TC for class C. TC (i, j) =
1 Te (i, j) N e∈C
with N the number of elements of class C in the training set. For the class “Detonator” the resulting template is for n = 8 given in Fig. 3. After constructing the template TC for each class C, the rejection threshold RT C for each class needs to be determined. This threshold is the maximum
Drawings as Input for Handheld Game Computers
121
Fig. 3. The template for the class “Detonator” for n = 8
distance of the templates Te and the class template TC . The maximum is taken over all the examples e in the training set belonging to class C. RT C = max e∈C {Te − TC } where Te − TC =
|Te (i, j) − TC (i, j)|
i,j
5
Test Results
To test the performance of the global system we once used again the available data set of 900 drawings. From this data set 600 examples were taken to train the decision tree, using C4.5 with pruning at a confidence level of 0.1, c.f. [9]. Also the rejection threshold for the template matching was determined using this training set, as described in the previous section, Sect. 4. 5.1
Classification Performance of the Decision Tree
On the test set the average performance of the decision tree was 90.4%. The most important confusions made by the decision tree are listed in Table 2. From this table it follows that there is a relatively large confusion between Television drawings and Nintendo drawings. Looking at the example drawings, Fig. 2, this can be explained by the fact that for the extracted features there is only a small Table 2. The important confusions made by the decision tree. Rows correspond to actual classes, columns to classified classes. Mobile Nintendo Mobile 82 % 9% Nintendo 18 % 76 %
Hammer Television
Hammer Television 84 % 8% 11 % 81 %
Bomb CD Bomb 90 % 2 % CD 10 % 88 %
122
M. Poel et al.
difference in the number of horizontal lines, namely 1. The confusion between a bomb and a CD is due to the fact that the inner circle of the CD is sometimes not detected. The confusion between a Hammer and a Television is not that easily explained. Confusion between a bomb and a CD could be detrimental for the game play. This confusion could be resolved by using other input modalities such as speech, or by letting the player select between the most likely alternatives. This last option would slow down the game speed. 5.2
Performance After Template Matching
After a drawing is classified, say as class C, then it is matched against the template of that particular class in order to determine if the drawing is out of domain, c.f. Subsect. 4.2. This lead to the following results, c.f. Table 3. It should be observed that we only tested with example drawings each of which belonged to a class, since we did not have out of domain drawings in our training and test set. From Table 3 it follows that 26 instances of misclassified drawings are Table 3. The performance of the template matching procedure. TD stands for the decision tree classification procedure and TM for template matching procedure. DT TM number correct not rejected 271 correct rejected 13 not correct rejected 26 not correct not rejected 4
also rejected by the template matching procedure which is a good sign. But 13 correctly classified drawings are rejected by the template matching procedure, which is a bad sign. The overall performance of the classification system, correctly classified by the decision tree and not rejected by the template matching procedure, is 271/314, which equals 86.3%.
6
Conclusions and Future Work
The results of our experiments show that with devices that are now becoming available in the the consumer market, effective image recognition is possible, provided a clear application domain is selected. In our case, this domain was the usage of simple images as input modality for computer games that are typical for small hand held devices. Since devices such as the Nintendo DS are likely to have possibilities for limited forms of speech recognition, it would be worthwhile to investigate the fusion of such speech data with the image data from the sketch pad. This could also resolve confusions, such as between a bomb and a CD, as such a confusion is detrimental for the game play.
Drawings as Input for Handheld Game Computers
123
The image recognition process itself could also be improved: The set of features that we have used turned out to be sufficient for a limited set of drawings consisting of straight lines and circles, but this might change as soon as we enlarge this class, and allow curved lines, filled areas, 3D pictures etcetera. Also, the classification that was based on decision trees might be replaced by other techniques, such as neural nets or Bayesian networks. The current approach assumes that aspects such as training or learning are offline processes: there is no learning phase where the end user can train its own device. Of course it is questionable whether end users are willing to go through such processes. It seems more attractive to introduce processes that will fine-tune an existing classifier based upon data that is collected while the final application is in use.
References 1. Douglas, D., Peucker, T.: Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. Canadian Cartographer 10 (1973) 112–122 2. Anzai, Y.: Pattern Recognition and Machine Learning. Academic Press (1989) 3. Theodoridis, S., Koutroumbas, K.: Pattern Recognition. Academic Press (1999) 4. Pandya, A., Macy, R.: Pattern Recognition with Neural Networks in C++. CRC Press (1996) 5. Fonseca, M., Pimentel, C., Jorge, J.: Cali: an online scribble recognizer for calligraphic interfaces. In: Proc. AAAI Spring Symposium on Sketch Understanding. (2002) 51–58 6. Caetano, A., Goulart, N., Fonseca, M., Jorge, J.: JavaSketchIT: Issues in sketching the look of user interfaces. In: Proc. AAAI Spring Symposium on Sketch Understanding. (2002) 9–14 7. Parizeau, M., Lemieux, A., Gagn´e, A.: Character recognition experiments using unipen data. In: Proc. Int. Conference on Document Analysis and Recognition. (2001) 481–485 8. Duda, R., Hart, P., Stork, D.: Pattern Classification. Wiley New York (2001) 9. Quinlan, R.: C4.5: programs for machine learning. Morgan Kaufmann Publishers Inc. San Francisco, CA, USA (1993)
Let’s Come Together — Social Navigation Behaviors of Virtual and Real Humans Matthias Rehm, Elisabeth Andr´e, and Michael Nischt Augsburg University, Institute of Computer Science, 86136 Augsburg, Germany {rehm, andre}@informatik.uni-augsburg.de http://mm-werkstatt.informatik.uni-augsburg.de
Abstract. In this paper, we present a game-like scenario that is based on a model of social group dynamics inspired by theories from the social sciences. The model is augmented by a model of proxemics that simulates the role of distance and spatial orientation in human-human communication. By means of proxemics, a group of human participants may signal other humans whether they welcome new group members to join or not. In this paper, we describe the results of an experiment we conducted to shed light on the question of how humans respond to such cues when shown by virtual humans.
1
Introduction
Synthetic agents have been employed in many games and entertainment applications with the aim to engage users and enhance their experience. However, to achieve this goal, it does not suffice to provide for sophisticated animation and rendering techniques. Rather, other qualities have to come in focus as well, including the provision of conversational skills as well as the simulation of social competence that manifests itself in a number of different abilities. While earlier work focused on one specific aspect of social behavior, such as the expression of socially desirable emotions, more recent research aims at the operationalization of complex models of social behavior between members of a group including emotions, personality and social roles as well as their dynamics. For instance, Prendinger and Ishizuka [16] investigate the relationship between an agent’s social role and the associated constraints on emotion expression. They allow a human script writer to specify the social distance and social power relationships among the characters involved in an application, such as a multi-player game scenario. Another approach has been taken by Rist and Schmitt [19] who aim at emulating dynamic group phenomena in human-human negotiation dialogues based on sociopsychological theories of cognitive consistency dynamics [13]. To this end, they consider a character’s attitudes towards other characters and model a character’s social embedding in terms of liking relationships between the character and all other interaction partners. Prada and Paiva [15] as well as Marsella and colleagues [17] developed a social simulation tool as a backend to interactive pedagogical drama applications. While the development of social M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 124–133, 2005. c Springer-Verlag Berlin Heidelberg 2005
Social Navigation Behaviors of Virtual and Real Humans
125
relationships in the approach by Prada and Paiva [15] is mainly determined by the type of social interactions between them, Marsella and colleagues regard the beliefs of agents about other agents as a key factor of social interaction and rely on a theory of mind to explicitly represent the beliefs of agents about other agents. Commercial games, such as TheSims, show that the simulation of social skills may render interactions between virtual characters more believable and engaging. In the systems described above, social behaviors are mainly reflected by the agents’ communicative behaviors. In contrast, Thalmann and colleagues [5] concentrate on the simulation of social navigation behaviours in virtual 3D environments including the social avoidance of collisions, intelligent approach behaviours, and the calculation of suitable interaction distances and angles. The work is based on an operationalization of empirically-grounded theories of human group dynamics, such as Kendon’s group formation system [9]. The objective of our work is to investigate navigation behaviors of humans that socially interact with virtual agents. Unlike Thalmann and colleagues [5] who focus on the simulation of human navigation behaviors using virtual agents, we involve the users as active participants in the scenario. In particular, we allow them to freely navigate through the scenario and join or leave groups of other agents by making use of a commercially available dancing pad that is employed in many computer games. As a consequence, interaction requires the users to move their full body in the physical space. Our approach is closely related to projects that make use of proxemics as an interaction parameter. An example of a proximity-based game includes Pirates [4]. In this game, possible actions are triggered if the player is close to a certain location. For instance, to attack another player, one has to be physically close to the other player, otherwise this option is not available. Another proximity-based game is MirrorSpace by Roussel and colleagues where the display of information is affected by the distance of the user to the screen [20]. Partala and colleagues studied the influence of an agent’s proximity on the affective response of the user [14]. Whereas valence and arousal seemed not to be influenced by proximity variances, a significant effect was found for the dominance dimension. In the next section, we first present a model for group dynamics that simulates changes in the social relationships between agents as a side effect of social interactions. Based on the model, we designed and implemented a game-like scenario which has been employed as a test bed to study social navigation behaviors of humans interacting with virtual agents. The model is augmented by a model of proxemics that simulates the role of distance and spatial orientation in human-human communication. By means of proxemics, a group of human participants may signal other humans whether they welcome new group members to join or not. The paper will report on the outcome of an experiment we conducted to investigate how humans respond to such cues when they are elicited by virtual agents. In particular, we were interested in the question of whether humans prefer to join open group formations as opposed to closed group formations.
126
2
M. Rehm, E. Andr´e, and M. Nischt
Description of the Underlying Model
Inspired by work on social psychological theories of group behaviors, we have designed and implemented a computational model of group dynamics. 2.1
Representation of Interpersonal Relationships
The profile of the single agents is characterized by their name, gender, marital status, age group, social status, sex orientation and personality. An agent’s personality is represented by a vector of discrete values along a number of psychological traits, such as extraversion or agreeableness, that uniquely characterize an individual [11]. Furthermore, the model is based on an explicit representation of the relationship between single agents, the relationship between agents and the groups they belong to and the attitude of agents towards objects. Interpersonal relationships are described by the degree of liking, familiarity, trust and commitment. These values are either specified by the user in advance or derived from known properties of the agent’s profile. For instance, agents with a similar social status are considered to trust each other more than agents with a different social status. The social role of an agent within a group is described by features, such as its power, prestige and affiliation. 2.2
Development of Interpersonal Relationships
To model how social interactions influence the social relationships between agents, we start from Interaction Process Analysis (IPA) [2]. IPA is based on a classification of social interactions that take place in small groups and essentially distinguishes between socio-emotional factors that refer to the social relationships within a group, such as positive or negative feedback to group members, and task-oriented factors that refer to group tasks, such as asking questions or summarizing and offering direction. It has already been successfully employed in other system of social group dynamics, see for example [5] or [15]. To determine the type and number of social interactions, we use ideas from social impact theory [8] which defines how the presence of others influences one’s behavior. The strength of this influence is calculated in a close analogy to physical phenomena, such as the amount of light visible on a table which depends on the number of light sources, their distances to the table and their strength. The social impact on a target person is calculated taking into account the strength, immediacy, and number of source persons where strength comprises features, such as status or power, and immediacy represents the physical distance between source and target. As any of these factors increases, the impact on the target also increases. For instance, if a subject has to perform a song, stage fright increases with the number of people in the audience and their status. It decreases if the subject has not to perform alone [7]. Furthermore, we were inspired by self attention theory [12]. Self attention theory is a theory of self-regulation that explains behavior modifications if one
Social Navigation Behaviors of Virtual and Real Humans
127
Fig. 1. Orientation variants of two agents (left) and distance zones for an agent (right)
is the subject of one’s own attentional focus. In this case, violations of standard social norms will be more salient. People’s peer groups influence self attention and regulation in at least two different ways. Behavioral standards will be set by the group to which the individual has to adhere on the one hand. On the other hand, group size matters. Larger groups result in decreased self awareness because single individuals will easier go unnoticed. The so called other-total ratio is used to describe this effect for the interaction between arbitrary groups. It represents the proportion of the total group that is comprised of people in the other subgroup. The influence between the social configuration of a group and the number and type of interactions is bi-directional. To describe how the nature of the interactions influences the development of interpersonal relationships between group members, we follow ideas by Schmitt [21] and make use of the “Congruity Theory” by Osgood and Tannenbaum [13]. The theory is based on the hypothesis that people tend to avoid unbalanced configurations or cognitive dissonances. For instance, when a statement of a speaker causes such an unwanted state in the addressees, they will either change their attitudes towards the subject matter, or their attitudes towards the speaker who caused a dissonance. As a consequence, the theory allows us to describe changes in social relationships as a side effect of interactions between agents. 2.3
Social Navigation
As noted earlier, we make use of proxemics as an interaction parameter. According to Hall, proxemics is the investigation of man’s spatial behavior which is of importance because “space is one of the basic, underlying organizational systems for all living things – particularly for people” [6]. One determinant of such spatial behavior is the acceptable distance between communication partners which influences their spatial interaction behavior. Hall
128
M. Rehm, E. Andr´e, and M. Nischt
Fig. 2. Interacting agents
[6] distinguishes four different distances that are related to behavior changes which occur if someone enters these distance zones. Intimate distance ranges up to 45 cm and is reserved for very close relationships. Personal distance ranges from 45 cm to 1.2 m which allows touching and is thus reserved for focused and private interactions. Social distance ranges from 1.2 m to 3.6 m, public distance starts at 3.6 m. The specific distances given by Hall are valid for white Northern Americans. Whereas the existence of the four different distances seems to be universal, the ranges itself and acceptable behaviors related to the distance zones are culture specific. Relying on Hall, Knowles [10] has developed a straightforward model based on proximity to handle the effects of crowding. In essence, the influence of others results from a combination of their number and their distance from the target person. Whereas Hall’s analysis is primarily concerned with distances, Kendon takes also the orientation of the interlocutors into account. Based on [9], a model for the interaction between pairs of people was integrated. Depending on their interpersonal relations, people will orient themselves differently when joining others in public places. The six most frequent orientations for communicating pairs were taken into account (see Fig. 1 left). Half of these are so called closed (upper line), the other half open orientations (lower line). Closed orientations indicate that the interlocutors won’t be disturbed by other people, whereas interloctors in open orientations will allow others to enter the conversation.
3
The Beergarden Scenario
To test our model, we have decided to implement a virtual beergarden where agents wander around to meet friends or to build up new relationships. The initial position of the agents’ in the scenario is randomly assigned by the system. When the system is started, the agents perform a random walk until they perceive another agent or a group they wish to start a conversation with.
Social Navigation Behaviors of Virtual and Real Humans
129
Figure 2 shows two different pairs of agents interacting with each other. On the left hand side a task-oriented interaction is depicted. The female agent brought up the topic of a car which is indicated by the car icon. The male agent gives his positive opinion on this topic indicated by the icon as well as his gesture (thumbs up). The pair on the right of Figure 2 finds itself in a negative social emotional interaction where the male agent shows antagonism indicated by the skull icon and his rude gesture (showing a fist). The female agent shows tension by “shouting” (indicated by the jagged outline of her speaking bubble), producing the lightning icon and putting her hands on her hips. The agents’ behavior is determined by a de-centralized behavior control mechanism that relies on rules derived from the theories described in Section 2. The left-hand side of a rule specifies a condition that has to be fulfilled to apply a rule, the right-hand side either refers to an elementary action or a complex behavior script that indicates what a character should do in a certain situation. For instance, we have defined several greeting scripts consisting of distance salutation, approach and close salutation based on [9]. To elicit a greeting script, a character must sight another character, identify it as someone it wishes to greet and believe that the other character is aware of it. At runtime, scripts are decomposed into elementary actions that are forwarded to the animation engine. For each elementary action, we have modeled a set of specific animation sequences. For the IPA actions, we modeled a number of postures and gestures relying on the descriptions in the Berlin dictionary of everyday gestures (”Berliner Lexikon der Alltagsgesten”, [3]). The current animation engine is built upon Managed DirectX to efficiently render the visual content of a scene and access the programmable pipeline stages of modern GPUs. As the individual characters lie in the main focus, their geometry can be modified using a variety of techniques, including skeletal-subspacedeformation (a.k.a. vertex skinning) and morph targets. This allows us to simulate body motions and gestures based on forward/inverse kinematics as well as to represent mimics by state vectors, which easily can be interpolated. Moreover, a fast and configurable path finding routine tells the virtual agents how to reach a certain place while distances to the others are maintained depending on their social relationships. In order to clearly separate the agents’ animation engine from the underlying behavior control mechanism, all types of animations can be triggered from an application-independent interface, which comprises a perception model, encapsulating qualitative information about spatial relations based on the position and orientation of the agent. 3.1
User Study
The Beergarden allows us to perform controlled experiments to shed light on the question of how users respond to encounters with agents that follow the aforementioned social navigation behaviors. Here, we will report on an experiment we conducted to find out whether users feel more encouraged to join open group formations as opposed to closed group formations. Furthermore, we investigate what social distance they maintain when deciding to join a group.
130
M. Rehm, E. Andr´e, and M. Nischt
Fig. 3. Joining a group of agents
According to the media equation [18], it should not make a great difference whether humans approach a virtual agent or another human. Indeed a number of studies by Bailenson and colleagues [1] revealed that the size and the shape of the personal space around virtual humans resembled the shape of the space that people leave around real, nonintimate humans. Nevertheless, Bailenson and colleagues also observed that the social navigation behavior depends on whether the agent is controlled by a computer or by a real human. In particular, they found out that people gave an avatar more personal space than an autonomous agent. In a second experiment, they examined what happens when virtual humans walk and violate the personal space of the participants. The experiment showed that human subjects avoided the virtual human more when it was obviously computer-controlled than an avatar that was controlled by another human. To investigate these questions, the following experimental setting was set up (see Fig. 3). The user was enabled to navigate by means of a pressure-sensitive dancing pad and with a first person view through the beergarden. The choice of the dancing pad instead of e.g., a joy pad or the keyboard, made sure that subjects were involved with their full body in the experience of navigating through the beergarden. Because the general scenario described above would have been to unconstrained, the number of agents was restricted to two groups with two agents each. The subjects were instructed to join one of the groups present in the beergarden. The graphical icons denoting the communicative meaning were disabled to prevent users from choosing a group on the basis of the communicative content. Each scenario consisted of two groups of agents where one group was positioned in an open (L,C,I), the other in a closed form (H,N,V). The nine
Social Navigation Behaviors of Virtual and Real Humans
131
Table 1. Orientation of agents in groups joined by subjects Conf. H N V C L I
# 11 3 20 52 72 58
% Class # % 5.1% 1.4% closed 34 16% 9.3% 24.1% 33.3% open 182 84% 26.9%
Table 2. Distance of subjects to agents Distance intimate personal social public close far # 0 0 7 5 0
resulting combinations were tested twice with each subject, changing the left and right position of the groups. To control for gender differences, each group was composed of one female and one male agent. For each subject, the 18 scenarios were presented in random order. 10 students and 2 staff members from the computer science department participated in this experiment ranging in age from 22 to 35. Nine subjects were male and three female. Ten of the subjects had a German, two an Arabic cultural background. This is of interest because studies in cross-cultural communication (e.g., see [22] and [6]) show that the proxemic behavior of these two cultures is noticeably different. Subjects were instructed to join one of the groups in each scenario. Prior to the experimental session, the subjects had the opportunity to familiarize themselves with the dancing pad and the beergarden environment which they could freely explore as long as they liked. Results. In 84% of the scenarios, the subjects joined the group which was positioned in an open formation. Two of the subjects never navigated to one of the other formations, another two chose closed formations in a third of the time (see Table 1). A paired-samples t-test showed that this difference is significant for p < 0.01. In 20 from the remaining 34 scenarios in which a closed formation was chosen, it was the V-formation which can sometimes be mistaken for a less open C-formation. The result for the proxemic behavior of the subjects is non-ambiguous (see Table 2). For all scenarios, subjects positioned themselves in a social distance which has a range between 1.2 m to 3.6 m with a close area from 1.2 m to 2.1 m and a far area ranging from 2.1 m to 3.6 m. From the German subjects, five positioned themselves in the far area and five in the close area. In general, such a behavior also depends on the personality of people [6] and is influenced by their cultural background. We did not test the personality influence in this study. With respect to the cultural background parameter, we observed the following.
132
M. Rehm, E. Andr´e, and M. Nischt
The two Arabic subjects positioned themselves in the close area. With only two subjects with this cultural background, not very much can be said about this effect - only that they positioned themselves as predicted. One of the subjects was well aware of the cultural differences in proxemic behavior because during the experiment he explained that he does not move closer because he didn’t want to terrify the guys (which he considered to have a different cultural background due to their appearance).
4
Conclusions
In this paper we described a multi agent system in which the development of interpersonal relationships as well as the control of interactions is based on models of social group behavior. To integrate the user in this system and allow her to freely interact with the agents we devised a first experiment to shed light on the question of how the proxemic behavior of users in this scenario adheres to results found in the literature. By navigating through the environment employing a dancing pad subjects experienced full body movements. So far, the reciprocal interactions between the virtual humans and the humans are still very limited since the model described in Section 2 was mainly used to simulate the social group dynamics of virtual humans (as opposed to virtual AND real humans). Nevertheless, the positioning behavior of the human subjects can be described as natural. They joined open group formations of agents significantly more often than closed ones and positioned themselves in a social distance. Thus, the first step of integrating the user was successful. During the experiment, some of the male subjects expressed their preferences for the female agents. All groups were composed of one male and one female agent to control for such possible gender differences. Next, the users will be confronted with purely male and female groups to test for this difference which might well override a choice based solely on open vs. closed formations. We will also look more closely at the interaction angles of the humans which we recorded in the experiment, but which were not yet analyzed. Finally, we will investigate the influence of non-verbal agent behaviors, such as looking at the users. A different line of research will concentrate on modelling more sophisticated proxemic behavior in agents. Hall [6] gives some insights in the relationship between distance and non-verbal behavior which changes according to the distance between interlocutors. For instance, the farther away interlocutors are from each other, the louder they speak, starting from whispering in an intimate distance to shouting in a public distance.
References 1. J. N. Bailenson, J. Blasovich, A. C. Beall, and J. M. Loomis. Interpersonal distance in immersive virtual environments. Personality and Social Psychology Bulletin, pages 819–833, 2003. 2. R. F. Bales. Interaction Process Analysis. Chicago University Press, Chicago, 1951.
Social Navigation Behaviors of Virtual and Real Humans
133
3. BLAG. Berliner lexikon der alltagsgesten. http://www.ims.uni-stuttgart.de/ projekte/nite/BLAG/, last visited: 22.03.2005. 4. Jennica Falk, Peter Ljungstrand, Staffan Bj¨ ork, and Rebecca Hansson. Pirates: Proximity-Triggered Interaction in a Multiplayer Game. In Proceedings of CHI, pages 119–120, 2001. 5. A. Guye-Vuilli`eme and D. Thalmann. A high level architecture for believable social agents. Virtual Reality Journal, 5:95–106, 2001. 6. Edward T. Hall. The Hidden Dimension. Doubleday, 1966. 7. J. M. Jackson and B. Latan´e. All alone in front of those people: Stagefright as a function of number and type of coperformers and audience. Journal of Personality and Social Psychology, (40):72–85, 1981. 8. Jeffrey M. Jackson. Social Impact Theory: A Social Forces Model of Influence. In Brian Mullen and George R. Goethals, editors, Theories of Group behavior, pages 111–124. Springer, New York, Berlin, 1987. 9. Adam Kendon. Conducting Interaction: Patterns of Behavior in Focused Encounters. Cambridge Univ Press, Cambridge, 1991. 10. E. S. Knowles. Social physics and the effects of others: Tests of the effects of audience size and distance on social judgements and behavior. Journal of Personaliy and Social Psychology, (45):1263–1279, 1983. 11. R. R. McCrae and O. P. John. An introduction to the five factor model and its applications. Journal of Personality, (60):175–215, 1992. 12. Brian Mullen. Self-Attention Theory: The Effects of Group Composition on the Individual. In Brian Mullen and George R. Goethals, editors, Theories of Group behavior, pages 125–146. Springer, New York, Berlin, 1987. 13. C. E. Osgood and P. H. Tannenbaum. The Principle of Congruity in the Prediction of Attitude Change. Psychological Review, (62):42–55, 1955. 14. Timo Partala, Veikko Surakka, and Jussi Lahti. Affective Effects of Agent Proximity in Conversational Systems. In Proceedings of NordCHI, pages 353–356, 2004. 15. Rui Prada and Ana Paiva. Intelligent virtual agents in collaborative scenarios. In Proceedings of Intelligent Virtual Agents (IVA), pages 317–328, 2005. 16. Helmut Prendinger and Mitsuru Ishizuka. Social Role Awareness in Animated Agents. In Proceedings of Agents ’01, Montreal, Canada, pages 270–277, 2001. 17. D. V. Pynadath and S. C. Marsella. PsychSim: Modeling Theory of Mind with Decision-Theoretic AgentsM. In Proceedings of the Fifteenth IJCAI. Morgan Kaufman Publishers Inc., 2005. to appear. 18. Byron Reeves and Clifford Nass. The Media Equation — How People Treat Computers, Television, and New Media Like Real People and Places. Cambridge University Press, Cambridge, 1996. 19. Thomas Rist and Markus Schmitt. Applying socio-psychological concepts of cognitive consistency to negotiation dialog scenarios with embodied conversational characters. In AISB Symposium on Animated Expressive Characters for Social Interactions, 2002. 20. Nicolas Roussel, Helen Evans, and Heiko Hansen. Utilisation de la distance comme interface ` a un syst`eme de communication vid´eo. In Proceedings of IHM, pages 268– 271. ACM, 2003. 21. Markus Schmitt. Dynamische Modellierung interpersoneller Beziehungen zwischen virtuellen Charakteren. PhD thesis, Universit¨ at des Saarlandes, Saarbr¨ ucken, 2005. 22. Stella Ting-Toomey. Communicating Across Cultures. The Guilford Press, New York, 1999.
Interacting with a Virtual Rap Dancer Dennis Reidsma, Anton Nijholt, Rutger Rienks, and Hendri Hondorp Human Media Interaction Group, Center of Telematics and Information Technology, PO Box 217, 7500 AE Enschede, The Netherlands {dennisr, anijholt, rienks, hendri}@ewi.utwente.nl http://hmi.ewi.utwente.nl/
Abstract. This paper presents a virtual dancer that is able to dance to the beat of music coming in through the microphone and to motion beats detected in the video stream of a human dancer. In the current version its moves are generated from a lexicon that was derived manually from the analysis of the video clips of nine rap songs of different rappers. The system also allows for adaptation of the moves in the lexicon on the basis of style parameters.
1 Introduction “Rapping is one of the elements of hip hop as well as the distinguishing feature of hip hop music; it is a form of rhyming lyrics spoken rhythmically over musical instruments, with a musical backdrop of sampling, scratching and mixing by DJs.” (Wikipedia). Hip hop includes break dancing. Well-known are break dance moves as the six step or the head- and hand-spin. Rappers also move; they move to the rhythm and the lyrics, performing hand, arm and bodily gestures. But most of all there are lyrics with content and form that characterize rap. Rap and hip hop have been studied by scholars. The usual viewpoints are ethnographic, cultural, social and geographic [1]. Time, place and cultural identity are considered to be important aspects that need to be studied and obviously, the lyrics invite to look at race issues and at the rage, the sexism and the dislike of authorities that make the message. The rap lyrics, and that brings us closer to hand and bodily gestures, also invite us to study the miming of meaning in words and phrases (iconicity) [2]. In this paper we look at a hardly studied phenomenon of rap performances: the series of gestures and bodily movements that are made by rappers while performing. It is not meant as a study of a rapper’s movements and gestures from the point of views of the issues mentioned above. Our modest study consists of observing various rappers with the aim of distinguishing characteristic movements and regenerating them in an interactive and entertaining virtual rapper. Clearly, since the lyrics contain an enormous amount of violence, F-words, sexual references, obscenities and derogatory terms for women, one may expect that movements and gestures will reflect that. This is certainly true for iconic gestures that appear in series of gestures and movements. For example, hand signs can indicate gang (Crips, Brims, Bishop, etc.) or certain attitudes (f*ck you) and obscene hand gestures accompany the obscenities in the lyrics. How culture reflects in movements has not yet been studied. Rappers or rap-fans M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 134 – 143, 2005. © Springer-Verlag Berlin Heidelberg 2005
Interacting with a Virtual Rap Dancer
135
make distinctions between East, North, West and South, but they as easily apply it to a continent, a country or a city. As mentioned, we concentrated on the recognition of the characteristics of the sequences of gestures and movements rappers are making. Can we distinguish different styles, can we distinguish and characterize different rap gestures and body movements and can we model and animate them in a virtual rap dancer? These questions have a much more general background. This background is about the design of creating autonomous embodied agent whose behavior is influenced with continuous real time input and feedback to what it perceives and is able to present through its different sensor and output channels. Its sensor channels may exist of video and audio channels, but we may also think of other, preferably non-obtrusive sensors, like chips, tags, and wearables. Hence, in such a situation, the main issues in the behavior of an embodied agent are the coordination of its behavior with the sensor input with appropriate timing and the selection and execution of the behavior in its own style. That is, there need to be fusion of information coming from different media sources to such an agent and there needs to be fission of information to be presented by an embodied agent. Again, this is a preliminary study. We looked at the behavior of rappers. We extracted and analyzed their movements from video-clips. We imitated these movements, made modest attempts to distinguish styles, animated these movements, and designed and build a system that uses our findings in two different ways. Firstly, there is the design of a virtual dancer that moves along with the music of a performing rapper. This dancer, a simple VRML avatar, globally follows the music and retrieves its movements and gestures from a database. A slightly more sophisticated system has been obtained where the virtual rap dancer gets its input not from a real-time analyzed rap song, but rather from input from microphone and camera, that allows a human user to steer the rap movements of the virtual dancer. Hence, in this paper we present a virtual dancer that is able to dance to the beat of music coming in through a microphone and to motion beats detected in the video stream of a human dancer. In the current version the moves of the dances are generated from a lexicon that was derived manually from the analysis of the video clips of nine rap songs of different rappers. The system also allows for adaptation of the moves in the lexicon on the basis of style parameters.
2 Related Work Above we gave a general characterization of our work. That is, we have an embodied agent that is able, through its sensors (audio, vision, keyboard, and maybe others) to sense the environment and possible interaction partners. Moreover, it has means to display its understanding of the communication situation, admittedly in a very limited way, that is, by displaying (nonverbal) rapping behavior in an embodied agent. Embodied agent design and interaction design with embodied agents has become a well-established research field ([3,4,5]). Hence, when looking at related research we can take the view of humans communicating with embodied agents, where, in ecommerce application, the agents have to sell and demonstrate products, where the agents have to guide a visitor in a virtual or augmented reality environment, or where
136
D. Reidsma et al.
the agent plays a role in a training, education, entertainment, or simulation environment. Our aim, to accept nonverbal input and to transform this input to nonverbal output, is not essentially different from much of ongoing recent research, but it certainly differs from regular research because of its emphasis on nonverbal input in the form of music, beats and physical movements, and its nonverbal output in the form of series of rap gestures and body movements in virtual reality. Hence, here we are more interested in interactive systems that are able to provide various kinds of feedback to expressivity in music, dance, gestures and theatretical performance and that view has been chosen to look at related work. Dance, music and how to interact with computerized dance and music applications are the most important issues we address in the following subsection. In the section on conclusions and future work (Section 5) we look at other related work that we consider being important for the further development of our virtual rap dancer. 2.1 Music, Dance and Interaction: Some Previous Research There have been many research and art projects where movements of dancers or players are captured by motion capture sensors or cameras. The movements, together with other input signals (e.g., speech, facial expressions, and haptic information) can be analyzed, manipulated and mapped on avatars, (semi-) autonomous agents, robots, or more abstract audio and visual representations. This mapping or re-generation can be done in real-time, allowing applications such as interactive theatre and virtual storytelling (see [6,7] for some pioneering work), or off-line, allowing more advanced graphics and animation, but less direct interactive applications. These latter applications are for example the simulation of traditional Japanese [8] or baroque dances [9] with virtual characters and the generation of new dances composed from extracted primitive dance movements and newly made or learned new choreographies. Attempts to identify emotions expressed in dance or act movements and choreography can also be made part of the mapping from dance to a re-generation. There are also many examples of music interfaces. Music is used as input and analysis and interpretation allows for applications for education, entertainment and cultural heritage. Previously we have looked at recognizing and distinguishing percussion instruments and music visualization by an embodied performer [10]. The performer is a 3D animation of a drummer, playing along with a given piece of music, and automatically generated from this piece of music. The input for this virtual drummer consists of a sound wave that is analyzed to determine which parts of the percussion instruments are struck at what moments. The Standard MIDI File format is used to store the recognized notes. From this higher-level description of the music, the animation is generated. In an interactive version of this system we use a baton-based interface (using sensors on the tip of two drumsticks), as has been done by many others. Goto et al. [11] introduced an embodied agent that enables a drummer and a guitarist connected through Ethernet not only to musically interact with each other, but also, by observing this virtual character, through the animations of the character. Motion timing for the character comes from the drum, performed dance motions are chosen from the improvisations of the guitarist. A jam session system that allows a human guitarist to interplay with (non-embodied) virtual guitar players can be found in [12]. Different reaction models for human players can be obtained and imitated. An
Interacting with a Virtual Rap Dancer
137
example of a system that extract acoustical cues from an expressive music performance in order to map them on emotional states is CUEX (CUe EXtraction) [13]. This system has been interfaced with Greta, an embodied conversational agent in which the emotional states obtained from the music are transformed to smoothly changing facial expressions providing a performer with visual feedback [14]. In [15] a system is described that allows a musician to play a digital piano and sing into a microphone, from which musical features are extracted (pitch, specific chords), and responsive behavior in synthetic characters is displayed in real-time. A cognition layer of their system incorporates rules about the relationship between music and emotion. In the expression layer emotions are translated to parameters that guide the character’s animations [16]. Well known research on extracting emotions from dance, body movements and gestures is performed by Camurri (see e.g., [17,18]).
3 Analysis of Rap Gesture Sequences Various rap-video clips have been analyzed by our students1, characteristic movements have been extracted and, in order to get more feeling for them, exercised. The following music video clips were selected for further study: • • • • • • • •
Dr. Dre - Forgot about Dre (Westside) Jay Z - Big Pimpin’ (Eastside) Xzibit - Paparazzi (Westside) Blackstreet & Dr. Dre - No diggity (Westside) Westside connection - Gangsta nation (Westside) Erick Sermon, Redman & Keith Murray - Rappers Delight (Eastside) LL Cool J - Headsprung KRS One - Hot (Eastside)
Frame by frame movements in these clips have been studied and positions of limbs and body of performers have been extracted. One of the questions that we hoped to answer is whether different bodily gesture styles can be distinguished in rap music and, if so, how we extract features to recognize them. It turned out that although rappers repeat their own movements and gestures, different rappers very much have their own style. One global observation, not based on many data, is that Westside rappers generally move more aggressive, faster and more angular, while Eastside rappers move more relaxed. In Table 1 we mention the 14 rap movement sequences that have been distinguished in the clips and that have been selected for a database from which our virtual rapper is fed. Table 1. Fourteen sequences of bodily gestures observed from rap video clips
Yo! Get down! Drive by Magic Feet Yo Hallelujah 1
Bombastic Hit your Cap .45 Wave, get down
Bitching ho no! Hold me Lullaby
D.F. van Vliet, W.J. Bos, R. Broekstra and J.W. Koelewijn.
Cross Again Kris Kros Wuzza
138
D. Reidsma et al.
Fig. 1. Bitching ho no!
Fig. 2. Kris Kros
All movement sequences were studied, exercised and photographed, requiring, depending on the complexity of the movement, five to eighteen positions to be distinguished. In Figure 1 we illustrate the rather simple ‘Bitching ho no!’ movement sequence. Another, more complicated movement sequence (‘Kris Kros’) is
Interacting with a Virtual Rap Dancer
139
illustrated in Figure 2. Detailed verbal descriptions for these movement sequences have been made in order to allow corresponding animations of an avatar. For example, the Bitching ho no! sequence is a movement sequence where only the right arm is used. First the right hand is moving at head height from the right of the head to the head, making three duck quack movements, next the right hand index finger turns around on the right-hand side of the head. All sequences, with detailed information can be found in [19].
4 Architecture of the Virtual Rap Dancer The (real time) architecture of the virtual rap dancer consists of several main parts. The sensor channels analyze incoming audio and video in order to detect beats in each separate channel to which the dancer can time its dance moves. The beat predictor module combines the different streams of detected beats, trying to merge beats that were detected in two different channels at the same time into one beat and trying to predict when a next beat is most likely to occur. This prediction is then used by the motion controller that will plan a next dance move in such a way that its focus point will coincide with the predicted next beat. The animation system finally will execute the planned movements after adapting them to some style parameters. The full architecture has already been implemented. The separate modules though are still in a first stage: each module will in the future be extended to achieve a more advanced system. 4.1 Analyzing and Combining Input There are two types of modules that analyze the input (though of each type, a larger number of instances can be present in a running system). One takes an audio signal from the microphone; the other takes a video signal from a camera. Both modules attempt to recognize beats in the input, albeit in a very simple way in this version of the system. As soon as a module recognizes a beat, it will send out a BeatEvent. The audio system does this largely based on the energy of the audio signal. Clearly, vocal percussion is also possible and, similar as in the first (non-interactive) version of the system, any rap song can be given as input. The video system tracks the face and hands of the person in the camera view (based on the Parlevision system2 described in [20]), and will recognize beats based on the hands or face crossing (implicit) trigger-lines in the image. Two incoming beats that are too close together are assumed to be the same beat recognized by different sources. A simple beat-prediction algorithm takes the time-span between the previous beat (from any source) and this beat, and uses that to predict when a next beat is expected. In Figure 3 we show both the audio and video input and manipulation by the system.
2
http://hmi.ewi.utwente.nl/showcase/parlevision/
140
D. Reidsma et al.
Fig. 3. Microphone tap input (left) and dance movement analysis (right) for beat detection
4.2 Generating Dance Moves from the Input and Some Style Parameters The animation module uses a database of rap-dance moves which was constructed from the analysis of example dancers. A large number of rap-clip videos were analyzed. Standard recurring moves were stored in a database by extracting joint angles for key frames, plus some extra information such as the type of rap music that this moved was most often used in and some information about which key frames are to be performed on a (musical) beat (focus key frames). The movement controller selects the next moves from the database to be executed based on the style parameter indicating what type of dance the user wants to see. These moves are then planned using the information about which key frames are to be aligned to musical beats, the incoming beat information, and the predicted next beat. If the beat predictor is able to successfully predict the next beat the movement planner will cause the dancer to be exactly at a focus key frame when that next beat occurs. Furthermore the dance moves are, before execution, modified with a style-adaptation. At the moment the only style adaptations that are actually implemented are one that
Fig. 4. Style settings for the virtual rap dancer
Interacting with a Virtual Rap Dancer
141
causes the dancer to dance sharper on the beat or more loosely around the beat and one that modifies the shape of the interpolation between key frames. In the future the system should also allow adaptation of more formation parameters. The dance moves are animated using the animation package developed at HMI, used before in [21]. Summarizing, presently the user can interact with the system by dancing in front of the camera, giving music input to the microphone or changing the style settings (for now, move-selection parameters and sloppiness of execution). See Figure 4 for the interface with sliders.
5 Evaluation Although the work described in this paper is sill in a very early stage, some evaluative remarks can already be made. In the first place, the beat-boxing interface using the microphone allows us to perform a (simplistic) qualitative evaluation of the beat recognition and the movement synchronization. Tapping on the microphone in a regular measure does indeed lead to the virtual dancer executing moves in that same measure. Tapping faster or slower causes the dancer, as one would expect for such a simple test, to move faster or slower in the same amount. When music is used as input, it is harder to give a good evaluation of the rhythmic quality of the movements. In general, strong beats are recognized better than subtle rhythmic movements. This means that music with a lot of drum and bass in it works better than, say, classical music.
6 Conclusions and Future Work The architecture as described above has been fully implemented3. It works in real time, integrating microphone input, animation, and the video input (running on a separate computer). Future work will consist, among other things, of improved beat extraction, both from audio [22] and video [8], better beat prediction and fusion of different streams of beat extraction [23], looking at the relationship between joint angles and beats. Results from music retrieval, e.g. query by beatboxing [24], can be used to have the virtual rap dancer choose her movements based on vocal percussion. The obvious next step, as should become clear from our discussion of related work, is to include more interaction between the virtual rap dancer and its human partner, by extracting motion primitives and looking at the expressiveness of bodily gestures (related to the rap) and make a translation to emotion primitives that impact the choice of bodily gestures, the hand movements and the expressiveness of the virtual rap dancer. While one the one hand the rapper can learn from its human partner and adapt its movements to those of the human partner, on the other hand, the virtual rap dancer can act as a teacher to its human partner. This would mean a great step closer to a real ‘joint dancing experience’. The main challenge there would be to adapt the algorithms, developed for the very precise movement and posture capturing methods, to the much less precise motion and posture recognition from camera images. More inspiring creative ideas about possible interaction between rappers and a VJ can be found in a clip of video director Keith Schofield: “3 Feet Deep” of a rap performed by DJ Format. 3
See http://hmi.ewi.utwente.nl/showcase/v-rapper/ for a live demonstration of the result.
142
D. Reidsma et al.
Acknowledgements The first version of the virtual rap dancer, which did not include interactivity, was made by four of our students (D.F. van Vliet, W.J. Bos, R. Broekstra and J.W. Koelewijn). They were also responsible for the analysis of the video clips and the identification of the 14 rap movement sequences. Ronald Poppe helped us with his image processing software to identify beats in the movements of the human dancer interacting with the rapper. Dennis Hofs and Hendri Hondorp took care of audio processing software that allows beatboxing.
References 1. T. Rose. Black Noise: Rap Music and Black Culture in Contemporary America. Reed Business Information, Inc. (1994) 2. Attolino, P. Iconicity in rap music: the challenge of an anti-language. Presentation at Fifth International Symposium Iconicity in Language and Literature, Kraków (2005) 3. Ruttkay, Zs., Pelachaud, C. From Brows to Trust. Evaluating embodied conversational agents. Kluwer Academic Publishers, Dordrecht Boston London (2004) 4. Payr, S., Trappl, R. (Eds.). Agent Culture. Human-Agent Interaction in a Multicultural World. Lawrence Erlbaum Associates, Mahwah London (2004) 5. Prendinger, H., Ishizuka, M. (Eds.) Life-Like Characters. Tools, Affective Functions, and Applications. Cognitive Technologies Series, Springer-Verlag Berlin Heidelberg New York (2004) 6. Tosa, N., Nakatsu, R. Emotion recognition-based interactive theatre –Romeo & Juliet in Hades -. Eurographics ’99, M.A. Alberti, G. Gallo & I. Jelinek (Eds.) (1999) 7. Pinhanez, C., Bobick, A. Using computer vision to control a reactive graphics character in a theater play. Proceedings ICVS ’99 (1999) 8. Shiratori, T., Nakazawa, A., Ikeuchi, K. Rhythmic motion analysis using motion capture and musical information. IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (2003) 89–94 9. Bertolo, M., Maninetti, P., Marini, D. Baroque dance animation with virtual dancers. Eurographics ’99, M.A. Alberti, G. Gallo & I. Jelinek (Eds.) (1999) 10. Kragtwijk, M., Nijholt, A., Zwiers, J. An animated virtual drummer. International Conference on Augmented, Virtual Environments and Three-dimensional Imaging (ICAV3D), V. Giagourta and M.G. Strintzis (eds.), Mykonos, Greece (2001) 319-322 11. Goto, M., Muraoka, Y. A Virtual Dancer “Cindy” - Interactive Performance of a Musiccontrolled CG Dancer, Proceedings of Lifelike Computer Characters '96 (1996) 65 12. Hamanaka, M., Goto, M., Asoh, H., Otsu, N. A learning-based jam session system that imitates a player’s personality model. Proceedings International Joint Conference on Artificial Intelligence (2003) 51-58 13. Friberg, A., Schoonderwaldt, E., Juslin, P.N., Bresin, R. Automatic real-time extraction of musical expression. International Computer Music Conference - ICMC 2002, San Francisco International Computer Music Association (2002) 365-367 14. Mancini, M., Bresin, R., Pelachaud, C. From acoustic cues to expressive ECAs. 6th International Workshop on Gesture in Human-Computer Interaction and Simulation. Valoria, Université de Bretagne Sud, France (2005) 15. Taylor, R., Torres, D., & Boulanger, P. Using music to interact with a virtual character. International Conference on New Interfaces for Musical Expression (NIME05), Vancouver, BC, Canada (2005) 220-223
Interacting with a Virtual Rap Dancer
143
16. Taylor, R., Boulanger, P., Torres, D. Visualizing emotion in musical performance using a virtual character. 5th International Symposium on Smart Graphics, Germany (2005) 17. Camurri, A., Lagerlöf, I., Volpe, G. Recognizing Emotion from Dance Movement: Comparison of Spectator Recognition and Automated Techniques. International Journal of Human-Computer Studies, 59(1-2) (2003) 213-225 18. Camurri A., Mazzarino B., Volpe, G. Analysis of Expressive Gesture: The EyesWeb Expressive Gesture Processing Library. In A. Camurri, G. Volpe (Eds.), Gesture-based Communication in Human-Computer Interaction, LNAI 2915, Springer-Verlag Berlin Heidelberg New York (2004) 19. Piage Day Project: Internal documentation. University of Twente (2004) 20. Poppe, R., Heylen, D., Nijholt, A., Poel, M. Towards real-time body pose estimation for presenters in meeting environments. International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision (WSCG’2005) (2005) 21. Nijholt, A., Welbergen, H., Zwiers, J. Introducing an embodied virtual presenter agent in a virtual meeting room. In Proceedings of the IASTED International Conference on Artificial Intelligence and Applications (AIA 2005) (2005) 579–584 22. Goto, M. An audio-based real-time beat tracking system for music with or without drumsounds. Journal of New Music Research, 30 (2) (2001) 23. Kim, T-h., Park, S., Shin, S.Y. Rhythmic-motion synthesis based on motion-beat analysis. ACM Trans. Graph., 22 (3) (2003) 392–401 24. Kapur, A., Benning, M., Tzanetakis, G. Query by Beatboxing: Music Information Retrieval for the DJ. Proceedings of the International Conference on Music Information Retrieval, Barcelona, Spain (2004)
Grounding Emotions in Human-Machine Conversational Systems Giuseppe Riccardi and Dilek Hakkani-T¨ ur AT&T Labs–Research, 180 Park Avenue, Florham Park, New Jersey, USA {dsp3, dtur}@research.att.com
Abstract. In this paper we investigate the role of user emotions in human-machine goal-oriented conversations. There has been a growing interest in predicting emotions from acted and non-acted spontaneous speech. Much of the research work has gone in determining what are the correct labels and improving emotion prediction accuracy. In this paper we evaluate the value of user emotional state towards a computational model of emotion processing. We consider a binary representation of emotions (positive vs. negative) in the context of a goal-driven conversational system. For each human-machine interaction we acquire the temporal emotion sequence going from the initial to the final conversational state. These traces are used as features to characterize the user state dynamics. We ground the emotion traces by associating its patterns to dialog strategies and their effectiveness. In order to quantify the value of emotion indicators, we evaluate their predictions in terms of speech recognition and spoken language understanding errors as well as task success or failure. We report results on the 11.5K dialog corpus samples from the How may I Help You? corpus.
1
Introduction
In the past few years, there has been a growing interest in the speech and language research community in understanding the paralinguistic channel in humanmachine communication. The paralinguistic component includes such information as speaker’s age, gender, speaking rate and state. In this paper we are going to address the latter, that is the emotional state of users engaged in goal oriented human-machine dialogs. In goal-oriented human machine communication the user might display different states due to prior conditions (e.g. previous attempts at solving the task) or poor machine cooperativeness in acknowledging and/or solving a problem (e.g. machine misunderstanding) or poor reward (e.g. the dialog is not successful). Prior conditions affect the user state in a way that can be detrimental or beneficial to the outcome of the interaction. Being able to detect users’ emotional state is crucial and would require to be able to know or estimate the profile of the user. Cooperativeness in a human-machine dialog allows the machine to elicit important task-related information at the early stages of the interaction M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 144–154, 2005. c Springer-Verlag Berlin Heidelberg 2005
Grounding Emotions in Human-Machine Conversational Systems
145
and/or resolve problematic turns due to speech recognition, understanding and language generation performance. The outcome of the interaction will impact the final user state which will probably re-emerge later in time. In this paper we analyze the role of emotions in human-machine dialogs by grounding them into the action-reaction traces of human-machine dynamics. Previous work emphasis has been on detecting and predicting emotion (positive/negative) states [1,2,3]. This paper is aimed at analyzing the effect of emotional patterns and its impact on machine reactions and performance. The fundamental issues we are going to address are: – The impact of the user state on machine’s dialog strategies. – The impact of the user state on machine’s performance. – The impact of the user prior conditions on future state transitions. The measures used to quantify the impact are done along two dimensions. The first is machine accuracy in recognizing, understanding and managing user’s spoken input. We show that user state has serious effect on the accuracy of stateof-the-art models which are based on statistical models (speech recognition and understanding) or hand-crafted machine action strategies (dialog manager). The second dimension is based on the success or failure of the user-machine dialog in accomplishing the user’s task. While the granularity of the first dimension is at the utterance or sub-dialog level the second dimension is a measure of the complete sequence of user-machine exchanges. As most of thestate-of-the-art spoken dialog systems, the current system is emotionless. Thus the analysis that will be carried out in this paper will aim at exposing the emotional component of the user state in such class of conversational machines. The ultimate goal is to provide a set of parameters or machine actions that could benefit from using emotion indicators. We provide a statistical analysis based on the How May I Help You? spoken dialog database [4]. The database includes transcriptions of spoken utterances, transcriptions of system prompts, semantic tags (user intent or calltype) estimated by the machine and labeled by a human, dialog acts and manually labeled emotion tags. In the following section we describe the database, its annotation labels and protocol. In Sections 3, 4, 5 we quantify the relations between emotion patterns, machine behavior and performance respectively.
2
The How May I Help You? Spoken Dialog System
“How May I Help You?SM ”, AT&T’s natural language human-computer spoken dialog system, enables callers to interact verbally with an automated agent. Users may ask for their account balance, help with calling rates and plans, explanations of bill charges, or identification of numbers on bills that they do not recognize. The machine is expected to understand their requests and route them to the correct information. If the system wants to confirm or clarify a customer’s response, the dialog manager asks for more information; if it is still not clear, it routes the caller to a service representative. Speech data from the deployed
146
G. Riccardi and D. Hakkani-T¨ ur
System: User: System: User: System: User: System:
How may I help you? I need to find out about a number that I don’t recognize. Would you like to look up a number you don’t recognize on your bill? Yes I would. Are you calling from your home phone? Yes I am. ... Fig. 1. Sample dialog from the HMIHY Corpus
“How May I Help You?SM ” system has been compiled into a corpus referred to as HMIHY [4,5]. Figure 1 presents the transcription of an example dialog from the corpus. In the HMIHY spoken dialog system the machine is trained to perform large vocabulary Automatic Speech Recognition (ASR) based on state-of-the-art statistical models [6]. The input spoken utterance is modeled as a sequence of acoustic and lexical hidden events (acoustic and language models). The word sequence output from the ASR module is then parsed to determine the user intent (calltype) using robust parsing algorithms [4]. This spoken language understanding (SLU) step provides a posterior probability distribution over the set of intents. While the speech recognition and robust parsing models are trained off-line, the posterior distribution is computed on-line from the spoken input. The posterior probabilities are used by the Dialog Manager (DM) to infer the most appropriate system dialog act. The algorithm used by the DM is heuristicbased and partially domain-dependent. The DM algorithm principle is designed to cope with ASR and SLU errors and converge to a dialog final state [7]. 2.1
System Performance Metrics
The ASR performance is evaluated using the standard word error rate (WER) measure. The SLU performance is evaluated using top class error rate (TCER), which is the percentage of utterances where the top-scoring calltype output of SLU is not among the true call-types labeled by a human. In order to evaluate the dialog level performance, three labelers labeled a 747 dialog subset of the HMIHY corpus with three labels: Task Failure, Task Success, and Other. The labelers were given the instructions to label each dialog with one of these labels, using the prompt transcriptions, user response transcriptions, system call-types, and human labeler call-types for each utterance. We used the first 100 dialogs in order to compare the errors and strategies among the three labelers, and converge to stable annotation guidelines. In the comparisons in this paper, we only use the following 647 dialogs. For the initial 100 dialogs, Cohen’s Kappa statistic was 0.42 for the three labelers, and for the final 647 dialogs it was 0.52, showing an improvement in the labeling guidelines.
Grounding Emotions in Human-Machine Conversational Systems
2.2
147
Corpus Description and Annotation
We have annotated the HMIHY corpus in two phases. In the first phase [8], 5,147 user turns were sampled from 1,854 HMIHY spoken dialogs and annotated with one of seven emotional states: positive/neutral, somewhat frustrated, very frustrated, somewhat angry, very angry, somewhat other negative, very other negative. Cohen’s Kappa statistic, measuring inter-labeler agreement, was calculated using this data set. A score of 0.32 was reported using the full emotion label set whereas a score of 0.42 was observed when the classes were collapsed to positive/neutral versus other. The small emotion label set is not equivalent to the larger one, but it provides us with more consistently labeled data. In the first phase we have encountered a high degree of variability (with respect to the number of labels) and unreliability (annotator agreement). In the second phase we have quantized the emotion labels into two labels, tokenized the corpus in terms of complete dialogs and increased the size of the corpus to 11,506 complete dialogs (40,551 user turns). Each new user turn was labeled with one of the emotion labels mentioned above. We used this expanded corpus labeled with positive versus negative user states for the experiments presented in this paper. In 8,691 dialogs, all turns are labeled as positive, and 2,815 dialogs have at least one turn labeled as negative. 35,734 of the user turns are labeled as positive, and the rest (4,817) of the user turns are labeled as negative.
3
Emotions and Machine Behavior
In modeling the user state s(t) we assume that there is a component dependent on prior conditions and a component dependent on the dynamic performance of the human-machine interaction. In the next sections, we investigate the relations between each component and user transcriptions, semantic and dialog annotations. For each component we evaluate the effect on system behavior and performance. The number of positive (negative) state labels in a complete dialog trace will be indicated with p (n). The value of a state label s(t) at time t (i.e., turn t + 1) is 0 (1) for positive (negative) labels. 3.1
Empirical Distributions over Time
As most of the current state-of-the-art spoken dialog systems, the HMIHY system is emotionless both from the input (detection) and output (generation) side. The DM representation of the user state is defined only in terms of dialog act or expected user intent. On the hand, in the following analysis we investigate how the observed emotional component of the user state impacts such a system. The first question that we address is on the state probability over time to be in a negative state. During the interaction the machine processes noisy input (e.g., speech recognition errors) and makes an estimate of the noise level. There are two types of DM strategies that would exploit this noise estimate at each turn (intent posterior probability). The first is to assume that the information
148
G. Riccardi and D. Hakkani-T¨ ur 25
Percentage Negative (%)
20
15
10
5
0
1
2
3
4 Turn Number
5
6
7
Fig. 2. Percentage of negative turns over time (turn number) within a dialog. The histogram is truncated at t = 7.
acquired is correct and act accordingly. The second is to assume the input is not correct (partially or totally) and apply an error recovery DM strategy. Fig 2 shows the percentage of spoken utterances with negative emotions over time (dialog turns). The monotonic increase of P (s(t) = 1) over the course of the dialog can be explained in two different ways. First, at each turn there is a non-zero probability of misrecognizing and/or misunderstanding the user1 . The system reacts to these errors by acting on it (e.g. using error recovery strategies) or ignoring them (e.g. asking the user to confirm the wrong intent). Second, there is a compounding effect on the user tendency to remain (with higher probability) in the negative state once it has been reached. This behavior shows user tolerance to system errors is not time independent. DM strategies should take into account the current user state, emotion indicators as well as its history. As we will see in section 4, user state is a good predictor of system performance as well. Thus we might expect that a rising percentage of negative turns might be due to highly correlated variables such as user tolerance to system errors and to system over-reactions (e.g. inflated error-recovery subdialogs). From Fig. 2 we infer how critical it is to engage the user into a positive state early on. Thus, we need to know also when (t¯) to fire a specific DM action. In order to estimate the time t¯ we have sampled a subset of all dialogs that contained at least one negative turn. We have then computed the probability that a negative state occurs at time t = t¯ when preceding turns are all positively biased. In Fig. 3 we plot the estimate for the probability P (t = t¯|s(1) = 0, . . . , s(t¯ − 1) = 0, s(t¯) = 1). From Fig. 3 we observe that, for those users that are bound to be in a negative state, the transition from positive to negative state will occur most likely early in the dialog. Such statistics could be exploited by the DM to calibrate the most 1
The average WER is 27.7% and TCER is 16.3%
Grounding Emotions in Human-Machine Conversational Systems
149
40
35
30
Percentage (%)
25
20
15
10
5
0
1
2
3
4
5
6
Turn
Fig. 3. Empirical estimate of time for transition probability (P (t = t¯|s(s(1) = 0, . . . , s(t¯ − 1) = 0, s(t¯) = 1), of going into a negative user state, knowing that the user will change state in the next turn. The histogram is truncated at t = 6.
likely turn to fire strategies that are user state dependent. In the next section we will estimate the DM strategy distributions over the negative and positive labels and elaborate on state dependent DM actions. 3.2
Machine Actions
Another side effect of either poor system performance or inability to detect complete user state is the time it takes to complete the domain task. In Table 1 we give different statistics for the average length of the human-machine interactions. It is evident that the negative user responses will increase the dialog length by up to 30%. As the number of turns increase, it will lead to an increased probability of turning the user into a negative state (see Figure 2). Table 1. The average dialog length in number of user turns for various conditions p ≥ 1 n ≥ 1 s(0) = 0 s(0) = 0 n=0 n ≥ 1 s(tf ) = 1 Dialog 3.2 4.4 4.6 4.4 Length
Table 2 gives for each machine dialog act such as Reprompt, Confirmation, Closing and Error Recovery the distribution of negative user reactions. Most Confirmation moves occur when the system is confident (high intent posterior probability). Closing actions usually lead to either the end of the user dialog engagement or a transfer to a domain specialist or an automated system. Both Confirmation and Closing receive positive feedback from the user point of view. Reprompt and
150
G. Riccardi and D. Hakkani-T¨ ur
Table 2. The percentage of negative and positive states in response to various types of prompts (machine dialog acts) Positive Negative State (%) State (%) Reprompt 75.5 24.5 Error Recovery (1) 74.0 26.0 Error Recovery (2) 58.8 41.2 Confirmation 85.9 14.1 Closing 88.5 11.5
Error Recovery are machine actions geared towards recovery of speech recognition and understanding errors. Reprompt actions are used at the very beginning of the interaction following the “How may I help you?” prompt. From Table 2 is evident that although most of the time the reprompt succeeds in maintaining the user in a positive state, almost 25% of the times it has a negative effect. A more compelling evidence of the negative effect is for two different error recovery strategies ((1) and (2) in Tab. 2 ). The two strategies differ for the prompt text realization and their usage over time. The relative frequency of occurrence turn numbers for each prompt is depicted in Figure 4. The second recovery strategy usually occurs later in the dialog and receives the largest negative responses among all DM actions. From Table 2 we observe that the second error recovery is penalized by achieving a poor feedback from the user. Error Recovery (1)
Relative Frequency
0.8
0.6
0.4
0.2
0
2
3
4
5
6
7
6
7
Turn Error Recovery (2)
Relative Frequency
0.8
0.6
0.4
0.2
0
2
3
4
5 Turn
Fig. 4. Histograms (over time) for the two different Error Recovery strategies (1) and (2) in Tab. 2
4
Emotions and Machine Performance
In this section we quantify the impact of the user state on the performance of the spoken dialog both at the utterance level and in terms of the overall task success.
Grounding Emotions in Human-Machine Conversational Systems
151
Table 3. Utterance length (words) of spoken input, system performance in recognizing and understanding spoken utterances. Average error rates are computed for the negative and positive/neutral label partitions of the test set (Overall). Sentence WER (%) TCER (%) Length (in words) Neutral 7.9 24.8 14.7 State Negative 15.5 31.8 25.1 State Overall 9.9 27.7 16.3
We randomly split the initial set of utterances into a training (35K) and a test set (5K). The test set has 1, 344 utterances labeled as having a negative user state, and 3, 656 utterances labeled as having a positive user state. We trained ASR models and SLU models on the training set annotated with transcriptions and 65 user intentions. For the ASR models we trained state-of-the-art acoustic and language models [6] and achieved test set WER of 27.7%. For the user calltypes we trained a multi-class classifier based on the boosting algorithm [9]. We ran 1100 iterations and achieved an average of TCER 16.3%. In Table 3 we compute the word error rates for the two set of utterances with negative and positive emotion labels. There is a large gap in performance between the two classes (22% relative WER increase). This might be due to the indirect effect of increasing the utterance length (see Table 3). We observe a similar pattern for the spoken language understanding task. This task is defined as the classification of user utterances at each turn, into one or more intent labels [4]. The increased classification error rate is due to the known limitations of SLU to handle long utterances [10]. These results support the finding that emotion predictors could be used to improve the prediction of word or classification error rate. Similarly, we might expect that the topic or intent of the user could be predictive of the user’s emotional state. In Figure 5 we plot the posterior probability of having a negative user turn P (s(t) = 1|ci ) for the most frequent call-types, ci . Most of the semantic tag posterior probabilities fall below the prior probability P (s(t) = 1), while a small set are significantly higher. The highest posterior corresponds to the request for help, as can be expected. While WER and TCER provide an utterance-based system performance metric, dialog level metrics factor in the overall success of DM strategies in accomplishing the task. On the subset (647 dialogs) of the test set we have computed task success (failure) statistics and their association with different emotion traces. In Table 4 we show that there is a strong correlation between users consistently in positive state (n = 0) and task success (first column). Similarly, the final state (tf ) of the dialog’s being negative (s(tf ) = 1) is a strong indicator of task failure. These statistics could be used to estimate prior conditions in the case of repeat-users. A sporadic transition into a negative state (second column, n ≥ 1) does not necessarily correlate with the success (or failure) of the task
152
G. Riccardi and D. Hakkani-T¨ ur 0.25
P(s(t) = 1 | Call−type)
0.2
0.15
P(s(t) = 1)
0.1
0.05
0
1
2
3
4
5
6 7 8 Call−type Index
9
10
11
12
13
Fig. 5. Empirical estimate of the posterior probability P (s(t) = 1|ci ) for call-type ci (only the posterior probabilities of a subset of the 65 call-types is shown)
completion. However, if the initial state of the user is positive and the user moves into a negative state, this is a strong indicator of task failure. The last two emotion trace statistics support the relevance of prior conditions in modeling the user state. Table 4. Task success (failure) performance of the machine over different user state statistics n = 0 n ≥ 1 s(0) = 0 s(0) = 0 n ≥ 1 s(tf ) = 1 Task 70.7% 42.3% 38.9% 29.8% Success Task 29.3% 57.7% 61.1% 70.2% Failure
5
Prior Conditions
Prior conditions refer to the user state being polarized negatively or positively prior to the user-machine interaction (t = 1). While the initial state might depend on a variety of causes directly or indirectly related to the actual user goal, it can greatly affect the expected user behavior and consequently impact the machine performance. In Fig. 2 we plot the histogram of negative state labels over time (turn) as the user-machine interaction proceeds within a dialog. At t = 1 the user is prompted with opening prompt (How May I Help You?) and state statistics show that 5% of users are negatively biased. The state dynamics are significantly different for user groups with s(1) = 0 and s(1) = 1. In Figures 6 and 7 we plot a bar chart with percentage of negative and positive labels for
Grounding Emotions in Human-Machine Conversational Systems
153
Percentage(%)
Users that start "Positive Neutral" 100 90 80 70 60 50 40 30 20 10 0
NEG PN
2
3
4
5
Turn
Fig. 6. Bar chart with percentage of negative/positive labels at each turn (t ≥ 2) when s(1) = 0. The complement to 100% for each bin is the percentage of users that exit through the final dialog state or hang-up.
Percentage(%)
Users that start "Negative" 100 90 80 70 60 50 40 30 20 10 0
NEG PN
2
3
4
5
Turn
Fig. 7. Bar chart with percentage of negative/positive labels at each turn (t ≥ 2) when s(1) = 1. The complement to 100% for each bin is the percentage of users that exit through the final dialog state or hang-up.
t > 1 following a positive and negative initial turn, respectively. The complement to 100% for each bin is the percentage of users that exit through the final dialog state or hang-up. Fig. 7 shows that relative (with respect to positive) percentage of negative labels is constant in time. Therefore, if the user is in state s(1) = 1 it will be very unlikely (on average) to leave that state, given current system limitations. From this analysis it becomes evident how important it is to detect such prior conditions early in the dialog and adapt the machine’s DM strategies accordingly.
6
Conclusion
In this paper we have investigated the role of emotions in human-machine spoken dialogs. Emotion levels have been quantized into positive/negative and user state statistics have been drawn from the How May I Help You? spoken dialog system. For each human-machine interaction we have acquired the temporal emotion sequence going from the initial to the final conversational state. These statistical
154
G. Riccardi and D. Hakkani-T¨ ur
traces characterize the user state dynamics. We have grounded emotion patterns into dialog management strategies as well system performance. Our findings show that recognizing emotion temporal patterns can be beneficial to improve machine actions (dialog strategies) as well as to predict system error (ASR and SLU error rates). Acknowledgements. We would like to thank Frederic Bechet for his contributions to the annotation of the dialog database.
References 1. Lee, C.M., Narayanan, S.: Towards detecting emotions in spoken dialogs. IEEE Transactions on Speech and Audio Processing 13 (2005) 293–303 2. Ang, J., Dhillon, R., Krupski, A., Shriberg, E., Stolcke, A.: Prosody-based automatic detection of annoyance and frustration in human-computer dialog. In: Proceedings of ICSLP, Denver, Colorado, USA (2002) 2037–2039 3. Litman, D., Forbes-Riley, K.: Predicting student emotions in computer-human tutoring dialogues. In: Proceedings of the 42nd Annual Meeting of the Association for Compuational Linguistics (ACL), Barcelona, Spain (2004) 4. Gorin, A.L., Riccardi, G., Wright, J.H.: How may I help you? Speech Communication 23 (1997) 113–127 5. Gupta, N., Tur, G., Hakkani-T¨ ur, D., Bangalore, S., Riccardi, G., Rahim, M.: The AT&T spoken language understanding system. IEEE Transactions on Speech and Audio Processing (To appear) 6. Goffin, V., Allauzen, C., Bocchieri, E., Hakkani-T¨ ur, D., Ljolje, A., Parthasarathy, S., Rahim, M., Riccardi, G., Saraclar, M.: The AT&T WATSON speech recognizer. In: Proceedings of IEEE ICASSP-2005, Philadelphia, PA, USA (2005) 7. Abella, A., Gorin, A.G.: Construct algebra: Analytical dialog management. In: Proceedings of the 42nd Annual Meeting of the Association for Compuational Linguistics (ACL), Washington D.C. (1999) 8. Shafran, I., Riley, M., Mohri, M.: Voice signatures. In: Proceedings of The 8th IEEE Automatic Speech Recognition and Understanding Workshop (ASRU 2003), St. Thomas, U.S. Virgin Islands (2003) 9. Schapire, R.E., Singer, Y.: BoosTexter: A boosting-based system for text categorization. Machine Learning 39 (2000) 135–168 10. Karahan, M., Hakkani-T¨ ur, D., Riccardi, G., Tur, G.: Combining classifiers for spoken language understanding. In: Proceedings of IEEE workshop on Automatic Speech Recognition and Understanding, Virgin Islands, USA (2003)
Water, Temperature and Proximity Sensing for a Mixed Reality Art Installation Isaac Rudomin, Marissa Diaz, Benjam´ın Hern´ andez, and Daniel Rivera ITESM-CEM, Atizap´ an, M´exico
Abstract. ”Fluids” is an interactive and immersive mixed reality art installation that explores the relation of intimacy between reality and virtuality. We live in two different but connected worlds: our physical environment and virtual space. In this paper we discuss how we integrated them by using water and air as interfaces. We also discuss how we designed mechanisms for natural and subtle navigation between and within the different environments of the piece, how we designed the environments and the installation so as to take advantage of the low cost alternatives that are available today.
1
Introduction
In this paper we explore “Fluids”, an interactive and immersive mixed reality art installation that uses water and air as interfaces. It explores the relation of intimacy between reality and virtuality, two different but connected worlds. Mixed reality and tangible interfaces have been proposed as ways to close the gap between atoms and bits [1,2]. We attempt to integrate them by using water waves and air temperature as interfaces, as well as a natural navigation system using proximity sensors. We think that this approach, rather than the purely visual approach of traditional mixed reality, or even the use of solid objects enhanced with computation, as traditionally has been done in tangible interfaces, can be very fruitful. Fluids thus become interfaces, by way of custom designed, inexpensive devices based on the measurement of luminous or infrared radiation. When interacting with water we perceive flow, texture, temperature [3], sound. Used as interface, water allows for a high level of immersion by involving several senses. It is a known fact of the biological sensory system that the temperature of water, for example, can be seen by using MRI to activate the parietal lobe. By touching water, we are in fact changing cyberspace from a dry world of mere pixels into a moist, wet place much like the real world [4]. Ever since the 60’s, art has been incorporating the participation of the public as part of the artistic work, in particular in the form of happennings, performance and installations. Work by “Fluxus” being one of the most representative of the time. By the 90’s, with the emergence of interactive art, technology that include the public as part of the piece. In this context, the work of Char Davies [5] being very important in that it introduced breathing for virtual navigation. When we breathe we interact in one most intimate ways with the world. One can see this in Char Davies’s work mentioned above. This we attempt to do M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 155–163, 2005. c Springer-Verlag Berlin Heidelberg 2005
156
I. Rudomin et al.
in a different way, however, trying to avoid encumbering the user with vests or using the diving metaphor. By using a temperature measuring device, a kind of microphone, but isolated from extraneous noise, the interface becomes one with the user. Navigation in “Fluids” tries to function much like in the real world: we can move around freely, within the different environments, as well as between them, yet without the use of the standard interfaces that are not subtle enough for our needs in this piece. The environments we use are surreal, and yet, because of the intimate connection with the user via the fluids, seem very real and alive. We are using inexpensive new generation graphics cards and low cost stereoscopic projection: this combination can achieve in real time and with low costs effects that have previously required equipment not generally available for artists or scientists in our part of the world.
2
Interfaces and Environments
The sense of touch is complex. It includes feeling pressure, temperature, and haptics. Applications that use some aspects of this sense are a powerful way of creating an illusionary link between what the user is seeing and what the user is touching. This tactile reference expands the imagination of the user. Seeing the changes in a virtual world when touching and feeling natural elements is a powerful way of achieving feedback [6,7] and is an effective interface. Touching real water and blowing on a model tree (a modified microphone) have already been explored[8,9] and they turn out to be rather dramatic interfaces. 2.1
Wave Sensor
In the case of touching water, it is not only the liquidness of water that affects the perception of the user, also the temperature of the water, the pressure of water, the amount of water, and even the sound of water: all this can change the user’s perception and reactions. Even simple applications, become very complex and immersive. In these applications, a wave sensor attached above a receptacle filled with water is used to control the physics of the virtual water in the pond. This wave sensor is a differential sensing device that responds to the alterations on the water surface and that returns a frequency that is proportional to the wave’s frequency in the water. Our wave sensors work by generating a frequency that is proportional to the amplitude of the wave in the water; This is accomplished by a retro fed circuit that oscillates at a fixed frequency that is given by the closed loop generated by the emission and reception of light. To measure the frequency of the water wave, it is necessary to count how many times the sensor reports the amplitude to be the same and multiply this value by a given factor that depends on the
Water, Temperature and Proximity Sensing
157
distance between the sensor and the water. Due to the nature of the dispersion of light the response of this system is not linear. Light normally does not travel in only one direction, rather, it spreads trough the air and water. This makes it necessary for us to linearize the output of our sensor. To achieve this kind of response using an optical sensing device we used at first an infrared emitter coupled with a single receiver in the order of the 960nm of the spectrum. This allowed us to test the capabilities of our sensor but it turned out to have some drawbacks: 1. Some common illumination sources emit infrared light and cause interference to our system. This was not normally a problem, but when a TV crew came to film the system, it was very difficult to make it work. 2. Water absorbs part of the infrared emission making it hard to read the light that bounces. 3. Using only one receiver makes it impossible to capture all the reflected light due to the aperture of the angle of reflection. This system was tested in a small water container in our university and presented to the public with good response from the users. After this experience we adapted the original sensor to make it more reliable and efficient by taking the next steps: 1. Changing the type of light reflected and using now blue light that is not absorbed by water. 2. Improving the system’s emission and reception of light by having one receptor on each side of the sensor probe. 3. Narrowing the light path with a plastic piece to ensure the correct reading and lowering the risk of interference. 4. Making the connection more foolproof by using coaxial cable. A drawing of the wave sensing device as well as a functional diagram can be seen in figure 1. We wanted to explore this interactive sensation of touching water in a more complete manner. We decided to improve the sensing by adding more wave sensors. Having more wave sensors allows us to triangulate and thus obtain the location of the perturbation, the origin of the wave. We can also use the extra sensors to emulate correctly the interference of waves generated at the corners or when two or more waves collide. This is necessary because water waves experience interference when diffracted and when they bounce against the edges of the deposit containing the water (where the ripple was generated). In a pond, for example, if we start a water wave ripple, it will start out near the center and spread radially outward. When this ripple reaches one edge it will generate another ripple. If there are more ripples being generated at the same time, when two waves encounter each other and two crests meet, the result is the sum, and where a crest and trough meet, they cancel. For this piece, then, four sensors that can detect water waves are attached above a receptacle filled with water.
158
I. Rudomin et al.
Blue LED Emmiter
a)
Blue LED Receivers
Coaxial Joint
Output signal A wave signal is generated
Blue light emmision and reception
Reflected light varies depending on water osscillation
b) Fig. 1. Wave sensing device, functional diagram
The four sensors are connected via coaxial cable to a motherboard, which contains the control electronics of this device, as well as the navigation system described later. This control box is connected to the computer via USB. The four wave sensors and the motherboard are shown in figure 2. 2.2
Temperature Sensing Device
The other fluid we are exploring is air. In this case, a subtle effect that accompanies breathing. At first we tried a piezoelectric device that we had used to detect blowing. But we felt this piece required detecting breathing rather than blowing. We tried using a standard microphone. The results were interesting, but not what was desired. An interesting situation arose when we were trying out the application and an airplane flew by. The commotion caused on the virtual environment by this unanticipated sound was deemed interesting by the artist, at first, but not what was really needed. Therefore, instead of a microphone,
Water, Temperature and Proximity Sensing
159
Fig. 2. Wave sensors and motherboard
but connected similarly, we developed a device that detects breathing but is invariant to noise and vibration. The system consists of a temperature sensing probe (based on a Siemens Silicon Spreading Resistance Temperature Sensor ) which detects when the air temperature around it goes down more than 0.3 degrees celsius. This device detects the changes in temperature which occur when a person is breathing. To avoid calibration, an additional temperature sensing component placed in a box is used for reference. You can see a diagram of the device in figure 3a. 2.3
Navigation
An intuitive navigation system that can provide interaction without having to explain to users was designed. It is modular and independent from the water sensing system. Four proximity sensors (reflected capacitance sensors) are used to interpret the users movements within an area above the water tank and generate the navigational information. When the front(back) sensor detects we are close to it, the virtual world camera moves forward (backwards), while if the left-right sensors detect we are close, it rotates left-right, respectively. The navigation system diagram is shown in figure 3 b. All in all, then, a user will interact with: 1. Four proximity sensors that are placed on the outside edge of a water tank and that allow the user to navigate within and between environments. A metal plaque is placed on the inside edge of the water tank to maximize sensor sensibility on the inside of the tank. An arrow is placed on the plaque as a signalling device. 2. The four wave sensors are placed above the water tank, and the user interacts with it by touching or moving the water with the hands. These two sensor
160
I. Rudomin et al. Temperature difference sensor
Temperature reference box To sound card microphone input
Headset
Mount
PC
3D stereo glasses
a)
Water sensing influence area
Sensor 2
Sensor 3
Sensor 4
Motion sensing influence area b)
Sensor 1
Fig. 3. Temperature sensing and Navigation systems
Fig. 4. Installation previzualization and final installed version (with users interacting, the room had to be completely dark, so the quality of the pictures suffers)
systems are connected to a common motherboard control that is connected to the computer as a USB device. 3. Finally, connected as a microphone, and assembled with one of the polarized glasses, the temperature sensor detects the breathing of the user. A low cost Geowall style passive stereo projection system involving 2 DLP projectors, a low cost PC card with 2 outputs, circular polarization filters and glasses, and a very low cost 2m × 1.5m silver screen (5 dollars worth of fabric) was built for the exhibit.A previsualization of the piece can be seen in figure 4 a. All these devices, as installed for the exhibit, and a guest of this exhibit interacting with
Water, Temperature and Proximity Sensing
161
the system, are shown in figure 4 b and c. Note that these pictures were taken in total darkness, so the quality is not the best.
3
The Environments
Having sensing and navigation devices is all very well, but an important part of “Fluids” is the integration of these technologies in a seamless and natural fashion into specific environments. The environments were designed not only to take advantage of the capabilities of the system, but with illustrating some scientific principles along the way (using equations, graphs to generate a sort of mathematical landscape). Nevertheless, given extreme time constraints, as an art
a)
b)
c) Fig. 5. Snowy city, Strange Lake and Cloudy River environments
162
I. Rudomin et al.
piece the environments were meant to be an initial exploration of the interfaces rather than a finished version. As exhibitied, the piece consisted of three environments: The first environment consists of a snowy city corner, with a building, where the breathing sensor changes speed of falling snow and the navigation system lets us wander, but changes to the next environment when you get inside the building (this is similar in all environments). Here the water sensor is not used. This environments is shown in figure 5 a. The second environment consists of terrain, mountains that do not exist in nature. At its center there is a lake, surrounded by areas where some grass grows. This is a strange lake, where touching the water moves the water in the lake, while the breathing sensor changes speed of movement of grass around the lake, and also makes the lake become psychodelic. This environment is shown in figure 5 b. Finally, a landscape that has a topography generated from mathematical graphs, an open environment consisting of a large, open, sterile land mass, brought to life by a river and shadows of clouds. In this cloudy river the wave sensors change the flow of water and clouds and the breathing sensor changes speed of moving clouds. This environment is shown in figure 5 c.
4
Conclusions and Future Work
Different and appropriate interaction techniques and tricks must be developed for each virtual environment or art installation, in order to achieve the effect the artist is seeking. Using the standard interfaces available can not always deliver the reactions, emotional responses and suspension of disbelief that are required. After testing our custom devices and interfaces with users, both at art and technical exhibits, we were able to gauge the reaction of users to the applications, and thus modify them as needed for subsequent and more polished artwork. Applying technology to art implies an attention to detail and to user perception that is not the rule in computer graphics research. Artists have a lot to teach our community, and the interaction between our different fields is very fruitful. The installation was not completely how we had planned it, but rather, we achieved 90% of what we wanted. Figure 6 is a photo taken from a clip, broadcast on TV about the piece: a user is shown interacting with the breathing device. The second image in figure 6 is another frame from this sequence that shows the result of the interaction in the screen with the snowy city environment. The public loved the piece. It was deemed interactive in a novel way, by users of all ages and backgrounds. Stereo projection of interesting scenery increased the immersive effect dramatically, but the interfaces were the heart of the piece. Exhibiting the piece allowed us to observe the interfaces in action with a diverse group of users. This observation will let us make the piece better for future exhibits (there are at last 3 planned in different countries). This will also make our interface research much more relevant. Observation of the interaction of
Water, Temperature and Proximity Sensing
163
Fig. 6. Interacting with snowy city, as shown on broadcast TV
the public has been anectdotic. Interviews and quizzes are not good enough to capture nuances of this interaction. We are thinking of ways to monitor the interactions in a more systematic way.
References 1. P. Milgram and F. Kishino, ”A taxonomy of mixed reality visual displays,” in IE-ICE Trans. on Information and Systems (Special Issue on Networked Reality), vol.E77, no.12, pp.1321-1329, 1994. 2. H. Ishii , B. Ullmer, ”Tangible bits: towards seamless interfaces between people, bits and atoms,” in Proceedings of the SIGCHI conference on Human factors in computing systems, Atlanta, Georgia, ACM pp.234-241, March 22-27, 1997 3. L. Jones, M. Berris, “The Psychophysics of Temperature Perception and ThermalInterface Design”, in Proceedings 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems Orlando, Florida, March 24 - 25, 2002 IEEE, pp. 137 4. R. Ascott, “Edge-life : technoetic structures and moist medias”, inArt, technology, consciousness : mind@large. - Bristol : Intellect, 2000 pp 2 5. C. Davies, “Landscape, Earth, Body, Being, Space and Time in the Immersive Virtual Environments Osmose and Eph`em´ere”, in Women, Art, and Technology. Judy Malloy, ed. London, England: The MIT Press (2003), pp. 322-337 6. Yonezawa, K. Mase, “Tangible Sound: Musical Instrument Using Tangible Fluid Media” , in International Computer Music Conference (ICMC2000) Proceedings, pp.551-554, 2000. 7. S. Marti, S. Deva and H. Ishii,“WeatherTank: Tangible Interface using Weather Metaphors”, in Technical Report 2000, http://web.media.mit.edu/ nitin/papers/sand.html. 8. M. Diaz, E. Hernandez, L. Escalona, I. Rudomin, I., and D. Rivera, “Capturing water and sound waves to interact with virtual nature”, in Proceedings of ISMAR 2003, IEEE Press, IEEE, 325. 9. M. Diaz, and I. Rudomin, “Object, function, action for tangible interface design”, in Proceedings of Graphite 2004, ACM Press ACM SIGGRAPH, 106–112.
Geogames: A Conceptual Framework and Tool for the Design of Location-Based Games from Classic Board Games Christoph Schlieder, Peter Kiefer, and Sebastian Matyas Laboratory for Semantic Information Processing, Otto-Friedrich-University Bamberg, 96045 Bamberg, Germany {christoph.schlieder,peter.kiefer,sebastian.matyas} @wiai.uni-bamberg.de
Abstract. Location-based games introduce an element that is missing in interactive console games: movements of players involving locomotion and thereby the physical effort characteristic of any sportive activity. The paper explores how to design location-based games combining locomotion with strategic reasoning by using classical board games as templates. It is shown that the straightforward approach to “spatialize” such games fails. A generic approach to spatialization is presented and described within a conceptual framework that defines a large class of geogames. The framework is complemented by a software tool allowing the game designer to find the critical parameter values which determine the game’s balance of reasoning skills and motoric skills. In order to illustrate the design method, a location-based version of the game TicTacToe is defined and analyzed.
1 Introduction The traditional image of home entertainment based on game consoles that confine physical involvement to letting the player move a joy stick is certainly obsolete. Currently, the integration of bodily action into games receives much attention from research in academia and industry. Examples of commercial products which allow the player to interact via more or less complex movements are Donkey Konga (Nintendo), EyeToy (Sony) and Dancing Stage (Konami). The motions of the player that are taken into account can be as simple as hitting a drum or stepping on a dancing mat. More intricate forms of movement without physical sensor contact, e.g. waving gestures, are captured by video or IR. All these games involve movement of parts of the body but only very limited displacement of the body as a whole. In contrast, locomotion of players – and the physical effort it implies – has been a major motivation for developing location-based games. A classification of spatial scales from cognitive psychology proves useful to clarify the issue. Montello (1993) distinguishes figural space which is smaller than the body and accessible to haptic manipulation or close visual inspection, vista space which is as large or larger than the body but which can be visually apprehended from a single place without locomotion, and environmental space which is larger than the body and cannot be M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 164 – 173, 2005. © Springer-Verlag Berlin Heidelberg 2005
Geogames: A Conceptual Framework and Tool for the Design
165
experienced without considerable locomotion. Using these terms, we can say that location-based games consider body movements beyond figural space, i.e. beyond the space of computer screens and small 3D-objects, and that they focus on locomotion in vista space, typically the space of a single room or a sports ground, or locomotion in environmental space such as the space of a neighbourhood or a city. Many location-based games have been designed for environmental space with GPS as localization technology. Some of them are just adaptations of popular computer games such as ARQuake (Thomas & al. 2000) or Pirates! (Björk & al. 2001); others are rather straight forward chase games like Can You See Me Now (Flintham & al. 2003). A totally different type of game is obtained by combining the intellectual appeal of classic board or card games with the physical involvement of location-based games, that is, by merging strategic elements from the former with real-time locomotion from the latter. A typical example of this type of game design is CityPoker (Kiefer & al. 2005). Although the idea to directly map classic board games to the real world is not entirely new, see Nicklas & al. (2001) for another example, to our knowledge no general framework for location-based games with strategic elements has been presented yet. The contribution of this paper consists in defining and implementing a framework which helps a game designer to create a challenging location-based game. A game is considered challenging if it addresses both, the player’s reasoning skills and the player’s motoric skills. Neither a chess tournament nor a 100 m sprint would constitute a balanced challenge in this sense. We are looking for games blending chess-style and sprint-style elements. The main body of the paper is structured as follows. In section 2 we show that a straight-forward spatialization of a board game leads only to trivial non-challenging location-based games. A general solution to the problem of spatialization is proposed. Section 3 introduces the conceptual framework which defines a large class of geogames. The geogame analysis tool allows the game designer to find the critical parameter values which determine the game’s balance of reasoning skills and motoric skills. As an illustration of the design method, a locationbased version of the game TicTacToe is analyzed in section 4. Finally, related work and future research directions are discussed (section 5).
2 Synchronization as Problem in Spatial Versions of Board Games Board games come in many variants not all of which are intellectually as demanding as chess or Go. These two games belong to a large class of games that game theory describes, namely two-person games which are deterministic (no random element such as a dice exists) and provide full information about the game’s state to each player (no hidden elements such as cards in the hand of the opponent exist). In the following we concentrate on this rich class of games as a source of inspiration for strategic elements for location-based games. Throughout the paper the term board game is used in this narrow sense. A well known and structurally simple instance of this class serves as our running example. In TicTacToe two players move alternately placing marks – the first player to move uses X, the second O as mark – on a game board consisting of 3×3 squares. The player who first places three marks in a row, a column or one of the two diagonals wins the game. If neither player wins, the game ends in a draw.
166
C. Schlieder, P. Kiefer, and S. Matyas
Physically, board games are played in figural space. To obtain a location-based version of the game, the game should be mapped onto vista or environmental space in a way that each move requires locomotion of the player introducing time as a new dimension in the game. We call the result a spatial version of the board game and refer to the process of producing a spatial version as spatialization. The straightforward approach to spatialization consists in mapping the game board to vista or environmental space by assigning a geographic footprint to each of the board’s positions. To simplify matters, points are used as geographic footprints for TicTacToe board positions, that is, spatialization assigns a geographic coordinate to each of the 9 squares of the board. Players need to physically move to a board position in order to place a mark. The time it takes to complete a move therefore depends on the distance of the board positions in vista or environmental space. Note that it is not necessary that the geographic footprints of the TicTacToe board positions are arranged in form of a regular 3×3 array of points (left in Fig.1). In general, spatialization does not preserve the distance relationships that hold on the game board in figural space because this gives game designers an important additional degree of freedom (right in Fig. 1). Start
Start Fig. 1. Spatial version of TicTacToe played in vista or environmental space
Unfortunately, the straightforward approach to spatialization results in locationbased games that are not challenging in the sense defined in section 1. The problem is due to the fact that the logical appeal of board games is linked to the complexity of the state space of the game which is altered drastically if the two players do not move in alternation. Consider the game illustrated by the left graph in Fig. 1. The traces of the two players’ locomotion reveal that the X-player moves significantly faster than the O-player. Obviously, in this case there is a simple winning strategy for the player who moves faster: be the first to reach the lower right square, proceed to the central square and move to the upper left square at which point the game is won. This spatial version of TicTacToe amounts to a race between the two players which lacks any elements of strategic reasoning and therefore cannot be considered a challenging location-based game. In a game with strictly alternating moves, on the other hand, the players’ speed has no impact at all resulting in a non-challenging board game. The type of game one would like to see is illustrated by the right graph in Fig. 1. In such a game, moves are very often – but not always – played in alternation. To put it
Geogames: A Conceptual Framework and Tool for the Design
167
differently, the challenge of designing a location-based game from a board game consists in limiting the occurrence of multiple moves from the faster player. We call this the synchronization problem because it levels to a certain extent differences between the two players’ speed resulting in a deliberately non-perfect synchrony of moves. We propose a surprisingly simple solution to the synchronization problem which is inspired by a strategy to resolve timing problems in computer hardware. After reaching a board position and placing the mark, a player is required to spend a certain pre-defined synchronization time at the position before moving on. It is to the game designer how to build this synchronization time into the game: Either the player might be forced to wait idle, or he could be obliged to perform some time-consuming tasks like solving a puzzle or searching an item. As we will show in section 5, the length of the synchronization time interval constitutes the parameter that determines whether the spatial version of a board game becomes a challenging location-based game or whether it deteriorates into a race-style game or classic board game respectively. For now, we just note that the synchronization problem is not a problem of the specific choice of geographic footprints and the starting position in the left graph of Fig. 1. It appears as an effect of speed difference between players in pretty much the same way in the right graph of Fig. 1.
3 Geogames 3.1 A Framework for Describing Location-Based Games Although a spatial version of TicTacToe has some interest in its own, the aim of the designer is to possess a general method permitting to reuse board games as templates for location-based games. The descriptive framework for location-based games should handle other spatial versions of board games or games that could be considered such games as, for instance, CityPoker. Common traits of these games are: a fixed number (often but not always two) of players move between a fixed number of board positions called locations taking up and putting down resources when they reach a new position. A resource is anything that can be transported by players and deposed in locations, e.g. an X-mark or O-mark in TicTacToe or a playing card in CityPoker. The state of a game is defined by the locations of the players and by how the resources are distributed over players and locations. Actions are described as transitions between states. They describe the combined effect of moving from a location to new one and of taking up and/or putting down resources there. Definition: Let P denote a set of players, L a set of locations and R a set of resources. A state s = (location, resources) is a tuple of two mappings, location: P → L and resources: R → L ∪ P. By S we denote a set of states, usually the states of a game. An action a on S is a mapping a: S → S. A set of actions is denoted by A. The first basic constraint for actions is spatial coherence: a player can pick up or dispose of a resource only at the player’s current location, and no resource may appear or disappear at a location without involvement of a player. To describe the temporal aspect that differentiates location-based games from board games, a duration is assigned to all actions. Game states are assigned a value which expresses the
168
C. Schlieder, P. Kiefer, and S. Matyas
interest of the state for the players. In TicTacToe the values of interest are {open, Xwins, O-wins, draw} with the intuitive semantics that open is assigned to non-end states of the game. The length of the synchronization interval is specified by a constant. The second basic constraint for actions is temporal coherence: every action consumes time equivalent to the sum of its duration and the synchronization interval. Definition: Let S denote a set of states and A a suitable set of spatio-temporally coherent actions. A geogame G = (S, A, time, value, sync) consists of two mappings, + time: A → R and value: S → V where V denotes the value space for state evaluation, + and a constant sync∈ R . Although the definition describes a large class of games, not all location-based games are geogames. Most important, games that do not satisfy the spatio-temporal coherence constraints – in which resources magically jump around the board – are not geogames. A spatial version of TicTacToe which we call GeoTicTacToe can easily be described in a way that fits with the above definitions. The game is played by two players, P = {PX, PO} on a board with locations L = {L11, …, L33, Start} where X and O are used as marks, R = {X1, …, X6, O1, …, O6. The states of the game are described by their distribution of resources, for instance start = (locationStart, resourcesStart) with locationStart(PX) = locationStart(PO) = Start and resourcesStart(X1) = … = resourcesStart(X5) = PX , and similarly for the resources denoting the O-marks. In practice, only the starting state is described explicitely while other states are constructed from applying actions to the starting state. 3.2 Geogame Analysis Tool The synchronization time interval specified by the sync constant of a geogame has already received some attention. In principle, it would be possible to find suitable values for the parameter by playing and evaluating a large number of games in reality with different parameter settings. Although some successful games such as CityPoker were developed that way, it is hardly satisfactory as a general method for game design. The geogame analysis tool supports the designer to determine the sync parameter by systematically exploring the game’s state space. As geogames are defined as a rather generic concept, the state space analysis must handle significantly more special cases than the analysis described by Kiefer et al. (2005) for CityPoker. We assume that the players in a geogame always behave in the following way: They first decide which location to move next (several possibilities), then they move towards that location and arrive after some time. Now they select which resources to change, before they finally have to wait synctime and move on to the next location. The geogame analysis tool makes a number of additional assumptions, for instance, that players move as fast as they can and that they don’t waste time by waiting longer than the synchronization interval. Finally, rational players are assumed who try to win the game. With these assumptions, the geogame analysis tool explores the state space using a generalization of the minmax algorithm (see Russell and Norvig (2003) for a description of standard minmax). The generalization handles multiple players and determines the next player to move by the time units players need for their actions. The modifications of minmax are not trivial. Consider, for example, two or more players arriving at a location in the same instance of time, i.e. with remaining time
Geogames: A Conceptual Framework and Tool for the Design
169
units 0 (concurrent resource change), necessitating the incorporation of randomized elements (see Kovarsky and Buro (2005) as an example). Furthermore, appropriate pruning strategies become essential for state spaces larger than the one of GeoTicTacToe. A geogame ends when an end-of-game condition becomes true. End states are evaluated using an evaluation function that is derived from the value mapping of the geogame. The result is propagated through the tree similar to standard minmax until the starting state is reached. Finally, the values at the starting state induce a ranking of the players and thus represent the outcome of the game under the assumption that all players act optimal. This ranking gives the game designer a first idea about the fairness of the game he has created: a completely fair game ends with all players having the same rank. Note that fair games are not necessarily challenging in the sense of the definition of section 1. The geogame analysis tool is implemented using a flexible architecture with the four layers search mechanism, geogames engine, concrete geogame and parametrized geogame. This architecture allows to easily model and analyze any geogame and to experiment with different search mechanism with only little effort.
4 GeoTicTacToe: A Case Study With the help of the geogame analysis tool an appropriate value for the synchronization time interval sync can be found. We describe the analysis of GeoTicTacToe for the case where the X-player is 10% faster than the O-player. Keeping this speed ratio fixed, synchronization time was varied between 0 and 12 in steps of 0.1. Three types of results were logged for each set of parameters. (1) The ranking of the players for which there are two possible outcomes as the slower Oplayer is not able to win: X-player wins and the game ends with a draw. (2) The depth of the game, that is, the number of X- and O-marks that have been made when the game finishes. Each end state has a depth value between 3 (one player could set three marks) and 9 (all marks have been set) which is propagated through the tree along with the corresponding evaluations. Having the choice between two winning successor states, a player would prefer the one with lower depth. Obviously, depth correlates with the ranking: A depth of smaller than 9 always comes along with a win for the X-player. On the other hand, a depth of 9 could result in either a win or a draw; nevertheless, we did not have any win situation at depth 9 in our study. (3) An optimal path through the game tree which corresponds to the game in which both players act optimally. Usually, more than one optimal path exists. Fig. 2 shows the results for GeoTicTacToe played with the geographic footprint configuration illustrated in the inset which is the same as the one shown in the left graph of Fig. 1. Note that the grey square denotes the common starting point of both players. The result very clearly shows the effect of the length of the synchronization interval. For small values of sync (synctime in Fig. 2), the depth of the game does not exceed 4 or 5 respectively. These are games which the faster X-player wins by racing. The O-player cannot prevent the X-player from setting the X-marks in the diagonal.
170
C. Schlieder, P. Kiefer, and S. Matyas
9
board game
depth of the game
8
7
6
5
race
4
geogame
3
0
1
2
3
4
5
6
7
8
9
10
11
12
synctime
Fig. 2. Depth of the game for varying synchronization intervals for GeoTicTacToe
On the other hand, high values for sync (synctime in Fig. 2) lead to alternating moves of the two players as in the board version of TicTacToe. We conclude that the interesting range for the parameter sync lies between 4.6 and 11.3 and leads to games that end at depth 7. Fig. 3 illustrates an optimal path (course of the game) for sync=10. Here, the Xplayer has to wait long enough at the first location to allow the O-player to set an Omark in the centre. With the next move, the X-player forces the O-player to move to the top right corner which opens up the possibility to fill in the missing X-marks in the bottom row with the O-player being too far away to reach the lower left corner in time. This type of move sequence which blends logical reasoning with physical locomotion is generally found at depths between 6 and 8 and creates just the kind of game that can be considered a challenging geogame.
Fig. 3. Challenging geogame (optimal path) at sync=10.0
With the geogame analysis tool also the effect of different choices of geographic footprints is easily studied. Fig. 4 shows a depth versus sync plot comparable to that of Fig. 2 but for the geographic footprint configuration shown in the right graph of Fig. 1.
Geogames: A Conceptual Framework and Tool for the Design
171
9
8
depth of the game
race 7
6
5
geogame
4
board game
3
0
1
2
3
4
5
6
7
8
9
10
11
12
synctime
Fig. 4. Depth of the game for varying synchronization intervals for GeoTicTacToe
Without going into detail, we note that different boundary values delimiting racestyle games, challenging geogames, and classical board games are found. In other words, the choice of the geographic footprints has an effect. We can derive even more from the analysis: the footprint configuration from Fig. 4 promises more interesting games than that from Fig. 2 since all depth values between 3 and 9 actually occur. To sum up, we propose to choose a synchronization interval within the value range corresponding to challenging geogames with game depth between 6 and 8. The exact choice within this range is left to the designer giving him the freedom to put more emphasis on speed or on reasoning. Certainly, any other choice of geographic footprints for GeoTicTacToe or other assumptions about the speed ratio X-player and O-player may be analyzed the same way.
5 Related Work and Future Research We have described a conceptual framework defining geogames and a tool for analyzing them, especially with the goal of tuning the game to be challenging. The game designer may now proceed as follows: (1) select a classical board game with interesting strategic elements, (2) choose alternative sets of geographic footprints in vista space or environmental space for the board positions, (3) model the resulting location-based game within the conceptual framework of geogames, (4) use the geogame analysis tool to derive interesting values for the sync parameter. Location-based real-time games abandon the idea of turn-taking of classical board games. Nicklas et al. (2001) observe a consequence, namely that “lifting turn-based restrictions can make a game unfair“, and propose a solution which is inspired by
172
C. Schlieder, P. Kiefer, and S. Matyas
methods of allocating machine resources to concurrent processes. Similarly, Natkin and Vega (2003) and Vega et al. (2004) show how to assist the game designer in finding dead locks in the game flow using Petri-nets to describe the game. This type of research focuses on concurrency but does not address, yet answer the problem of synchronization that characterizes the difference between race-style games, challenging geogames and classical board games. AI techniques, like variants of minmax-search, have been applied to board games and are constantly improved to create increasingly smart computer opponents, e.g. for Othello (Buro 1999). This is an interesting line of research; however, the focus of our paper is not the development of optimal search algorithms or pruning strategies. Although most location-based games have been developed for environmental space, our analysis showed that this is no fundamental limitation. In a vista space version of GeoTicTacToe two players are moving each on a dancing mat with the geographic footprints of the game board (Fig. 5). The state of the game is communicated through a wall-mounted display. Synchronization is easily achieved: when a player reaches a board position, a small X- or O-mark appears on the display which changes to big size when the synchronization interval has passed and the player is free to move on. Note that on a small game of the size of a dancing mat, the synchronization time interval would be very small, e.g. some seconds. Comparing different spatializations of Geogames starting with GeoTicTacToe in vista space and environmental space will be subject of future research. As another location-based game we will map the above-mentioned game CityPoker to the Geogames framework. Even a spatialization of chess with modified rules is imaginable and would hold some further interesting synchronization problems. These rule modifications could be inspired by existing modifications of chess which lift turn-based restrictions like “progressive chess” or “double move chess” (see e.g. http://www.chessvariants.org/).
Fig. 5. GeoTicTacToe played in vista space at home
Furthermore, we plan to build into our model a parameter for the players’ cleverness. Imagine one player spending much time on reasoning but moving slowly, while the other player is moving fast but does not invest much effort in thinking. Simulating games with this constellation could make up an interesting case for testing the relationship between reasoning time and acting time. By varying one player’s
Geogames: A Conceptual Framework and Tool for the Design
173
search depth and the other’s speed, the balance of speed against reasoning could be emulated. This would also help in the design of a virtual smart opponent as described in Kiefer et al. (2005).
References Björk, S., Falk, J., Hansson, R., Ljungstrand, P.: Pirates! - Using the Physical World as a Game Board. Paper at Interact 2001, IFIP TC.13 Conference on Human-Computer Interaction, July 9-13, Tokyo, Japan. Buro, M. (1999): Experiments with Multi-ProbCut and a new high-quality evaluation function for Othello. H. J. van den Herik and H. Iida (Eds.): Games in AI Research Flintham, M., Anastasi, R., Benford, S. D., Hemmings, T., Crabtree, A., Greenhalgh, C. M., Rodden, T. A., Tandavanitj, N., Adams, M., and Row-Farr, J.: Where on-line meets on-thestreets: experiences with mobile mixed reality games. In Proceedings of the CHI 2003 Conference on Human Factors in Computing Systems (Ft. Lauderdale, FL, April 2003). ACM Press, New York. Kiefer, P., Matyas, S., Schlieder, C.: State space analysis as a tool in the design of a smart opponent for a location-based game. Proceedings of the Games Convention Developer Conference “Computer Science and Magic”, Leipzig, Germany (2005) Kovarsky, A. and Buro, M.: Heuristic Search Applied to Abstract Combat Games, Proceedings of the The Eighteenth Canadian Conference on Artificial Intelligence, Victoria 2005 Montello, D.: Scale and Multiple Psychologies of Space. In Proc. Conference on Spatial Information Theory (COSIT-93), LNCS 716, Springer: Berlin, pp. 312-321., 2003 Natkin, S. and Vega, L.: Petri net modeling for the analysis of the ordering of actions in computer games. In Mehdi, Q. and Gough, N., editors, GAME-ON, 2003, 4th International Conference on Intelligent Games and Simulation, pages 82-89. Nicklas, D., Pfisterer, C., Mitschang, B.: Towards Location-based Games. In: Loo Wai Sing, Alfred (ed.), Wan Hak Man (ed.), Wong Wai (ed.), Tse Ning, Cyril (ed.): Proceedings of the International Conference on Applications and Development of Computer Games in the 21st Century: ADCOG 21; Hongkong Special Administrative Region, China, November 22-23 2001. Russell, S. and Norvig, P. (2003), Artificial Intelligence A Modern Approach, M. Pompili, Ed. Prentice Hall Thomas, B., Close, B., Donoghue, J., Squires, J., De Bondi, P., Morris, M., Piekarski, W.: ARQuake: An outdoor/indoor augmented reality first person application, Fourth International Symposium on Wearable Computers (ISWC’00), Atlanta, Georgia, 2000. Vega, L., Grünvogel, S. M., Natkin, S.: A new Methodology for Spatiotemporal Game Design, Quasim Mehdi, Norman Gough (Eds.), Proceedings of the CGAIDE'2004, Fifth Game-On International Conference on Computer Games: Artificial Intelligence, Design and Education, pp. 109-113
Disjunctor Selection for One-Line Jokes Jeff Stark1, Kim Binsted1, and Ben Bergen2 1
Information and Computing Science Department, University of Hawaii, POST 317, 1680 East-West Road, Honolulu, HI USA
[email protected],
[email protected] http://www2.hawaii.edu/~binsted 2 Linguistics Department, University of Hawaii, 569 Moore Hall, 1890 East-West Road, Honolulu, HI USA
[email protected] http://www2.hawaii.edu/~bergen
Abstract. Here we present a model of a subtype of one-line jokes (not puns) that describes the relationship between the connector (part of the set-up) and the disjunctor (often called the punchline). This relationship is at the heart of what makes this common type of joke humorous. We have implemented this model in a system, DisS (Disjunctor Selector), which, given a joke set-up, can select the best disjunctor from a list of alternatives. DisS agrees with human judges on the best disjunctor for one typical joke, and we are currently testing it on other jokes of the same sub-type.
1 Introduction Here we will describe a model and implemented system that is able to solve what we will refer to as the disjunctor selection problem for one-line jokes. This problem is most straightforwardly introduced with an example. The following is typical of the subtype of jokes we will consider: I asked the bartender for something cold and filled with rum, so he recommended his wife.
(1)
Most simply, the disjunctor is a short piece of text, almost always at the end of the joke, that completes the text in a linguistically and logically consistent, yet unexpected and humorous, manner. In this example, the disjunctor is the phrase “his wife”. The disjunctor selection task is that of providing an appropriate disjunctor, given a joke text with the disjunctor removed. That is, one must provide a phrase that will complete the text and make it humorous. Our system is designed to solve a simplified version of this problem, namely the selection of a disjunctor from a limited set of choices.
2 Background Our basic premise is that one-line jokes, such as (1), are humorous because they violate the initial expectations of the listener, and that this violation is resolved by M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 174 – 182, 2005. © Springer-Verlag Berlin Heidelberg 2005
Disjunctor Selection for One-Line Jokes
175
shifting from the initial knowledge frame used to understand the joke to another completely different knowledge frame [4]. For reasons that will need to be further investigated by psychologists, listeners often identify such violations and subsequent frame shifts as humorous. In (1), the expectation that is violated is that the bartender will fulfill the speaker's request by recommending a drink. Two factors contribute to the generation of this expectation. First, the listener initially interprets the object of the request as referring to a drink. Second, the listener makes an assumption (based on world knowledge and an understanding of how the structure of English sentences affects their semantics) that the reply to the request will involve the object of the request, as the listener has initially interpreted it. Hence, the listener's expectation is violated when the object of the reply turns out not to involve the object the listener had in mind. The violation of the listener's expectations results from the semantic interplay between two elements within the joke, the disjunctor and the connector [1]. As previously noted, the disjunctor in (1) is the phrase “his wife”. The connector is the phrase “something cold and filled with rum”. The connector and disjunctor are always ultimately resolved by the listener to have the same referent, even though initially the listener assumes that the connector refers to something very different. This occurs for two basic reasons. First, the connector is semantically ambiguous (i.e. both “cold” and “filled with rum” have more than one meaning) which makes it possible for it to have more than one referent. Second, the structure of the joke demands that the connector and disjunctor have the same referent in order for the text to make sense. In (1), the structure could be described as a request immediately followed by a reply. The connector is the object of the request, while the disjunctor is the object of the reply. In general, for most listeners, a sentence with this structure will make sense only if the object of the reply semantically fits with the object of the request. In other words, the two objects should relate to each other in a meaningful way. One common way two objects can be said to semantically fit is if their meanings can be resolved to the same referent. Many one-line jokes use reference resolution to violate the listener's expectations. For example, the ultimate meaning of (1), i.e. that it conveys an insult of the bartender's wife, stems in part from the fact that the connector and the disjunctor can be resolved to the same referent. Since the connector and disjunctor refer to the same object, the properties of the connector become the properties of this object. Therefore, the bartender's wife is “something cold and filled with rum”. That the connector and disjunctor are resolved to refer to same object in (1), presents the listener with the problem of finding an overarching knowledge frame capable of accommodating this fact. Initially the listener attempts to understand the joke within the context of the speaker ordering a drink in a bar. This initial overarching knowledge frame doesn't fit with the final realization that the connector and disjunctor refer to the same object: after all, the bartender’s wife isn’t a drink. Therefore the listener must find a new overarching knowledge frame to shift to in order to make sense of the joke. This new knowledge frame is that of the bartender insulting his wife, which semantically accommodates the disjunctor, ‘his wife’, having the properties attributed to the connector, i.e. ‘cold and filled with rum’.
176
J. Stark, K. Binsted, and B. Bergen
This joke is an example of an insult that plays on a negative stereotype for wives. Humor, in general, is often based on stereotypes, particularly negative ones. In (1), the disjunctor could have been 'his accountant', creating the weak joke: I asked the bartender for something cold and filled with rum, so he recommended his accountant.
(2)
This new ‘joke’ still fits our criteria, in that the listener still has to reinterpret the connector and make a frame shift in order to understand the text, but most listeners won't find it as humorous as (1), because it does not appeal to any well known negative stereotypes. Again, we leave it to psychology to answer why plays on negative stereotypes are so often at the root of humor. Based on the theoretical considerations discussed above, we have devised the following operational model to solve the disjunctor selection problem: 1. Generate all the alternative meanings for the connector. 2. Find the negative stereotypes that semantically fit with the connector meanings. 3. Order the list of stereotype/connector-meaning pairs according to the closeness of the fit. 4. From the list of potential disjunctors, select the disjunctors that semantically fit with at least one of the stereotype/connector-meaning pairs. 5. Find the overarching knowledge frame that fits with at least one of the stereotype/connector-meaning/disjunctor triples, and which differs from the original overarching knowledge frame. This algorithm solves the disjunctor selection problem for one-line jokes that use reference-based unification, given a finite set of possible disjunctors. Now, we will describe our implementation of this algorithm in DisS.
3 Tools DisS is written in Prolog, because several steps in the algorithm depend on unificationn, which Prolog has built-in. Our representational formalism is based on the Embodied Construction Grammar [3]. Finally, much of the content for the basic ontology is taken from Frame Net [2].
4 System Architecture Overall, the system is composed of two basic parts: a knowledge base and a set of operations that query or manipulate the knowledge base. 4.1 Knowledge Base The knowledge base (KB) contains everything the system initially knows about the world. All of this knowledge is in the form of representations called schemata. These schemata are based on the Embodied Construction Grammar formalism, although not all aspects of that formalism are employed. Schema are frame-based representations
Disjunctor Selection for One-Line Jokes
177
and, as such, most of their representational power stems from organizing a set of name-value pairs called slots under a single reference, which is the name of the schema. Therefore all schemata have a name that is identified by the slot called schema_name and a set of slots, called roles, which are the name-value pairs that identify the attributes associated with the schema. Since this is a typical frame-based system, all the usual properties of such systems apply, such as inheritance and the overriding of ancestor slot values by descendents. However, currently none of the schemata in DisS have methods or demons associated with them. There are two basic types of schema in DisS: classes and instances. Classes represent categories, which delineate sets of objects in the world. The most general class in the system's ontology is that of thing. All other schemas in the KB inherit from the thing schema. The objects within a class are instances of that class. Hence an instance schema is member of the category defined by some class schema in the system. Class schemas and instance schemas are differentiated by the name of the slot that indicates their inheritance relationship. Class schemas have a slot called subcase_of' that contains the names of the schemas they inherit from, whereas instance schemas indicate the class schemas they inherit from in a slot called instance_of. Below are examples of both class and instance schemas: /* cold3 - property of being unfeeling or unemotional */ schema([ schema_name(cold3), subcase_of(experiencer_subj), roles([ambiguous(cold)]), constraints([]) ]). /* Instance of a drink that is cold and filled with rum */ schema([ schema_name(drink_inst1), instance_of(drink), roles([ contents([rum]), temperature(cold1) ]), constraints([]) ]). The KB contains both general world knowledge (i.e. a very basic and minimal ontology) and domain specific knowledge. The domain specific knowledge includes classes and instances necessary to represent the incomplete jokes for the DisS to process, possible disjunctors for those jokes and the stereotypes that these jokes play off of. 4.2 Representing Connectors and Disjunctors Each incomplete joke represented in the KB is an instance of the joke schema and, as such, has a slot for the connector. The connector is itself an instance of a referent schema and has, amongst others, slots for the category of the referent and its attribu-
178
J. Stark, K. Binsted, and B. Bergen
tions. Each part of the connector that is semantically ambiguous is represented by making it an argument to a predicate called ambiguous. So since the word “cold” in “something cold and filled with rum” is ambiguous, it is represented as ambiguous(cold). The phrase “filled with rum” is handled in a similar manner. The connector schema for (1) therefore, looks like the following: schema([ schema_name(ref_inst_thing), instance_of(referent), roles([ category(thing), restrictions([]), attributions([ambiguous(cold), ambiguous(filled_with_rum)]), number(singular), accessibility([]), resolved_ref([]) ]), constraints([]) ]). Based on this knowledge, the system is able to generate alternative meanings by creating a set of new referent schema each with a different combination of meanings for each of the ambiguous parts of the connector. The KB contains three distinct meanings for “cold” and two for “filled with rum”, therefore the total number of possible meanings for the connector “something cold and filled with rum” is six. Note that all possible meanings are generated whether or not they make sense. Also note that the generated meanings are not semantically ambiguous. Once all the possible meanings of the connector have been generated, the system next tries to semantically fit each with a stereotype in the KB. Each stereotype has a slot, applies_to_category that indicates which category it is a stereotype of. The remaining slots of the stereotype identify the typical attributes associated with it. Below is an example of the bad wife stereotype as it is represented in the KB: schema([ schema_name(bad_wife_stereotype), subcase_of(stereotype), roles([ applies_to_category(wife), emotional_disposition([cold3]), cognitive_disposition([intoxication]), moral_disposition([unfaithful]) ]), constraints([]) ]). Matching alternative meanings to stereotypes is a two-step process. First, an attempt is made to semantically fit the category of a stereotype with that of the meaning currently being considered. A stereotype and a meaning are considered to fit if they are identical, or if one subsumes the other. If there is a match, then an attempt is made to fit the values of the remaining slots of the stereotype with those of the attributions
Disjunctor Selection for One-Line Jokes
179
of the meaning being considered. Again, a stereotype attribution and an alternative meaning attribution fit if they are identical, or if one subsumes the other. A count is made of the number of meaning attributions that fit with stereotype slot values as well as those that do not. The degree of how well a stereotype fits with a possible meaning is indicated by the difference between the number of meaning attributions that matched and the number of those that did not. Only those stereotypes that both fit with the category of the possible meaning and have a positive degree of attribution fit are included in a list of best fitting stereotypes for that meaning. As an example, consider matching the stereotype above with the generated meaning represented below: schema([ schema_name(ref_inst_thing7), instance_of(referent), roles([ category(thing), restrictions([]), attributions([cold3, filled_with_rum2]), number(singular), accessibility([]), resolved_ref([]) ]), constraints([]) ]). The schema called cold represents the definition of the word “cold” that means unemotional. Likewise the schema called filled_with_rum2 represents the meaning of the phrase “filled with rum” that means to be in a state of intoxication. Since the bad wife stereotype applies to the category wife which is subsumed by the category thing, the ref_inst_thing7 passes the first matching test with bad_wife_stereotype. The degree of the match is two, because one of the attributions (cold3) of ref_inst_thing7 is identical with a slot value in bad_wife_stereotype, and the other attribution (filled_with_rum2) is subsumed by a slot value in bad_wife_stereotype (intoxication). Therefore the bad wife stereotype would be placed on a list of best fitting stereotypes for the referent. Incomplete jokes (i.e. missing a disjunctor) are also represented in the KB. The schema representing (1) without its disjunctor is called joke_instance1 and is shown below: schema([ schema_name(joke_instance1), instance_of(joke), roles([ basic_semantics(req_reply_inst), connector(ref_inst_thing), expectation(drink_inst1), inital_context(ordering_a_drink_in_bar) ]), constraints([]) ]).
180
J. Stark, K. Binsted, and B. Bergen
5 Results When operating on joke_instance1, the system is able to a) generate all the possible meanings of the connector (i.e. “something cold and filled with rum”) and b) for each possible meaning generate a list of those stereotypes in the KB that meet the criteria mentioned above. Table 1 shows the output generated by the system when operating on joke_instance1 (note that only matches with at least one attribution match are shown). Table 1. Matching alternative meanings of the connector with negative stereotypes
Meaning of “cold”
Meaning of “filled with rum”
Having a low temperature Having a low temperature Feeling chilled Feeling chilled unfeeling, unemotional
Containing rum Intoxicated with rum Containing rum Intoxicated with rum Containing rum
Unfeeling, unemotional
Intoxicated with rum
Matching stereotypes (degree of attribution fit) None Bad wife (-1+1=0) None Bad wife (-1+1=0) Bad wife (1+-1=0), accountant (1+-1=0) Bad wife (1+1=2), accountant (1+-1=0)
The system's output listed above makes clear that of all the possible meanings generated for the connector the one that has the best fit with a stereotype in the KB is the one that means “something unemotional and intoxicated”. The best fitting stereotype in this case is the bad wife stereotype, which only applies to things that are members of the wife category. According to our model, disjunctor must ultimately fit with one of the possible meanings of the connector and, in order to generate humor, a negative stereotype that fits with this possible meaning. Selecting as an alternative meaning for the connector the referent that means “something unemotional and intoxicated” will impose as a selection criterion for the disjunctor that it fit with the wife category. Therefore, only those possible disjunctors represented in the KB that fit with the wife category could be reasonably considered by the system. The referent schema that represents the bartender's wife, bartenders_wife exists in the KB,1 and is an instance of the category wife, so there is a fit. There is still one step required to show that bartenders_wife (or rather, an English phrase referring to this schema) would make a good choice as disjunctor. The system must still satisfy the major theoretical premise upon which the system rests, namely that jokes force a frame shift. What the system needs to do is show that the alternative meaning of the connector and the selected disjunctor fit within a frame that captures the overall meaning of the joke, and differs from the frame supported by the set-up (i.e. ordering_a_drink_in_bar). Luckily, such a frame exists in the KB, namely the insulting_someone schema, which requires that someone either directly 1
It was beyond the scope of this project to build a system able to infer the existence and attributes of the bartender’s wife, so a suitable schema was hard-coded into the KB.
Disjunctor Selection for One-Line Jokes
181
connected to someone in the scene (e.g. married to the bartender) or generally wellknown be identified with a negative stereotype. So, DisS is able to choose bartenders_wife as the best disjunctor for joke_instance1 from a list of alternative disjunctors, shown in Table 2. All human judges asked to choose from the same list of disjunctors made the same choice.2 Table 2. Alternative disjunctors for joke_instance1
Disjunctor schema bartenders_wife bartenders_accountant
Disjunctor text “his wife” “his accountant”
mai_tai
“a Mai Tai”
accountants_wife
“an accountant’s wife”
basketball
“a basketball”
Suitable disjunctor? Yes No: degree of match too low No: no negative stereotype, no frame shift No: not known or connected to scene, so no frame shift No: doesn’t fit with alternative meaning
The system should of course also be demonstrated on more than one joke.3 We are currently in the process of adding schemata to the KB to support processing of the following joke: I asked the bartender for the quickest way to the hospital, so he recommended a triple martini.
(3)
This is based on the rather more famous joke: I asked the lady how to get to Carnegie Hall, and she said “Practice, practice, practice!”
(4)
6 Conclusion We have described an operational model of the connector/disjunctor relationship in one type of one-line non-punning joke, and implemented a system, DisS, which can choose the best from a list of possible disjunctors for a joke set-up. DisS has been shown to agree with human judges on one typical joke, and we are currently testing it on other jokes of the same type.
2
So far, we have asked five fluent English speakers somewhat informally. We plan to complete a more formal evaluation in time for INTETAIN 2005. 3 We plan to have two more jokes implemented in time for INTETAIN 2005.
182
J. Stark, K. Binsted, and B. Bergen
References 1. Salvatore Attardo. 1994. Linguistic Theories of Humor. Mouton de Gruyter, Berlin and New York. 2. Collin Baker, Charles Fillmore, and John Lowe. 1998. The Berkeley FrameNet project. In Proceedings of the COLING-ACL 1998, Montreal, Canada. 3. Benjamin K. Bergen and Nancy C. Chang. 2002. Embodied Construction Grammar in Simulation-Based Language Understanding. Technical Report TR-02-004, International Computer Science Institute, Berkeley. 4. Seana Coulson. 2001. Semantic Leaps: Frame-shifting and Conceptual Blending in Meaning Construction. Cambridge University Press, Cambridge, U.K. and New York.
Multiplayer Gaming with Mobile Phones - Enhancing User Experience with a Public Screen Hanna Strömberg1, Jaana Leikas1, Riku Suomela2, Veikko Ikonen1, and Juhani Heinilä1 1
VTT Information Technology, P.O. Box 1206, 33101 Tampere, Finland {Hanna.Stromberg, Jaana.Leikas, Veikko.Ikonen, Juhani.Heinila}@vtt.fi 2 Nokia Research Center, P.O. Box 100, 33721 Tampere, Finland
[email protected] Abstract. We have studied the use of a public screen integrated to a mobile multiplayer game in order to create a new kind of user experience. In the user evaluations, the game FirstStrike was tested by eight user groups, each containing four players. The evaluations showed that communication between the players increased with the usage of the public display and alliances were built. It was also found that the possibility to identify the players by adding the players’ photographs into the shared display makes the game more personal. We believe that this new way of communication is a result of using the shared public screen in a mobile multiplayer game.
1 Introduction Mobile multiplayer games are the natural next evolutionary step for mobile entertainment. Although there are several approaches to the mobile gaming with shared handheld displays [e.g., 4, 12, 2] most of them do not study the possibility to have a shared large display. Naturally, the characteristics of the existing mobile games are more “on the go games”, where the players have the possibility to move from place to place while playing. There have been several attempts to use shared displays in stationary environments [e.g. multiple monitor use 6] and in face-to-face consultations [11]. Large interactive displays have been widely studied especially in the field of computer supported cooperative work (CSCW) and have been used to support various group-based and cooperative activities [e.g., 3, 9]. Also mobile collaboration environments have been mostly related to workspaces [e.g., 5]. Connecting a shared large display to mobile handheld devices has been investigated through various applications [e.g., 10, 5] but games have not been in focus. Related to entertainment there are applications that encourage into social interaction through a public display by e.g. publishing photos in a public space [e.g., 1, 16]. The fact that a public display enhances interaction between the participants has also been found e.g. in [13]. Sanneblad et. al. [12] have launched the term Collaborative Games M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 183 – 192, 2005. © Springer-Verlag Berlin Heidelberg 2005
184
H. Strömberg et al.
since the players are required to collaborate with each other to manage the sharing of each others displays. Our experimental study concentrates on multiplayer gaming on mobile phones enhanced with a shared large display. Public displays have been used in gaming for some time and the major consoles, for example Nintendo Gamecube, use a television to display public information to a group. In this work we use public screens as an extension to games played with mobile phones. We consider public screens in public spaces, and how they could be used to extend mobile games. Mobile phones are carried by the users most of the time, and public screens are found in many places. If the game and the public displays were combined, new and interesting games could be made. In this study our aim was to find out prerequisites for a successful user experience when playing a mobile multiplayer game. We wanted to see how people would respond to and use mobile multiplayer games on a public display, and what is the added value of the public display in the game. The study is part of an ITEA (Information Technology for European Advancement) project Nomadic Media [8] that has a vision of combining people, technologies and services in a way that allows consumers to enjoy content and interactive services at the times and in the places they prefer, using the devices that best suit their context, at home or on the move. One of the outcomes of the project is a public screen concept for multiplayer gaming with mobile phones in different contexts, e.g. at the airport.
2 FirstStrike Game We studied mobile multiplayer gaming through a sample game, which is described in this section. FirstStrike is a four player mobile game developed with MUPE [15]. MUPE is a client-server application platform that can be used in any mobile phone that have MIDP2. In addition to phones, any display can be connected to the system, which has been presented in [14].
Fig. 1. Displays of the public screen (in the middle) and of the mobile phones
Multiplayer Gaming with Mobile Phones - Enhancing User Experience
185
The game puts the players in control of nations that are at war. Each player has two cities at start. The game is a turn based strategy game, in which the players have a fixed time (10 s) to decide an action and a target city for it. The last player with a city alive wins the game. The players have a varying number of actions, from which they can choose one in each round. At the start of the game, the players can attack (nuke), defend, and spy on the status of other players’ cities. The players know the status of opponent cities from the start of the game, or from the last spy action, and the players know only their own status constantly. As the game progresses, the players get additional actions. When one of the cities has less than 30 inhabitants (100 at start), the player can evacuate the citizens from the smaller city to the bigger one. After one city is empty, the player gets two new moves: Megabomb and Nuclear Waste Army (NWA). NWA steals citizens from an opponents city. Megabomb makes massive damage, but can be only used once, and it also removes the NWA action. A public screen changes the game. Firstly, the public screen shows the status of all the players to all others, thus the spy action becomes useless. Secondly, the other players’ photos are seen on the large display, which makes it easier to recognize the other players, especially if playing in the same location. The game and the screenshots of the public and private screens can be seen in Fig. 1.
3 Evaluation Set Up To see how people would respond to and use mobile multiplayer games and a public display, we have conducted a user evaluation of the FirstStrike game with eight player groups (altogether 32 players). The tests were carried out in autumn 2004 and in spring 2005. The evaluation of the FirstStrike game was twofold. In the first evaluation session four groups of four persons played the game with the mobile phones. The test environment was at VTT Information Technology’s usability laboratory. In the first evaluation session the test players were students and business travelers. The aim of the first test was to study mobile playing as well as the added value of the multiplayer gaming in respect to user experience. In the second evaluation session the users played the same game using mobile phones and following the game events from a public screen at the same time. The tests took place at Nokia’s usability laboratory, where the game was displayed on the wall by a data projector. The player groups consisted of IT professionals, young adults and business travelers. The aim of this second phase of the evaluation was to get feedback about the public screen as means for acting together and about the added value of the large shared display. At the evaluation we aimed to test the game in a potential usage context, i.e. at the airport. Indirectly, the evaluation of user response was also expected to give essential input for assessing the business potential of the game. In the user tests the users played the game several times. In addition to predefined tasks the players played the game freely. During the game the researchers observed the playing. After each game session a short constructive discussion was carried out. The players were then interviewed as a group after the last game session. The group
186
H. Strömberg et al.
interview concentrated on mobile multiplayer gaming and the usage of the public screen. Below we highlight the main findings of the evaluations. First we describe the mobile device as a game controller, then the public screen as a shared display. After that, we describe the usage of the public screen in mobile multiplayer gaming through different attributes that were found suitable for conceptualising and evaluating the usage of a public screen in mobile multiplayer gaming.
4 Results of the Evaluation 4.1 Mobile Device as a Game Controller In the evaluations the Nokia phone models 6600 (in the first test) and 7160 (in the second test) were used as game controllers. The players attended the game through their individual phones using the navigation buttons and the joystick button. There was no need to touch the numeric keys in this specific game. The players felt that the joystick button of the phone is easy and intuitive to use. The accuracy of the screen of the phones is high enough for playing. As the game controller was a mobile phone, there were functions that are relevant for a general phone use but inhibit or disturb playing a game. One of these functions is the screen light that automatically goes off after being on for a while. The fact that the light on the mobile phone’s screen turned off in the middle of the game disturbed playing. The players suggested a possibility to choose the way of controlling the game. They felt that moving around in the game could also happen by using the numeric keys of the phone instead of using only the navigation buttons. This might of course increase the amount of errors as the number keys of the device are rather small. It was also suggested that the game control keys could be different than the keys to receive e.g. a phone call during the game. The mobile device could also act only as a control device and the actual game would realize on the public screen. Joining the game should be possible without having to go through any manuals or other long instructions. In this particular game the players have to be technically oriented as joining the game is technically rather complicated. The equality of the players regarding to their mobile phones was brought up. Everyone, despite of the model of the mobile phone, should have the chance to play the game. That is, it should be possible to play the game with a variety of mobile phone models. 4.2 Public Screen In our tests the public screen enhanced the user experience in the multiplayer game: “The public screen is ‘the thing’ on multiplayer game!” The public screen was described as “a multi-function screen” containing various functions and information besides the mobile multiplayer game, e.g. advertisements, sponsor adds, chat and most importantly, marketing the game. In this particular game also more information concerning the game events and the players’ actions was welcomed.
Multiplayer Gaming with Mobile Phones - Enhancing User Experience
187
In this set up the players were expected to follow two user interfaces at the same time, namely the public screen and the mobile device. Some players had difficulties to follow the two displays at the same time. However, the size and distance of the public screen that was used in the test conditions (see Fig. 2) was seen appropriate in other locations also. Information Sharing. Public information was seen meaningful to present on the public screen but any private information was seen very important to appear only on one’s own mobile device. Also, it is essential that the game is not interrupted when one of the players gets e.g. a phone call. If a person receives a phone call in the middle of a game there is a need to find out who is calling in order to make a decision whether to answer or not. However, the game should not be interrupted if the person answers the call. Especially in a team game the interruption could have a negative effect as all the players’ game stops if one of them answers a call. To avoid this the player could use e.g. a certain default action when (s)he is not able to play and does not want the game to stop. A mark on the absent players picture on the public screen was suggested to inform the other players about the absence of the player. Also using Artificial Intelligent (AI) to replace an absent player in the game was welcomed.
Fig. 2. Playing a mobile multiplayer game with a public screen
User Identification in the Game. We wanted to study the importance of identification of the user on the public screen. When playing without the shared display each mobile device participating in the game session played is assigned a unique color. The players did not know beforehand by what color each player was represented in the game and they had to find this out by asking each other. When the public screen was integrated into the game the players took photographs of them selves (see Fig. 3) and published them on the public screen. This way the players knew which cities belong to whom. Most of the players attending the evaluations were ready to publish their photos on the public screen. However, publishing the photo was not totally appreciated by everyone in the game. A personal photo on the screen was not that much welcomed at least among some young women who were worried about their looks. All in all, according to the players it brings added value to the game to see who the other players are. It was commented e.g.: “The game becomes more personal when it includes the pictures of the players.” Also a picture uploaded from a picture gallery on ones own
188
H. Strömberg et al.
device was suggested because it saves time. As well, a logo or icon designed by the players was welcomed. The quality (resolution etc.) of the picture was not seen important; the purpose off the picture is to recognize the other players. Registering to the game was suggested e.g. to avoid inappropriate photos to be published on the screen. Taking the photos with the mobile phone was smooth for those players, who were familiar with mobile phones equipped with a camera. The others had slight difficulties in using the camera for the first time. Many players asked the other players to take a photo to have the picture better adjusted on the small screen.
Fig. 3. The players are taking photos of each other to publish on the public screen
4.3 Interaction Between the Players All the players enjoyed playing with other players in the same location. The players felt that a multiplayer game is a social action and hence totally different from a one player game as a human opponent in the game changes the characteristics of the game. The game was seen as a good alternative to board games, also. The main finding when comparing playing with and without the public screen was that using the public screen increased communication between the players. When playing the game without the public display there typically was not much interaction between the players during the first game session. Later on some discussion arouse when the players learned the game and recognized each others’ game characters. The players claimed that there was nothing personal in the mobile game situation when played without the public screen and the content of discussions would have been the same even if they did not know the other players before. Knowing the rules of the game affects on playing and the feeling of being a member of a group gets stronger the more the group members learn to play. The players also allied themselves with other players more when using the public screen. The shared display gave the possibility to talk aloud about the game strategies. Also the possibilities to make alliances before starting the game or during the game e.g. using SMS-messages were suggested. Now alliances were made by eye contact or by whispering. In this game sitting in front of each other in the room helped making alliances because the eye contact was easier to make. An example of a comment to other player: “I attacked your city as revenge because a while ago you did not ally yourself with me.” During the game there were many alliances made and also some playful quarrels about the alliances (who allies with whom). The fact that all the information
Multiplayer Gaming with Mobile Phones - Enhancing User Experience
189
concerning the game played with the large display is public had an effect on alliances. The possibility to keep certain information private may have affected on the playing because the conspiracies could then be forced in secrecy as argued in [7]. 4.4 Context for the Public Screen The user experience of a game is determined not only by the game itself but also the context in which it is played. A mobile game works at the context, where people have idle moments. As contexts for the public screen the users suggested places where one has to wait and/or where there is nothing to do (e.g. railway station, entrance hall of a movie theatre, schools auditorium, work places, library, airplane, train, bus) and “hanging out” places (e.g. shopping mall, café, Internet café, bar). A natural context for a public screen, not mobile though, is a home TV-monitor. As regards to the player group, playing the game with friends was preferred but playing with strangers was not denied either. Most of the players found the game as a nice way to meet people. There were some differing comments as well: “I would not play the game with strangers. I usually play e.g. computer games to have nice time with my friends, not just because of the game itself.” The players mentioned that the game must be designed specially for its usage environment. Related to the context, the sounds are useful for the players but e.g. in the train one must have the chance to use a headset. At the public context the sounds were seen disturbing because of the surrounding noise. Public Screen at a Public Context. The attitude towards playing in a public context was twofold. Some players found that the public space would not affect on their playing. On the other hand, there were players who would be nervous when other people would watch the playing. The users’ concern to use a public system that can be observed by others deserves consideration, as discussed in e.g. [17]. The players mentioned that playing with friends lowers the threshold to play on a public location. Some players thought that in a public space they would not shout aloud their remarks concerning the game. A comment before the game: “I am not interested in playing computer games. Why would I play in front of an audience if I don’t even like to play alone?” It was suggested that the people following the game on the public space as audience could somehow comment the game. It was said e.g. that: “The idea of the public screen is that the others (an audience) can follow the playing. The game may also arouse discussions among the audience.” However, it was wondered if the game would be interesting and entertaining enough to follow. A comment: “Why would people watch other people playing a game? I understand watching a basketball game on the street but a mobile game on a public screen would not be that interesting to follow.” The main difference between a publicly available game and a public display is the fact that, in the latter case, everyone moving around in the public space can follow the playing. Particularly in a public context, e.g. at the airport, it has to be easy and smooth to adopt and join the game. At the public context the amount of players needed should not necessarily be set and the group was hoped to be preferably more than four players. Also an ongoing game was suggested, so the players could join and leave the
190
H. Strömberg et al.
game according to their time etc. preferences. On the public context there could be information about the free places in the game on the public screen. According to the players it is easier to join the game when there are already some people playing. Informing and advertising about the game and inviting to play was seen important at the public context. It was also suggested that the same game could be available in several locations and also the possibility to continue playing mobile (e.g. using Blutooth) after leaving the public screen was mentioned. 4.5 Distribution of the Service and Willingness to Pay Ordering the game was preferred to be done into one’s own mobile device (e.g. SMS, Bluetooth bookmark). Also tags near the public screen when ordering the game were mentioned. The price of the game should not be much more than the price of a phone call. It was suggested, that the game could be offered as a bonus e.g. by the airline company at the airport. All in all, the charges were hoped to be related to the amount of games, not the duration of the game. The players were quite suspicious towards the possibility to be connected to the game automatically. They do not want to receive anything to their phone what they have not asked for themselves. The player has to have the freedom to choose whether (s)he wants to be recognized by a system and receive any messages. Once having granted the permission to be informed, the player could get a message from the system to her/his mobile phone telling that the game is available and that, e.g., there is one place available in the game at the moment. Networked games were suggested among the users. In a networked game teams in different geographical locations play against each other on a shared public screen. Groups gathered e.g. in different airports or shopping centres could play against each other as teams and see the shared game on the screen. Here the photos of the teams would give a lot of added value to networked gaming.
5 Conclusions Public screen as such seems to be usable for gaming for several kinds of user groups. According to our studies it seems to enhance the interaction between the players through lifting the individual player’s sight from a personal single mobile device up to a larger shared environment. As the players follow the game on a large screen they more easily start to take eye contact and communicate with other players, also. If the game is well designed for public screen purposes it can serve as an “ice breaker” and encourage people to communicate with each other even if they do not know each other beforehand. In this specific public screen game the players even profited from communicating when making alliances with each other. This study concentrated on multiplayer gaming with mobile phones. To become a trend the a mobile multiplayer game on a public screen must be easy and smooth to adopt and join, error free with good functionality, inexpensive and well marketed. Publishing one’s photograph on a public screen was accepted in most cases. It was felt that personal photographs of the players’ faces brought added value to the game as the photos made it possible to get to know the players in a group of strangers. Also,
Multiplayer Gaming with Mobile Phones - Enhancing User Experience
191
even when playing among friends, the most valuable information that the photos gave was indicating the cities of the players. In this particular game it was important to know which city was represented by whom. However, a personal photo on the screen was not that much welcomed at least among some young women who were worried about their looks. In cases where a photograph was not seen necessary a possibility to upload a ready-made picture from a picture library or a usage of a personal avatar was welcomed. Based on our study, we believe that the use of shared public displays in mobile multiplayer games has potential to introduce new user experience and communication models, which would not have been found without this novel way on gaming.
6 Future Work Our future work involves the third phase of the FirstStrike game evaluation. The aim is to evaluate the mobile multiplayer game played in a public context with a public screen enhanced with gesture control. An interesting line for the game development could be to study the possibilities to let the audience following the game in the public locations to effect on or even join the game. In the future it would be advantageous to evaluate a multiplayer game in a public location to see how the audience really influences on the players as well as how the audience reacts on the situation. The evaluation raised many interesting questions. According to the evaluations it would be valuable to study if there are differences between genders when playing. Also the existence of cultural differences when playing games would be important to find out. Does e.g. communication with strangers vary from nationality to nationality? And finally, age related questions of gaming would be essential to find out, e.g. similarities and differences of gaming concerning children and elderly people.
References 1. Brignull, H., Izadi, S., Fitzpatrick, G., Rogers, Y., Rodden, T.: The Introduction of a Shared Interactive Surface into a Communal Space. In Proceedings of the 2004 ACM conference on Computer supported cooperative work. Chicago, Illinois, USA (2004) 2. Chalmers, M., Tennent, P. Recording and Reusing Mobile Multiplayer Play. Workshop paper in Games and Social Networks: A Workshop on Multiplayer Games. University of Leeds, UK, British HCI conference (2004) 3. CSCW 2002 Workshop on Public, Community and Situated Displays, November 16, New Orleans, http://www.appliancestudio.com/cscw/workshophome.htm 4. Danesh, A., Inkpen, K., Lau, F., Shu, K., Booth, K.: GeneyTM: Designing a Collaborative Activity for the PalmTM Handheld Computer. In Proceedings of CHI 2001. Minneapolis, USA (2001) 5. Divitini, M., Farshchian, B.A., Samset, H.: UbiCollab: Collaboration Support for Mobile Users. In Proceedings of the 2004 ACM symposium on Applied computing. Nicosia, Cyprus (2004) 6. Grudin, J.: Partitioning Digital Worlds: Focal and Peripheral Awareness in Multiple Monitor Use. In Proceedings of CHI 2001, Minneapolis (2001)
192
H. Strömberg et al.
7. Magerkurth, C, Stenzel, R., Streitz, N., Neuhold, E.: A Multimodal Interaction Framework for Pervasive Game Applications, In Proceedings of AIMS 2003. Seattle, USA (2003) 8. Nomadic Media project: http://www.extra.research.philips.com/euprojects/nomadicmedia/index.htm 9. Ohara, K., Perry, M., Churchill, E., Russel, D. (eds.): Public and Situated Displays: Social and Interactional Aspects of Shared Display Technologies. London: Kluwer (2003) 10. Paek, T., Agrawala, M., Basu, S., Drucker, S., Kristjansson, T., Logan, R., Toyama, K., Wilson, A.: Interactions with Shared Displays: Toward Universal Mobile Interaction for Shared Displays. In Proceedings of the 2004 ACM conference on Computer supported cooperative work. Chicago, Illinois, USA (2004) 11. Rodden, T., Rogers, Y., Halloran, J., Taylor, I.: Designing Novel Interactional Workspaces to Support Face to Face Consultations. In Proceedings of CHI 2003. Ft. Lauderdale, USA (2003) 12. Sanneblad, J., Holmquist, L.E.: “Why is everyone inside me?!” Using Shared Displays in Mobile Computer Games. In Proceedings of International Conference on Entertainment Computing (ICEC). Einhoven, The Netherlands (2004) 13. Sumi, Y., Mase, K.: AgentSalon: Facilitating Face-to-Face Knowledge Exchange through Conversations Among Personal Agents. In Proceedings of AGENTS’01. Montreal, Quebec, Canada (2001) 14. Suomela, R.: Public screens and private mobile phone screens in multiplayer games. In The Workshop on Applications for Wireless Communications WAWC, Volume 211 of Acta Universitatis Lappeenrantaensis (2005), 17-26 15. Suomela R., Räsänen E., Koivisto, A., Mattila, J.: Open-Source Game Development with the Multi-User Publishing Environment (MUPE) Application Platform. In Proceedings of the Third International Conference on Entertainment Computing 2004 (eds. Rauterberg, M.), Lecture Notes in Computer Science, Vol 3166. Springer-Verlag Berlin Heidelberg (2004), 308-320 16. Thoresson, J.: PhotoPhone Entertainment. In CHI '03 extended abstracts on Human factors in computing systems. Ft. Lauderdale, Florida, USA (2003) 17. Wasinger, R., Kruger, A., Jacobs, O.: Interacting Intra and Extra Gestures into a Mobile and Multimodal Shopping Assistant. In: Gellersen, H.W. et al. (eds.): Pervasive 2005, Lecture Notes in Computer Science, Vol 3468. Springer-Verlag Berlin Heidelberg (2005), 297-314
Learning Using Augmented Reality Technology: Multiple Means of Interaction for Teaching Children the Theory of Colours Giuliana Ucelli1, Giuseppe Conti1, Raffaele De Amicis1, and Rocco Servidio2 1
Fondazione GraphiTech, Salita dei Molini 2, 38050 Villazzano (TN), Italy {giuliana.ucelli, giuseppe.conti, raffaele.de.amicis}@graphitech.it http://www.graphitech.it 2 Dipartimento di Linguistica, Università della Calabria, Via Pietro Bucci, Cubo 17/B, 87036 Rende, Italy
[email protected] http://www.linguistica.unical.it/linguist/default.htm
Abstract. Augmented Reality technology permits the concurrent interaction with the real environment and computer-generated virtual objects, thus making it an interesting technology for developing educational applications that allows manipulation and visualization. The work described extends the traditional concept of book with rendered graphics to help children understand fundamentals of the theory of colours. A three-dimensional virtual chameleon shows children how, from the combination of primary colours, it is possible to get secondary colours and viceversa. The chameleon responds to children’s actions changing appearance according to the colours of the surroundings. Our tangible interface becomes an innovative teaching tool conceived for supporting school learning methods, where the child can learn by playing with the virtual character, turning over the pages of the book and manipulating the movable parts. The main scientific contribution of this work is in showing what the use of augmented realitybased interfaces can bring to improve existing learning methods.
1 Introduction The function and role of objects manipulation and interaction with tools for learning have been addressed by the Constructivism [1], which states that knowledge is the result of an active engagement of individuals, which learn by doing, and it is anchored to the individuals’ contexts. In the opinion of Bruner [2] there is a direct relation between mental faculties of the human beings and their capacity to use tools and technologies that allow them to amplify their power. The authors believe that the use of tools encourages two important aptitudes of humans: the capacity of exploring the contents of the observed phenomenon, and the use of tools which allow the reproduction of that phenomenon. From a behavioral point of view interaction and exploration are indivisible: they represent two aspects of the same process, the first is the internal aspect, the second the external means. Further, according to Vygotskij [3] the cognitive development of M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 193 – 202, 2005. © Springer-Verlag Berlin Heidelberg 2005
194
G. Ucelli et al.
the subjects is influenced by the relations that they are able to establish with the surrounding environment, with others, and with the cultural products of their society. Vygotskij also noted that when children encounter a problem too difficult to solve on their own, they often can cope with it through collaborative discovery if they receive assistance from a more knowledgeable person. This pedagogical approach follows also the indications of cognitive science that asserts that knowledge cannot be transferred by teachers enunciating already structured elements. It has to be piled up by individuals under the expert guide and mediation of teachers, via the interaction with others and with objects, materials, texts and experts in an environment that stimulates the personal learning style of each subject [4, 5]. Our knowledge is in fact the result of an active organization of our own experiences, which can be achieved through interactive learning methods and through tools and environments that encourage an active learning of concepts and their meanings. Recent technologies offer the right tools for developing integrated learning environments that support manipulation of physical objects and visualization of contents, enriching the learning experience. Manipulation of familiar objects (e.g. the pages of a book, toy bricks) and acting in a physical space enhance the sense of participation and of discovery of subjects. Augmented Reality (AR) technology provides a blend of manipulation and visualization through the overlay of synthetic environments over real ones. AR excites children’s imagination without forcing them to lose contact with reality and provides a tangible interface metaphor that is commonly in use in educational settings where physical objects are used to communicate meanings. According to Holt [6] children learn through play, everything becomes a learning experience for them, playing is learning, and playing is fun. Following this view the goal of this research work is in describing the potential offered by AR in enhancing traditional learning methods for children. The authors present as example of the new interaction means offered by AR an “augmented” pop-up book that teaches children rudiments of theory of colours. What the authors call the Book of Colours has been designed specifically for feeding children’s curiosity and for providing them with multiple means of interaction that could enhance their gaining of knowledge. In Section 2 the authors describe current work on tangible and augmented-reality interfaces for educational purposes as background on related work. Section 3 presents the Book of Colours providing an overview on its features, the description of cognitive aspects and multi-level interactions, a synthesis on the learning methodology that the authors pushed forward, and design and implementation aspects. In Section 4 the authors analyse the various steps of a test session and comment on the behaviour of children interacting with the book. Finally, Section 5 describes further research to enhance the Book of Colours and it provides some concluding remarks.
2 Blend of Manipulation and Visualization Using Tangible Interfaces and Augmented Reality During the last few years, many digital learning environments have been developed based on the constructivist approach. These environments allow the physical manipulation of everyday artefacts and to accomplish ordinary actions that then cause changes in an associated digital world, thus capitalising on people’s familiarity with
Learning Using Augmented Reality Technology
195
their way of interacting in the physical world [7 cited in 8]. The “tangible bits” paradigm follows this approach to computing [9], where the digital world of information is coupled with novel arrangements of electronically embedded physical objects. Thus providing different forms of user’s interactions and system behaviours that are in contest with the standard desktop set-up of keyboard, mouse and monitor [10]. Several examples include a Digital Manipulative [11, 12] which uses physical coupling between objects to control multimedia narratives, and the Ambient Room [13], a personal work space that explores the use of “ambient displays” to give the occupant a sense of both physical and virtual activity around them [9]. These new manipulatives with computational power embedded inside, are designed to expand the range of concepts that children can explore through direct manipulation, enabling children to learn concepts that were previously considered “too advanced” for them [11]. Some learning environments use Virtual Reality technology and tangible interfaces for teaching complex actions to children, such as how to grow a garden [14], how to read [15] and rudiments of colours [16]. Research into tangible computing has taken a step back and realised that, while we currently interact with computers through physical objects (i.e. keyboards, mice and displays), we can better exploit our natural skills if we focus on interacting with the physical objects themselves [9]. Billinghurst et al. [17] and Ishii et al. [18] use Augmented Reality for enriching tangible objects of virtual information and graphics. Billinghurst [17] in the MagicBook developed an augmented storybook where readers use Head-Mounted Displays for looking at three-dimensional animated virtual scenes while they interact with a physical book. In the MagicBook, Augmented Reality (AR) technology provides the means for merging manipulation and visual representation. This blend of means of interaction helps children better understand concepts and develop skills for acquiring higher level methods of thinking. AR enriches the educational experience with several interesting cognitive aspects [19], such as: • Support of seamless interaction with real and virtual environments • The use of a tangible interface metaphor for object manipulation • The smooth transition between reality and virtuality AR extends the physical world by enriching it with useful information and by providing a new level of interaction with the physical world in an intuitive manner. The manipulation of real objects provides an emotional richness that goes far beyond the cold screen-based representation. The adoption of tangible interfaces eliminates the problem of interaction with computers using keyboard and mouse that for young children could result in a significant obstacle. Tangible objects, such as cards or books [17], become transitional interfaces realizing the ultimate children’s fantasy, as in the fairy tales, of transforming real objects into fantastic ones. Such a playful approach facilitates the retention of information and it enhances interest in learning. The following section describes the Book of Colours, an example of tangible interface that uses Augmented Reality technology for teaching children fundamentals of theory of colours.
196
G. Ucelli et al.
3 The Book of Colours The Book of Colours is an example of implementation of Augmented Reality technology in children’s education. The authors realized a digital three-dimensional version of the children’s beloved cardboard pop-up and movable book that explains the basic concepts of the theory of colours (see Figure 1). Once experienced with HeadMounted-Displays (HMD), or using a camera and a display, this cardboard book is “augmented” with a three-dimensional character, a chameleon, which helps the comprehension of the theory of colours reproducing the colours with its skin. The Dogon people, an indigenous tribe of the Sahara Desert in Africa, consider the chameleon “the animal that received all the colours”, which well expresses the natural gift of this animal of changing colour of skin according to the surrounding environment. The choice of the chameleon introduces a new interesting concept for children and it can excite their curiosity. Further, our high detailed three-dimensional model offers a rare opportunity for children to observe this tropical animal closely. The cover shows the content of the book: the three primary colours (i.e. yellow, red and blue), which cannot be obtained as combination of other colours. The first page, as an index of the book, summarizes the content and presents to children the additive property of colours showing the secondary ones (i.e. green, purple and orange) as results of the combination of the primary colours.
Fig. 1. Three views of the Book of Colours: (left) the cover with the primary colours, (centre) the first page showing some properties of colours, (right) the virtual chameleon showing the result of the combination of two colours (i.e. yellow and red) through the colour of its skin
Children experience these initial pages as a traditional book but from the second page onwards the virtual chameleon shows children the formation of secondary colours changing the colour of its skin. The virtual chameleon becomes the key of the various pages of the book for its appearance characteristics. The Book of Colours can be seen as an AR version of a traditional book that can be experienced at two different levels: at a physical level turning over its pages, looking at the pictures and playing with its movable parts, and at a virtual level observing and interacting with the synthetic character that “augments” the real book. The next sections will provide more insight on the system implementation, on cognitive aspects of the book and on the learning methodology adopted.
Learning Using Augmented Reality Technology
197
3.1 Implementation and Design of the Layout The “augmented” part of the Book of Colours has been implemented as a customization of ARToolkit [20], a C++ AR library developed at Osaka University. This library supports single camera position and orientation tracking of square marker patterns for positioning virtual models in the space. Figure 2 shows the various phases that the system has to process to overlay the virtual chameleon on the physical book. The process starts receiving as input the video stream coming from the camera. Each frame is analyzed looking for black squared shapes (called markers). Once a marker has been recognized, the system calculates the position of the camera relative to the black square. Each marker differs from the other for its internal picture, called mark. Once the internal mark is recognized as one of those images stored in the internal database, its ID number is retrieved together with the associated 3D model in VRML format. The model is rendered on top of the video of the real world and so appears fixed on the square marker. The output is finally shown in the monitor and in the Head-Mounted Display so when children look at both output devices, they can see graphics overlaid on the real world.
Fig. 2. Augmentation process and output
The constraints brought by this technology, for instance the need to use particular squared images to allow the pattern recognition of tracking markers, has been used as an occasion to design iconographic pictures of the chameleon (see Figure 3). Attention in the design of the markers has been paid in order to facilitate the comprehension of the book for those readers not using AR equipment. Going through the pages of the book children can find clues of the presence of the chameleon through the images of the head, foot and tail in various positions of the virtual character.
Fig. 3. (left) Parts of the chameleon are used for the design of the markers; (right) Four markers
198
G. Ucelli et al.
The Book of Colours has been specifically designed for children’s use therefore its pages are built of robust and water-resistant material. The choice of the material was made paying particular attention to the tactile sensation that the readers perceive once having the book in their hands. The graphical structure of the book is rigid in order to focus children’s attention on the always-new means of interaction that are proposed over the pages. The use of colours is obviously central to the book. This is at the same time a way to attract children’s interest and the content of the book. 3.2 Cognitive Aspects and Levels of Interaction Constructivism [1] states that physical artefacts encourage and push children to action, and this acting in turn results in experiences that allow a gain of knowledge. Exploiting the effectiveness of physical artefacts, the Book of Colours amplifies the cognitive ability of children through manipulation, and it accompanies it with an adequate visual representation of the content. Children are provided with a tool that offers various levels of interaction in order to make them gradually comprehend the main properties of colours. The proposed interactions are not procedural mechanisms that would limit the child to a mere manipulation of objects, but they stimulate to acquire profound abilities that are confirmed through actions. Bruner [2] states that ability in acting requires the organization of elementary acts for achieving a target (often a modification of the environment), taking into account local conditions. Further, cognitive abilities help children to transform problems, which means that they allow the creation of a mental model that identifies actions to perform to solve a problem. Therefore, from a cognitive point of view, the use of tools allows to transform the difficulty of a task since the manipulation of physical objects eases the matching between what the subject thinks and what is the obtained result. In Table 1 we described the learning potential of the Book of Colours. This lists the cognitive aspects that are supported by the levels of interaction that the subject can experience with the system. Table 1. Cognitive aspects and levels of interaction of the Book of Colours Cognitive Aspects Active role of the learning subject Learning Efficacy
Levels of Interaction • • • • • • •
Constructive Learning
• • •
Interactivity of the system Ease of use of the tool 3D representation of the content Possibility to combine colours at different levels Representation of different properties of colours Learning by doing. Children study properties of colours, whose principles are represented through virtual objects Verification of knowledge acquired, through the manipulation of the pages of the book Exploration of colours on the basis of their combinations Sustain to action. Acquisition of knowledge is the result of an active participation of children It stimulates children in experimenting procedures that are alternatives to traditional learning methods
Learning Using Augmented Reality Technology
199
The use of learning tools engages positively students if these tools embody adequate external forms of representation of the content, which allow to verify the gain of knowledge of subjects. The representation system of the Book of Colours is the chameleon. The function of the chameleon, which is cognitively transparent to learners, is to stimulate children to interact with the system motivating them in discovering the properties of the colours. The relationship between cognitive and interactive aspects of the system shows how the combination of manipulation, interaction and visualization can incite children to reach deeper forms of reflection and exploration. Children are encouraged to build and to manipulate objects for posing hypothesis and to experiment their ideas. The Book of Colours implements a learning method that stimulates children to act. Children’s actions are the only means to interact with the book and to explore the properties of the colours. The active engagement of children in the learning process has significant benefits at a cognitive level. The engagement increases attention that in turn enhances children’s concentration and reflection, which affect positively the learning process [21]. Engagement comes together with the exploration of new concepts, which is essential for learning since it diminishes lack of confidence in case of new circumstances [22]. With the Book of Colours children can observe and explore properties of colours in an environment that reacts to their actions through the physical transformation of the chameleon. The authors are interested in understanding how children perceive the transformations of the virtual character, and how they formulate a synthesis out of the stimulus provided by the augmented book. The following section describes the involvement of some children in testing the book and their reactions to the new learning tool.
4 Testing the Book of Colours with Children The flexible and tangible interface of the book allows recreating didactic conditions that support children in learning various typologies of concepts and at different levels. The authors conducted some initial tests on primary school children (aged 9) and recorded their reactions while using the Book of Colours in didactic circumstances. Table 2 shows the steps of a session, and reports on children’s behavior during each phase. The test session permits to draw some interesting observations a) on the characteristic educational aspects of the system, b) on its effect on the learning process. In particular, the authors found that the interaction with the book supports a trial-error learning style. Children are stimulated to observe the results of each page of the book and to verify their answers. Children understand the importance of checking the content of each page giving relevance to the experimental aspects of the book. They are able to verify autonomously the results of the activities thanks to the visual feedback provided by the virtual chameleon. The chameleon is informative but not intrusive. Children analyze with great attention the properties of those colours showed by the skin of the chameleon and they show interest in finalizing the activities proposed during the session. The system stimulates them to act and this becomes the central aspect of the cognitive activity since it encourages reflection and it accustoms children to an
200
G. Ucelli et al.
active learning. Finally, after an initial period of training, children acquire familiarity with the system and they are able to individuate the properties of colours. Table 2. Observation of children’s behaviour interacting with the book during a test session STEP 1 Presentation of the book
STEP 2 Presentation of the technology
STEP 3 Adjustment to the novelty
STEP 4 Explaining the theory of colours
STEP 5 Introducing the chameleon
STEP 6 Initial Experimentation
STEP 7 Which is the right colour?
STEP 8 Looking for the right answer
STEP 9 An interactive quiz game to play in group
First impact with the Book of Colours. This is presented to children as a special book with a familiar look. This sense of familiarity eases the interaction and provides children with a better sense of embodiment. The basic principles of AR are presented to children to justify the use of technology for “augmenting” what they initially think is an ordinary book. We observed that the choice of using an HMD together with a monitor that shows the same scene was particularly convenient for demonstrating how to use the book. Children are slightly intimidated about using an HMD. This causes the need for an initial encouragement by adults. Once accustomed tough, children fully enjoyed the novelty. We observed that the child that was most hesitating at the first trial, she asked spontaneously to wear the equipment once again to keep on playing with the book. We explain the basic principles of colours. Why people perceive different colours, properties of white and black, subdivision of colours in primary, secondary and tertiary, which are the primary and secondary colours, what property they have, which secondary is obtained from the primary, which colour is complementary to another. We introduce the virtual chameleon in the game. We explain the special talent of the animal, which is able to camouflage itself. We show how the chameleon is related to the physical book and how it transforms its appearance according to the actions performed on the book (going though the pages of the book and using the movable parts). Children are free to experiment with the book and to interact with the chameleon. We observed that the popping-up of a 3D virtual character was considered by children normal and entertaining. They enjoyed the cause-effect relation between the interaction with the book and the change of appearance of the chameleon. Children notice that markers depict parts of the chameleon providing clues of animal in the physical book. Children try to guess which secondary colour results from the combination of the primary colours that are showed in each page of the book. In the picture on the left Blue+Red=Purple and the chameleon will provide the right answer. Children validate their answers through the chameleon. At the same time showing only the chameleon (using only the markers) children are asked to answer to the opposite question: which are the primary colours that combined make the colour of the chameleon? Children now interact at ease with the book. The observed initial apprehension for wearing an HMD has vanished, and the book has become an interactive game to play in group. The use of a display, apart from the HMD, assures the concurrent participation of all the children and the teacher to the game.
Learning Using Augmented Reality Technology
201
5 Concluding Remarks The interaction means offered by AR in the Book of Colours support what the authors call a “practical constructivism”, which results in solving specific problems while children learn new concepts supported by scaffolding [3]. What has been said so far shows how positively children’s active engagement affects learning effectiveness. The Book of Colours adds a new dimension to this, allowing children to carry out actions that make use of concepts, and to manipulate not only simple objects but structures having specific functions. The test session showed the need to improve the ergonomics of the HMD, which should be lighter and easily resizable to be fitted in children’s heads. This issue is out of scope for this research work but will be extensively addressed by the authors within another research project, financed by the European Commission [23]. The system, however, does not rely only on HMDs for visualizing the chameleon but also on an ordinary display, which proved to be very useful during the explanation of AR technology and for easing group participation in the game. The choice of two output mode (i.e. one “high tech” and the other “low tech”) was made for easing the economical and usability difficulties in acquiring and deploying high-end technology in schools. The positive and enthusiastic reaction of children encourages the authors to test the Book of Colours at a bigger scale also involving schools, and to conduct comparative studies to understand to which extent the learning method proposed by the book affects gain and retention of knowledge. The experience of the Book of Colours proves that AR can be effectively used for providing an innovative and attractive learning experience for children. The approach followed, according to the authors, can be considered of prime value for nurturing children's creativity and imagination.
Acknowledgments The authors would like to acknowledge the work of Valentina Giovannini who designed the interaction metaphor and the graphics of the book in collaboration with the staff of the Faculty of Design and Art of the Free University of Bolzano and the technical support of Artigianelli, Trento. Special thanks go to Stefan Noll and his kids.
References 1. Papert, S.: Mindstorms: Children, Computers, and Powerful Ideas. Basic Books, New York. (1980). 2. Bruner, J.: Toward a theory of instruction. Harvard University Press, Harvard (1966). 3. Vygotskij, S.L.: Thought and Language. M.I.T. Press, Cambridge (1962). 4. Cacciamani, S.: Psicologia per l’insegnamento. Carocci, Roma (2002). 5. von Glasersfeld, E.: Cognition, Construction of Knowledge, and Teaching. Synthese, 80(1), (1989) 121-140. 6. Holt, J. C.: How Children Learn. Perseus Publishing, New York (1995). 7. Underkoffler, J., and Ishii, H.: Illuminating light: An optical design tool with a luminoustangible interface. Proceedings of Computer-Human Interaction CHI’98, (1998) 542-549.
202
G. Ucelli et al.
8. Price, S., and Rogers, Y.: Let’s get physical: The learning benefits of interacting in digitally augmented physical spaces. Computers & Education, 43, (2004) 137–151. 9. Dourish, P.: Where the Action is: The Foundations of Embodied Interaction. MIT Press, Cambridge: (2001). 10. Price, S., Rogers, Y., Scaife, M., Stanton, D., and Neale, H.: Using ‘tangibles’ to promote novel forms of playful learning. Interacting with Computers, 15, (2003) 169–185. 11. Resnick, M., Martin, F., Berg, R., Borovoy, R., Colella, V., Kramer, K., and Silverman, B.:. Digital Manipulatives: New Toys to Think With. Proceedings of Computer-Human Interaction CHI ‘98, (1998) 281-287. 12. Resnick, M., Berg, R., & Eisenberg, M.: Beyond Black Boxes: Bringing Transparency and Aesthetics Back to Scientific Investigation. Journal of the Learning Sciences, 9(1), (2000) 7-30. 13. Wisneski, C., Ishii, H., Dahley, A., Gorbet, M., Brave, S., Ullmer, B., and Yarin, P.: Ambient Displays: Turning Architectural Space into an Interface between People and Digital Information. Lecture Notes in Computer Science, Vol. 1370, (1998) 22-32. 14. Johnson, A., Roussos, M., Leigh, J., Barnes, C., Vasilakis, C., and Moher, T.: The NICE Project: Learning Together in a Virtual World. Proceedings of the Virtual Reality Annual International Symposium, (1998) 176-183. 15. Weevers, I., Sluis, W., van Schijndel, C., Fitrianie, S., Kolos-Mazuryk, L., and Martens, J.:. Read-It: A Multi-modal Tangible Interface for Children Who Learn to Read. Proceedings of the 3rd International Conference on Entertainment Computing, (2004) 226-234. 16. Gabrielli, S., Harris, E., Rogers, Y., Scaife, M., and Smith, H.: How many ways can you mix colour? Young children's explorations of mixed reality environments. Proceedings of the Conference for Content Integrated Research in Creative User Systems. (2001). Available: http://www.informatics.sussex.ac.uk/users/hilarys/ [Checked April 2005]. 17. Billinghurst, M., Kato, & H., Poupyrev, I.: The MagicBook: A Transitional AR Interface. Computers and Graphics, 25(5), (2001) 745-753. 18. Ishii, H., & Ullmer, B.: Tangible Bits: Towards Seamless Interfaces between people, Bits and Atoms. Proceedings of Computer-Human Interaction CHI’97, (1997) 234-241. 19. Billinghurst, M.: Augmented Reality in Education. (2003). Available: http://www. newhorizons.org/strategies/technology/billinghurst.htm [Checked April 2005]. 20. ARToolkit. Available: http://www.hitl.washington.edu/artoolkit/ [Checked April 2005]. 21. Stoney, S., & Oliver, R.: Can higher order thinking and cognitive engagement be enhanced with multimedia? Interactive Multimedia Electronic Journal of Computer-Enhanced Learning. (1999). Available: http://imej.wfu.edu/articles/1999/2/07/index.asp [Checked March 2005] 22. Diamond, J.: Play and Learning. ASTC Resource Center (1996). Available: http://www. astc.org/resource/ learning/diamond.htm [Checked March 2005]. 23. IMPROVE web site Available: http://www.improve-eu.info/ [Checked April 2005].
Presenting in Virtual Worlds: Towards an Architecture for a 3D Presenter Explaining 2D-Presented Information Herwin van Welbergen, Anton Nijholt, Dennis Reidsma, and Job Zwiers Human Media Interaction Group, University of Twente Enschede, The Netherlands
[email protected] Abstract. Entertainment, education and training are changing because of multi-party interaction technology. In the past we have seen the introduction of embodied agents and robots that take the role of a museum guide, a news presenter, a teacher, a receptionist, or someone who is trying to sell you insurances, houses or tickets. In all these cases the embodied agent needs to explain and describe. In this paper we contribute the design of a 3D virtual presenter that uses different output channels to present and explain. Speech and animation (posture, pointing and involuntary movements) are among these channels. The behavior is scripted and synchronized with the display of a 2D presentation with associated text and regions that can be pointed at (sheets, drawings, and paintings). In this paper the emphasis is on the interaction between 3D presenter and the 2D presentation.
1
Introduction
A lot of meeting and lecture room technology has been developed in previous years. This technology allows real-time support to physically present lecturers, audiences and meeting participants, on-line remote participation of meetings and lectures, and off-line access to lectures and meetings [1,2,3]. Whether it is for participants that are physically present (e.g. while being in the lecture room and looking back on part of the presentation or previous related presentations), for remote audience members or for off-line participants, multi-media presentation of captured information needs a lot of attention. In previous research we looked at the possibilities to include in these multimedia presentations a regeneration of meeting events and interactions in virtual reality. We developed technology to have a translation from captured meeting activities to a virtual reality regeneration of these activities that allows adding and manipulation of information. We looked at translating meeting participant activities [4], and at translating presenter activities [5]. While in the papers just mentioned our starting point was the human presenter or meeting participant, in the research reported here our starting point is a semi-autonomous, virtual presenter (Fig. 1) that is designed to perform in a virtual reality environment. The audience of the presenter will consist of humans, humans represented by embodied agents in the virtual world, autonomous M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 203–212, 2005. c Springer-Verlag Berlin Heidelberg 2005
204
H. van Welbergen et al.
Fig. 1. The virtual presenter
agents that decided to visit the virtual lecture room or have been assigned roles in this room, or any combination of these humans and agents. In this paper we confine ourselves to models and associated algorithms that steer the presentation animations of a virtual presenter. The presentations are generated from a script describing the synchronization of speech, gestures and movements. The script has also a channel devoted to slides and slide changes; they are assumed to be an essential part of the presentation. Instead of slides, this channel may be used for the presentation of other material on a screen or wall. 1.1
Organization of This Paper
In section 2 of this paper we highlight previous research on presentation agents that can interact with visual aids. Section 3 of this paper introduces our architecture of a virtual presenter. In the sections 4, 5, 6 and 7 the separate parts of this design are further discussed. Section 8 concludes this paper and discusses possible further work on the virtual presenter.
2
Presentations by Embodied Agents
A great amount of projects about presenting and virtual agents can be found. In this section we highlight a few projects projects featuring human-like presenters that use visual aids. Prendinger, Descamps and Ishizuka worked on specifying presentations related to web pages, which are executed by MS-agents, a robot, or a 3D presenter. Their main focus has been the development of a multi-modal presentation language (MPML) with which non-expert (average) users can build web-based interactive presentations [6]. MPML allows one to specify behaviour in several modalities including gestures, speech and emotions.
Presenting in Virtual Worlds: Towards an Architecture
205
The work of Noma, Zhao and Badler aims at simulating a professional presenter such as a weather reporter on TV [7]. The presenter can interact with a visual aid (usually a 2D screen). The animation model is rather simple. It implements two posture shifts, namely those needed to look at the screen and then back in the camera. The arm movement is determined by pointing actions specified in the animation script and by the affirmation level (neutral, warm, enthusiastic). For the hand movement, canned animations for grasping, indicating, pointing and reaching were used. Andr´e, Rist and M¨ uller [8] describe a web agent drawn in a cartoon style to present information on web pages. Most of their work focuses on planning such a presentation script. The agent is capable of performing pointing acts with different shapes (e.g. punctual pointing, underscoring and encircling). It can express emotions such as anger or tiredness. “Idle animations” are performed to span pauses. The character is displayed in 2D using completely predefined animations. Pointing acts are displayed by drawing a pointing stick from the hand to the pointing target.
3
An Architecture for a Virtual Presenter
Building a virtual presenter brings together many different techniques, including facial animation, speech, emotion/style generation and body animation. The main challenge is to integrate those different elements in a single virtual human. The two major concerns for this integration are consistency and timing [9]. Consistency. When an agents internal state (e.g. goals, plans and emotions) as well as the various channels of outward behavior (like speech, body movement and facial animation) are in conflict, inconsistency arises. The agent might then look clumsy or awkward, or, even worse, it could appear confused, conflicted, emotionally detached, repetitious, or simply fake. Since in the current version of the virtual presenter its behavior is derived from the annotated script of a real presentation, consistency conflicts arise mostly between channels that are implemented and those that are not. When the presenter gets extended to dynamically generate its behaviour, consistency will become a more important issue. Timing. The timing is currently a more crucial concern. The different output channels of the agent should be properly synchronized. When an agent can express itself through many different channels the question arises what should be the leading modality in determining the timing of behavior. For example, BEAT [10], a toolkit used to generate non-verbal animation from typed text, schedules body movements as conforming to the time line generated by the text-to-speech system. Essentially, behavior is a slave to the timing constraints of the speech synthesis tool. In contrast, EMOTE takes a previously generated gesture and shortens it or draws it out for emotional effect. Here, behavior is a slave of the
206
H. van Welbergen et al.
constraints of emotional dynamics. Other systems focus on making a character highly reactive and embedded in the synthetic environment. In such a system, behavior is a slave to the environmental dynamics. To combine these types of behavior you need at least two things. In the first place, the architecture should not fix beforehand which modality is the “leading modality” wrt the synchronization. In the second place, different components must be able to share information. If BEAT has information about the timing constraints generated by EMOTE, it could do a better job scheduling the behavior. Another option is to design an animation system that is flexible enough to handle all constraints at once. Norman Badler suggests a pipeline architecture that consists of ‘fat’ pipes with weak up-links. Modules would send down considerably more information (and possibly multiple options) and could pull downstream modules for relevant information (for example, how long it would take to point to a certain target, or how long it would take to speak a word). 3.1
The Architecture
The architecture of the virtual presenter (Fig. 2) is inspired by Norman Badler’s pipeline and the work described in [9]. Expressions on separate channels are specified in the presentation script. The planners for different channels determine how those expressions are executed. This can be done using information from another module (for example, the text-to-speech engine can be asked how long it takes to speak out a certain sentence, or the sheet planner can be asked which sheet is visible at what time) or from human behavior models. The planner could even decide not to execute a certain expression, because it is physically impossible to do so, or because it would not fit the style of the presenter. We choose to implement a selected set of dimensions of the behavior of a presenter in a theoretically sound way. To be able to insert new behavior later on, the presenter is designed to be very extensible. The rest of this paper discusses the different implemented aspects in more detail.
4
Presentation Script
As can be seen in Figure 2, the script is the starting point for (re)visualization of presentations. A script can be created from annotation of behavior observed in a real presentation, or generated from an intention to convey certain information. The multi-modal channels used by the presenter are scripted at different abstraction levels. Gestures are specified in an abstract manner, mentioning only their ’type’ or communicative intent (deictic reference, stressing a word in speech, indicating a metaphor, etc.), leaving the exact visualization (e.g. which body-part, hand shape or movement path to use) to the planners. Speech is annotated at a word level. Resting poses and pose shifts are annotated at a very low level, the joint rotations in the skeleton of the presenter are specified for every pose. Sheet changes are specified whenever they should occur.
Presenting in Virtual Worlds: Towards an Architecture
207
Fig. 2. Full system architecture of the virtual presenter
4.1
MultiModalSync
For the synchronization and timing of the presentation in its different channels we developed the MultiModalSync language. The synchronization is realized by setting synchronization points in one modality and using these synchronization points in another modality. For example, a synchronization point can be set before a word in the verbal modality. This point can then be used in the pointing modality to define a pointing action that co-occurs with the spoken word. Synchronization points can be set and used on all modalities so that the “leading” modality can be changed over time. [11] describes the constraints and synchronization definitions of the MultiModalSync language in greater detail and explains why it was necessary to develop a new script language.
5
Presentation Planning
The animation planner (Fig. 3) is responsible for the planning and playback of body animation. It makes use of movement models derived from neurophysics and behaviorial science to perform this task. Details on the exact use of these models can be found below. Currently, the animation planner is capable of playing deictic gestures, pose shifts and speech (mouth movement) specified in the script. Static speaker characteristics influence how this behavior is executed. The architecture can easily be extended to execute other gesture types. The verbal planner and the sheet planner regulate the Text To Speech generation and the sheet changes, respectively.
208
H. van Welbergen et al.
Fig. 3. The animation planner
5.1
Speech Planning
Loquendo’s Text-To-Speech engine is used to generate speech, lipsync and speech timing information from the verbal text. A very simple form of lip synchronization is used by the Animation Planner: the rotation of the jaw is proportional with the volume of the speech, averaged over a short time period. 5.2
Involuntary Movement
Even while standing still, our body still moves in very subtle ways: we try to maintain balance, our eyes blink and our chest moves when we breath in and out. An avatar that does not perform such subtle motion will look stiff and static. Our presenter uses an involuntary movement method described in [12]. Involuntary movement is simulated by creating noise on some of the joints in the skeleton of the avatar. This method is chosen to avoid the repetitiveness of predefined scripted idle animation and because the presenters’ model is not detailed enough to use realistic involuntary movement models. The choice of which joints to move is made ad-hoc. For example, the two acromioclavicular joints (the joints between the neck and the shoulder) can be moved to simulate small shoulder movement that occurs when breathing. Small rotations of the vl1 joint are used to simulate subtle swaying of the upper body. 5.3
Posture
A pose is defined as the resting position of the body. Poses are used as start and end positions for the limbs within a gesture unit. In “real” monologues, posture shifts frequently occur at discourse segments [13]. In our presenter system, each pose is specified separately in a library containing the joint positions. In the current scripts, references to these poses are included based on the human presenter’s pose in the real presentation (Fig. 4).
Presenting in Virtual Worlds: Towards an Architecture
209
Fig. 4. Annotating and simulating poses. From left to right: the pose in the meeting room, a manually created representation in the Milkshape modelling tool and the virtual presenter showing that pose.
5.4
Pointing
During presentation, a presenter can refer to areas of interest on the sheet, by using a gesture with a pointing component. Our pointing model takes several aspects of pointing movement in consideration, so that the pointing movement can be generated given only the intention to point and a pointing target. Like Noma, Zhao and Badlers presenter [7], the presenter points to the right using its right hand and to the left using his left hand, to keep an open posture. When this preferred hand is occupied the presenter will point using gaze. Timing. Fitts’ law predicts the time required to move from a certain start point to a target area. It is used to model rapid, aimed pointing actions. Fitt’s law could thus give a minimum value for the duration of the preparation phase of a pointing action in a presentation. The virtual presenter uses a 2D derivation of D Fitt’s law described in [14]: T = a + b · log2 ( min(W,H) + 1), where T is the time necessary to perform the pointing action, a and b are empirically determined constants, D is the distance to the object to point to, W is the width and H the height of the object. Movement in the Retraction Phase. Gestures are executed in three phases. In the (optional) preparation phase the limb moves away from the resting position to a position in gesture space where the stroke begins. In the (obligatory) stroke phase we find the peak of effort in a gesture. In this phase, the meaning of the gesture is expressed. In the (optional) retraction phase the hand returns to a rest position. Preparation (or retraction) will only occur if a gesture is at the beginning (or end) of a gesture unit. According to [15], gesture movement is symmetric. We conducted a small experiment to find out if this is also true for more precise pointing actions. This was done by creating and looking at videos of pointing gestures, the same
210
H. van Welbergen et al.
Fig. 5. Screen capture of the preparation (upper half) and retraction phase (lower half, backward) of a movie with a pointing action
pointing gestures played backward and a single video with both the forward and backward played gestures. Figure 5 shows screen captures of such a video. The following was found out: – Pointing gestures that form a complete gesture unit by themselves are rare. – Those gestures that do form a gesture unit by themselves have symmetric looking preparation and retraction phases. – As Kendon noted, it is hard to tell if such a gesture is played backward or forward Based on these findings, the retraction phase is defined by moving the arm back to the resting position in the same way it would be moved to the position of the stroke from the resting position (but backward). Pointing Velocity. The velocity profile of a pointing movement is bell shaped [16]. This bell can be asymmetric. The relative position-time diagram is sigmoidshaped. We use the sigmoid f (t) = 0.5(1+tanh(a(tp −0.5))) to define the relative position of the wrist. In this function t represents the relative movement time (t = 0 is the start time of the pointing movement, at t = 1 the wrist reached the desired position). f (t) describes the relative distance from the start position: f (0) = 0 is the start position, f (1) = 1 represents the end position. The steepness of this sigmoid can be adjusted using a. p can be used to set the length of the accelatory and decelatory phases. Pointing with Gaze. Gaze behaviour during pointing movements is implemented on the basis of Donders Law [17], which defines both the necessary movements and end orientations for the eyes and for the head, given the fact that the presenter will look at its pointing target. Shoulder and Elbow Rotation. If the wrist position is determined by the location and size of the pointing target, the rotations of the elbow and shoulder joints can calculated analytically using the inverse kinematics strategy described in [18]. The elbow though is still free to swivel on an circular arc, whose normal is parallel to the axis from the shoulder to the wrist. To create reasonably good looking movements, the presenter always rotates the elbow downward.
Presenting in Virtual Worlds: Towards an Architecture
5.5
211
Sheet Planning
To display the sheets, the virtual presenter uses a virtual projector screen. On these sheets, areas of interest are defined, at which the presenter can point. The sheets for a presentation are described in an XML presentation sheets library. The sheet planner is responsible for the display and planning of sheet changes. It can provide other planners with planning information (e.g. what sheet is visible at what time).
6
Digital Entertainment in a Virtual Museum: Moving to a Different Domain
In order to demonstrate the broader applicability of the technology that was developed for the Virtual Presenter we are currently investigating a different domain, namely that of a Virtual Museum Guide. A corpus of annotated paintings such as the Rijksmuseum database used in [19] shares many characteristics with the presentation sheets. The information about the painting covers general aspects as well as remarks about specific subareas of the painting (e.g. relative composition, details in a corner of the painting, etc). The multimedia presentations they generated automatically from this content could easily be made interactive by using the virtual presenter as a museum guide who talks about the paintings while pointing out interesting details. In those cases where the relations between the text and areas in the paintings is only implicitly encoded in the text, techniques could be developed for automatic extraction of those relations from the text.
7
Conclusions and Future Research
Using the architecture described in here, we were able to create a speaking, involuntary moving, posture shifting and pointing presenter. The behavior on these different output channels is synchronized using a script. The designed architecture has already been adapted in two student projects with respectively new gesture modalities and possibilities to allow interruption of the presenter by an audience. Further work could broaden the presenters abilities to express itself. This can be done by adding additional gesture types (like beats, iconics or metaphoric gestures). It is also possible to raise the virtual presenting process to a higher abstraction level. Currently, the script determines what part of the presentation is expressed in speech and what part is expressed by gestures. The next logical abstraction step would be to implement a process that determines what to say and what gestures to make, based on the content of what the presenters wants to tell. Such a selection process can be guided by the presenters style and emotional state.
212
H. van Welbergen et al.
References 1. Rui, Y., Gupta, A., Grudin, J.: Videography for telepresentations. CHI Letters 5(1) (2003) 457–464 2. Rogina, I., Schaaf, T.: Lecture and presentation tracking in an intelligent meeting room. In: Proc. IEEE Intern. Conf. on Multimodal Interfaces. (2002) 14–16 3. Waibel, A., Steusloff, H., Stiefelhagen, R., the CHIL Project Consortium.: CHIL - computers in the human interaction loop. In: 5th Intern. Workshop on Image Analysis for Multimedia Interactive Services, Lisbon, Portugal (2004) 4. Reidsma, D., op den Akker, H., Rienks, R., Poppe, R., Nijholt, A., Heylen, D., Zwiers, J.: Virtual meeting rooms: From observation to simulation. In: Proc. Social Intelligence Design, Stanford University, CA (2005) 5. Poppe, R., Heylen, D., Nijholt, A., Poel, M.: Towards real-time body pose estimation for presenters in meeting environments. In: Proc. of the Intern. Conf. in Central Europe on Computer Graphics, Visualization and Computer Vision. (2005) 41–44 6. Prendinger, H., Descamps, S., Ishizuka, M.: MPML: a markup language for controlling the behavior of life-like characters. J. Vis. Lang. Comput. 15(2) (2004) 7. Noma, T., Zhao, L., Badler, N.I.: Design of a virtual human presenter. IEEE Computer Graphics and Applications 20(4) (2000) 79–85 8. Andr´e, E., Rist, T., M¨ uller, J.: Webpersona: A life-like presentation agent for the world-wide web. Knowledge-Based Systems 11(1) (1998) 25–36 9. Gratch, J., Rickel, J., Andr´e, E., Cassell, J., Petajan, E., Badler, N.I.: Creating Interactive Virtual Humans: Some Assembly Required. IEEE Intelligent Systems 17(4) (2002) 54–63 10. Cassell, J., Vilhj´ almsson, H.H., Bickmore, T.: BEAT: the Behavior Expression Animation Toolkit. In: SIGGRAPH ’01: Proceedings of the 28th annual conf. on Computer graphics and interactive techniques. (2001) 477–486 11. Nijholt, A., van Welbergen, H., Zwiers, J.: Introducing an Embodied Virtual Presenter Agent in a Virtual Meeting Room. In: Proc. of the IASTED Intern. Conf. on Artificial Intelligence and Applications. (2005) 579–584 12. Perlin, K.: Real time responsive animation with personality. IEEE Transactions on Visualization and Computer Graphics 1(1) (1995) 5–15 13. Cassell, J., Nakano, Y., Bickmore, T., Sidner, C., Rich, C.: Annotating and Generating Posture from Discourse Structure in Embodied Conversational Agents. In: Workshop on Representing, Annotating, and Evaluating Non-Verbal and Verbal Communicative Acts to Achieve Contextual Embodied Agents, Autonomous Agents 2001 Conference, Montreal, Quebec (2001) 14. MacKenzie, I.S., Buxton, W.: Extending Fitts’ law to two-dimensional tasks. In: Proc. of the SIGCHI conf. on human factors in computing systems. (1992) 219–226 15. Kendon, A.: An Agenda for Gesture Studies. The Semiotic Review of Books 7(3) (1996) 16. Zhang, X., Chaffin, D.: The effects of speed variation on joint kinematics during multi-segment reaching movements. Human Movement Science (18) (1999) 17. Donders, F.C.: Beitr¨ age zur Lehre von den Bewegungen des menschlichen Auges. Holl¨ andische Betr¨ age Anat. Physiol. Wiss. (1) (1848) 104–145 18. Tolani, D., Goswami, A., Badler, N.I.: Real-time inverse kinematics techniques for anthropomorphic limbs. Graph. Models Image Process. 62(5) (2000) 353–388 19. Smeulders, A., Hardman, L., Schreiber, G., Geusebroek, J.: An integrated multimedia approach to cultural heritage e-documents. In: Proc. 4th Intl. Workshop on Multimedia Information Retrieval. (2002)
Entertainment Personalization Mechanism Through Cross-Domain User Modeling Shlomo Berkovsky1, Tsvi Kuflik2, and Francesco Ricci3 1
University of Haifa, Computer Science Department
[email protected] 2 University of Haifa, Management Information Systems Department
[email protected] 3 ITC-irst, Trento
[email protected] Abstract. The growth of available entertainment information services, such as movies and CD listings, or travels and recreational activities, raises a need for personalization techniques for filtering and adapting contents to customer's interest and needs. Personalization technologies rely on users data, represented as User Models (UMs). UMs built by specific services are usually not transferable due to commercial competition and models' representation heterogeneity. This paper focuses on the second obstacle and discusses architecture for mediating UMs across different domains of entertainment. The mediation facilitates improving the accuracy of the UMs and upgrading the provided personalization.
1 Introduction Nowadays, digital entertainment content becomes more and more available to consumers via new information services and devices. As a result, consumers increasingly appraise the services enabling them to efficiently navigate through large volumes of available entertainment content and access the most valuable items. This enriches personal entertainment experience and increases consumers' retention, which in turn, boosts the revenue of service providers and content owners. However, this wealth brings with it information overload and rises the need for personalization services. Providing personalized information retrieval, filtering and recommendation services to the consumers requires users' data, representing their preferences, needs and wishes, to be accessible by service providers [12]. This data is referred in the literature as the user model (UM) [10]. Typically, UMs stored by one service are proprietary and tailored to the specific content offered by the service. Since the accuracy (or simply the usefulness) of the provided personalization depends largely on the quality and richness of the input UMs, different services would benefit from enriching their UMs through importing and integrating scattered partial UMs built by other services. We refer to this functionality as Cross-Domain User Modeling (CDUM). CDUM raises a number of open issues. The first deals with the commercial nature of the digital entertainment realm. Due to competition, personalization services usually neither cooperate, nor share their partial UMs. The second deals with customers' M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 215 – 219, 2005. © Springer-Verlag Berlin Heidelberg 2005
216
S. Berkovsky, T. Kuflik, and F. Ricci
privacy. Partial UMs built by service providers may contain private, sensitive information about the customers, which can be disclosed for the requesting service only. Nowadays, there is growing awareness and concern about disclosure and misuse of such information [4]. The third deals with the heterogeneity in the structure, and the incompleteness of the UMs contents. The lack of standard representation, and the specific requirements posed by different personalization technologies, result in personalization services building their models in different, ad-hoc forms. As a result, large numbers of heterogeneously represented and possibly overlapping user data are scattered among various service providers. Thus, there is an emergent need for a mechanism capable of integrating heterogeneous partial UMs for the purpose of providing better personalized services to the customers. This work discusses the ways of applying CDUM mediation mechanism, initially proposed in [2], in the entertainment domain. This mechanism provides a standardized interface for user modeling data exchange, facilitates translation and integration of partial UMs built by different services, and ad-hoc generation of the UM required by the target personalization service. This bootstraps (if no UM exists), or enriches the UM of the target service and leverages the accuracy of the service personalization.
2 Digital Entertainment and Cross-Domain Personalization According to Wikipedia (www.wikipedia.org – Free Web Encyclopedia), entertainment is "an amusement or diversion intended to hold the attention of an audience or participants". Although other sources provide slightly different (and even ambiguous) definitions, they all coincide on the main sub-domains of entertainment: music, movies, television, radio, tourism and recreation activities, books and literature, humor and others. Clearly, most of them require personalization services to assist users to cope with the vast amount of available information. Moreover, users within these domains, when looking for information, would like to get an immediate and accurate response, or a useful suggestion. They are looking for "a restaurant nearby where we would like to have dinner" or for "a movie in a nearby theatre that we would enjoy" or "an interesting book to read next" and so on. For any of the above, users require immediate and accurate response, which, in turn, requires as rich as possible knowledge about user's interests and needs, and the current context (e.g., time and location, service availability and more). This information may not be readily available to the service providers. Information Filtering [8] and Recommender Systems [6] are two popular examples of personalized information access services. Although they exploit a wide range of different techniques, both of them attempt at finding information that may interest a user. Recommender Systems are systems that limit the amount of information reaching the user by selecting and displaying the relevant information only. Information Filtering systems aim at achieving the same functionality through filtering out the irrelevant information from user's incoming information stream. Growing number of researches tackle the issue of personalization in entertainment. For example, [6] exploits Information Filtering agents to identify movies that a user would find worthwhile. In [14], the authors discuss the Personalized TV Listings system providing a personalized TV guides matching the preferences of individual
Entertainment Personalization Mechanism Through Cross-Domain User Modeling
217
users. In [5], a variety of recommendation techniques is used to build TV programs schedule answering user's needs. In [1], case-based reasoning technique is exploited to build personal music compilations. NutKing [13] also uses case-based reasoning technique provides personal recommendations for recreation events and tourism activities. Finally, [7] describes online system capable of recommending jokes basing on a small number of jokes' ratings provided by a user. Typically, service providers independently build up proprietary UMs. Such UMs are not transferable to other services due to commercial competition, privacy restrictions and models' representation heterogeneity. For example, UMs are stored in [6] as ratings of the users movies, whereas in [1] they compilations of music tracks. However, importing partial UMs built by other services, or by other components of the current service, and integrating them with a local UM may enhance the resulting UM, yielding better personalization. We refer to the importing of partial UMs as CDUM. Generation of a centralized UM as a composition partial UMs stored by different personalization services is proposed in [9]. To accomplish this task, each service maintains a mechanism capable of extracting the needed user modeling data and updating the general model. A similar approach is discussed in [11] proposing the use of Unified User Context Model (UUCM) for improving the UMs built by the individual services. To provide personalization, each service extracts the required data from the UUCM, delivers the service, and updates the UUCM.
3 Decentralized User Modeling Data Mediation in Entertainment The above studies focus on a centralized user modeling mechanism. A decentralized approach, basing on a mediator that generates an ad-hoc UM, according to the requirements of a target service, was proposed in [2]. The mediator is responsible for: (1) Determining UMs representation of the target service, (2) Identifying the services that may provide the needed user modeling data, (3) Integrating the partial UMs and generating UM for the target service. Figure 1 illustrates the above process: 1. A target service identifies a user requiring personalization and queries the mediator for the UM related to the application domain of the service. 2. The mediator identifies the personalization domain, and determines the representation of the UM in the target service. 3. The mediator extracts from the knowledge base (KB) a set of remote services that can provide the partial domain-related UMs. 4. The mediator queries the remote services for the partial UMs of the user. 5. Services, storing the needed information, send the local partial UMs. 6. The mediator integrates the partial UMs (using the KB) and assembles ad-hoc domain-related UM. 7. The generated domain-related model is sent to the target service facilitating provision of a personalized service. In order to illustrate the functionalities of the above mediator in the domain of digital entertainment, consider the following example. A big company developed a network of Web-sites supporting personalized entertainment services provision. The network contains music, movies, TV, books and humor personalization Web-sites.
218
S. Berkovsky, T. Kuflik, and F. Ricci
Fig. 1. Functional Flow of the Mediation Process
Consider a user requiring a personalized recommendation for a movie. To obtain an accurate UM, the movies Web-site sends a list of available movies to the mediator, and queries it for the movies' UM of the user (step 1). The mediator analyzes UMs representation in the domain of movies (for the sake of simplicity we assume that the UM is represented as a list of user's favorite genres and their respective weights), and identifies the remote Web-sites that can provide valuable partial UMs (steps 2 and 3). Clearly, only the Web-sites from domains, which are closely related to movies, can provide valuable partial UMs. Let us assume that user's favorite movie genres can be inferred from his partial UMs in the domains of TV, books and music, whereas humor-related partial UM can not enrich movies-related partial UM. The mediator communicates TV, books and music Web-sites and queries them for their local UMs of the user (step 4). As in our example all the Web-sites are owned by one company, it is reasonable to assume that the issues of commercial competition and privacy are alleviated. Thus, Web-sites storing the user's partial UMs, respond to the query and send their local UMs (step 5). The mediator integrates the acquired partial UMs into a single UM (step 6). As the UM is constructed, it is transferred to the movies Web-site, which can provide more accurate personalization.
4 Preliminary Results and Future Research Preliminary experiments, demonstrating the possibility of cross-domain integration of partial UMs in movies recommendations, were conducted in [3]. There, we partitioned movie ratings UMs to different databases by splitting them according to the movie genres (simulating different, but related domains). The accuracy of the recommendations generated by a single genre UMs was compared with the accuracy of the recommendations generated by the combined UMs (the complete database). The results showed that the recommendations' accuracy was similar, concluding that crossgenre recommendation (as a case of cross-domain personalization) is feasible. Within the PEACH project [15], we are working towards the provision of personalized guidance and support during museum visits. The visitors are equipped with personal hand-held devices that act as a personal guide through displaying to the user personalized presentations. To bootstrap the initial UM, we import user data from NutKing tourism planning system [13], storing UMs as cases describing user's general travel preferences, and interactions between NutKing and the user during the route planning (queries launched, viewed attractions, chosen route etc).
Entertainment Personalization Mechanism Through Cross-Domain User Modeling
219
In the future, we plan to study the issue of semantic distance between different domains, as different domains may be of different relevance to the target domain. Another issue is overcoming the heterogeneity and resolving the conflicts between partial UMs in order to alleviate their integration and to build more accurate UMs.
References [1] S.Aguzzoli, P.Avesani, P.Massa, “Collaborative Case-Based Recommender System”, in Proceedings of the ECCBR Conference, Aberdeen, UK, 2002. [2] S.Berkovsky, “Ubiquitous User Modeling in Recommender Systems”, in Proceedings of the UM Conference, Edinbirgh, UK, 2005. [3] S.Berkovsky, P.Busetta, Y.Eytani, T.Kuflik, F.Ricci, “Collaborative Filtering over Distributed Environment”, in Proceedings of the Workshop on Decentralized, Agent-Based and Social Approaches to User Modeling, 2005, Edinburgh, UK. [4] L.F.Cranor, J.Reagle, M.S.Ackerman, “Beyond Concern: Understanding Net Users' Attitudes about Online Privacy”, Technical report, AT&T Labs-Research, 1999. [5] W.Dai, R.Cohen, “Dynamic Personalized TV Recommendation System”, in Proceedings of the Workshop on Personalization in Future TV, Pittsburgh, PA, 2003. [6] N.Good, J.B.Schafer, J.A.Konstan, A.Borchers, B.Sarwar, J.Herlocker, J.Riedl, “Combining Collaborative Filtering with Personal Agents for Better Recommendations”, in Proceedings of the AAAI Conference, Orlando, FL, 1999. [7] K.Goldberg, T.Roeder, D.Gupta, C.Perkins, “Eigentaste: A Constant Time Collaborative Filtering Algorithm”, in Information Retrieval Journal, vol. 4(2), pp. 133-151, 2001. [8] U.Hanani, B.Shapira, P.Shoval, “Information Filtering: Overview of Issues, Research and Systems”, in User Modeling and User Adapted Interactions, vol.11 (3), pp.203-259, 2001. [9] J.Kay, B.Kummerfeld, P.Lauder, “Managing Private User Models and Shared Personas”, in Proceedings of the Workshop on UM for Ubiquitous Computing, Pittsburgh, PA, 2003. [10] A.Kobsa, “Generic User Modeling Systems”, in User Modeling and User-Adapted Interaction, vol.11(1-2), pp.49-63, 2001. [11] C.Niederee, A.Stewart, B.Mehta, M.Hemmje, “A Multi-Dimensional, Unified User Model for Cross-System Personalization”, in Proceedings of the Workshop on Environments for Personalized Information Access, Gallipoli, Italy, 2004. [12] P.Resnick, H.R.Varian, “Recommender Systems”, in Communications of the ACM, vol. 40(3), pp. 56-58, 1997. [13] F.Ricci, B.Arslan, N.Mirzadeh, A.Venturini, “ITR: a Case-Based Travel Advisory System”, in proceedings of the ECCBR Conference, Aberdeen, Scotland, 2002. [14] B.Smyth, P.Cotter, “The Sky's the Limit: A Personalised TV Listings Service for the Digital TV Age”, in Proceedings of the ES Conference, Cambridge, UK, 1999. [15] O.Stock, M.Zancanaro, E.Not, “Intelligent Interactive Information Presentation for Cultural Tourism”, in O. Stock, M. Zancanaro, ‘Multimodal Intelligent Information Presentation’, Springer Publishers, 2005.
User Interview-Based Progress Evaluation of Two Successive Conversational Agent Prototypes Niels Ole Bernsen and Laila Dybkjær NISLab, University of Southern Denmark, Odense {nob, laila}@nis.sdu.dk
Abstract. The H. C. Andersen system revives a famous character and makes it carry out natural interactive conversation for edutainment. We compare results of the structured user interviews from two subsequent user tests of the system.
1 Introduction The Hans Christian Andersen (HCA) system has been developed in the European NICE project on Natural Interactive Communication for Edutainment (2002-2005). Computer games company Liquid Media, Sweden, did the graphics, Scansoft, Germany, trained the speech recogniser with children’s speech, CNRS-LIMSI, France, did the 2D gesture modules and the input fusion, and NISLab developed natural language understanding, conversation management, and response generation. 3D animated fairytale author HCA is found in his study in Copenhagen where he wants to have edutaining conversation with children (target users are 10-18 years) about the domains he is familiar with or interested in, such as his life, fairytales, himself, his study, the user, and the user’s favourite games. HCA. The intended use setting is in museums and other public locations where users from different countries can have English conversation with HCA for a duration of 5-15 minutes. The user communicates via spontaneous speech and 2D gesture while 3D animated HCA communicates through speech, gesture, facial expression, and body movement.
2 The HCA Prototype Systems Two prototypes (PT1 [1] and PT2 [3]) of the HCA system were tested with representative users in January 2004 and February 2005, respectively, following similar protocols. In both tests, subjective user data was gathered in post-trial interviews. Perhaps the most important difference between PT1 and PT2 is that PT2 uses automatic speech recognition. In PT1, speech recognition was emulated by human wizards. PT1 thus has near-perfect speech recognition whereas PT2 must deal with the additional technical difficulties of recognising the speech of children who, moreover, have English as their second language. Other important differences include that in PT2, the user can change the topic of conversation, backchannel comments on what HCA said, or point to objects in his study at any time, and be responded to when appropriate. This yields a far more flexible conversation than was possible in PT1. Also, M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 220 – 224, 2005. © Springer-Verlag Berlin Heidelberg 2005
User Interview-Based Progress Evaluation
221
the handling of miscommunication has been improved in PT2. Although HCA’s domain knowledge has been extended in PT2 as well, the major change is in the restructuring of his knowledge, i.e., in how the user can converse with HCA and get access to his knowledge, and in what HCA does when he has, or takes, the initiative. Contrary to PT1, HCA can in PT2 display several gestures simultaneously and has semi-natural lip synchrony as well as some amount of face, arm and body movement. In PT1, HCA has a single output state, i.e., the one in which he produces conversational output. If no user is present, he does nothing but wait. In PT2, when alone, HCA walks around thinking, looks out his windows, etc. However, this new output state is not properly integrated with the conversational output state, and HCA’s behaviour when alone is also sometimes rather weird. A problem in PT1 was that the gesture recogniser was always open for input. Those users who had a mouse and no touch screen tended to create large queues of gestures waiting to be processed, which generated internal system problems as well as some contextually inappropriate conversational contributions by HCA. The PT2 gesture recogniser does not “listen” while processing input. The same is true for the speech recogniser which does not have barge-in.
3 The User Tests PT1 was tested with 18 users (17 Danes and 1 Scotsman, 9 girls and 9 boys), 10-18 years old. PT2 was tested with 13 Danish users (7 girls and 6 boys), 11-16 years old. Both tests included two test conditions and similar sets of user instructions for both conditions. Two test rooms were prepared with: a touch screen, except that for PT1 one of the rooms had a standard screen and a mouse for pointing; a keyboard for changing virtual camera angles and make HCA walk; a headset; and two cameras for recording user-system interaction. The software was running on two computers. The animation was on the computer connected to the user’s screen and the rest of the system was on the second computer which, for PT1, was operated by the wizard and, for PT2, was being monitored by a developer out of sight of the user. User input, wizard input (PT1-only), system output, and interaction between modules was logged. Each user test session took 60-75 minutes. Sessions began with a brief introduction to the input modalities available. The PT2 headset microphone was calibrated to the user’s voice. The users were not instructed in how to speak to the system. In the PT1 test, this did not matter since the wizards would type in what the user said, ignoring contractions, disfluencies, etc., and only making few typos. We wanted to collect baseline data on how second-language speakers of English, most of whom had never spoken to a computer, talk to a conversational system with no prior instruction. After the introduction followed 15 minutes of free-style interaction. It was entirely up to the user what to talk to HCA about. In the following break, the user was asked to study a handout listing 13 (PT1) and 11 (PT2) proposals, respectively, for what the user could try to find out about HCA’s knowledge, make him do, or explain to him. It was stressed that the user did not have to follow all the proposals. The second session had a duration of approx. 20 minutes. In total, some 11 hours of interaction were recorded on audio, video, and logfiles for PT1, and some 8 hours in the PT2 test.
222
N.O. Bernsen and L. Dybkjær
4 The PT1 and PT2 User Interviews Users were interviewed immediately after their interaction with the system. The PT1 and PT2 interviews comprised 20 and 29 questions, respectively, see also [2, 3]. In both cases, the first six questions concerned the user’s identity, background, computer gaming experience and experience in talking to computers. For PT2, we also asked about the user’s experience in using a touch screen. Below, we focus on the other sections of the interviews which address system interaction and usefulness issues. Due to increased functionality 14 PT2 versus only seven PT1 questions deal with the user’s interaction with the system. Six PT1 and seven PT2 questions address system usefulness and suggested improvements. The questions are identical but for a PT2 question on overall system quality. In both interview series users were asked for any other comments. This question did not add any new information. Each user’s verbatim response to each question was scored independently on a three-point scale by two raters. Rating differences were negotiated until consensus was reached. An average score per question was then calculated (Figure 1). Grouping the issues raised in the interviews, the following picture emerges. HCA’s spoken conversational abilities have improved significantly in PT2. Conversation management problems do not enter into the PT2 replies on whether it was fun to use the system and if it was easy to use, and only rarely into the PT2 replies on what was bad about the interaction, but those problems figure prominently in the corresponding replies regarding PT1. Regarding PT1 users focused on slow gesture understanding and various problems in being understood. PT2 users focused on minor difficulties of manual control of camera angles and HCA’s locomotion. Concerning the question of what was bad about the interaction, PT2 answers have much less of: did not change topic when the user wanted to, irrelevant replies, too much repetition, did not answer questions. For both PTs, the users want HCA to have more knowledge. The answers to whether HCA could understand what was said and what was good about the interaction, support the conclusion that conversation has improved considerably. Despite the very significant decrease in speech recognition performance in PT2, the PT1 problems of: many unanswered questions, several unwanted repetitions, and HCA not following user-initiated topic change due to an overly inflexible conversation structure, are gone. Wrt. the question of what was good about the interaction, the PT1 and PT2 users agree that it was good to talk to HCA in English, get information about himself and his life, and point to objects and get stories about them. Criticism of HCA’s conversational abilities surfaces in the question about suggested improvements. In both PT1 and PT2, there is a wish that HCA can understand more. Conversely, the increase in animation articulation and expressiveness, and the reduction of the number of graphics bugs, in PT2 over PT1, is not rewarded by the users, cf. the questions on naturalness of animation, what was bad about the interaction and what should be improved. Despite PT1 users quite strong reaction to the presence of graphics bugs, the PT2 users react even more strongly to HCA’s unnatural walk and antics. Other new functionality in PT2, such as lip synchrony, is appreciated, however.
User Interview-Based Progress Evaluation
P2 score
223
P1 score
Usability and improvements Fun to talk to HCA Learn anything from talking to HCA Bad about interaction Good about interaction Suggested improvements Are you interested in this type of game Overall system evaluation Interaction Ease of use Naturalness of animation Could you understand what he said Would you like to do more with gesture How was it to do the gestures Could he understand what you said How did it feel to talk to HCA What do you think of HCA HCA behaviour when alone Natural to talk and use touch screen Coping with errors and misunderstandings Lip synchrony Quality of graphics How was the contents of what he said Did you talk while pointing Was he aware of what you pointed to 0
0.5
1
1.5
2
2.5
3
Average scores
Fig. 1. Average user ratings for each PT1 and PT2 question. Score 1 is the top score.
The intelligibility of PT2’s speech synthesis is appreciated. The PT2 touch-screen is praised as giving more control than the mouse, even though this contradicts the PT1 scoring for mouse vs. touch screen. The use of input gesture is found more satisfactory in PT2 than in PT1. It is far more work to gesture using the touch screen than using the mouse and this may have reduced the PT2 users’ wishes for more gesturable objects (the number of objects is the same for PT1 and PT2). The users’ views on learning from the system are also better for PT2 than for PT1. The improved conversation may have affected the users’ answers. Finally, the users’ interest in speech/gesture computer gaming are similar for PT1 and PT2. Two questions were only asked wrt. PT1 and nine only wrt. PT2. On the PT1-only question, what the user thinks of the HCA character, he is basically perceived as authentic. Given this positive feedback we included instead a more general question in PT2 about the quality of the graphics, which was rated rather good overall. Two more questions on the visual impression of PT2 were: one on the lip synchrony which users found quite good; and a question about HCA’s behaviour when he is alone in his study which was evaluated quite negatively. The second PT1-only question addressed how it feels to talk to HCA. The answers mainly reflect that users took some time getting used to speaking to the system. The corresponding PT2 question asks how natural it is to talk and use the touch screen. Users replied very positively. A new, related PT2 question was if the user talked while pointing and if it worked. Half of the users did not talk while pointing while the rest did so occasionally. The score reflects that the multimodal input worked for almost all users who tried. The related question about HCA’s understanding of pointing input was also answered very positively.
224
N.O. Bernsen and L. Dybkjær
Of the three final PT2-only questions, one was about the quality of the contents of what HCA says which were generally felt to be fine though HCA tends to talk too much and is not sufficiently helpful in helping the user find something to ask him about. The question about how easy it was to cope with errors and misunderstandings received the harshest average score of all (2.3). This is where the system’s imperfect speech recognition and limited vocabulary and domain knowledge take centre-stage. Finally, the users’ overall evaluation was good with a majority of positive comments.
5 Conclusion This paper has reported results from two similarly protocolled user tests with two research prototype generations of “the same” system. We have highlighted three major differences between PT1 and PT2, i.e., that only PT2 used automatic speech recognition, that conversation management was significantly improved in PT2 over PT1, and that PT2’s animation was far more expressive and versatile. Whereas improvements in conversation management seem to have more than outweighed the adverse effects of a substantial amount of speech recognition failure in PT2, PT2’s more expressive animation was not really perceived as natural or fun. The fact that users were neither instructed nor trained in how to speak to the system seems to have had a strong effect on their perception of the helpfulness of the system’s metacommunication. Our next step is to correlate the subjective user evaluations with objective analysis of the conversations based on coding schemes for conversation robustness and success.
Acknowledgements We gratefully acknowledge the support by the EU’s HLT Programme, Grant IST2001-35293. We would like to thank all colleagues at NISLab, Liquid Media, Scansoft, and LIMSI-CNRS who contributed to making the prototypes fit for user testing.
References [1] Bernsen, N.O., Charfuelàn, M., Corradini, A., Dybkjær, L., Hansen, T., Kiilerich, S., Kolodnytsky, M., Kupkin, D., and Mehta, M.: First Prototype of Conversational H. C. Andersen. Proceedings of AVI 2004, Gallipoli, Italy, 2004, 458-461. [2] Bernsen, N. O. and Dybkjær, L.: Evaluation of Spoken Multimodal Conversation. Proceedings of ICMI 2004, Penn State University, USA, 2004, 38-45. [3] Bernsen, N.O. and Dybkjær, L.: User Evaluation of Conversational Agent H. C. Andersen. Proceedings of Eurospeech, Lisbon, Portugal, 2005.
Adding Playful Interaction to Public Spaces Amnon Dekel1,2, Yitzhak Simon2, Hila Dar2, Ezri Tarazi2, Oren Rabinowitz2, and Yoav Sterman2 1
The Hebrew University Jerusalem
[email protected] 2 The Bezalel Academy of Art and Design, Jerusalem
[email protected] {hila, ezri}@bezalel.ac.il,
[email protected] Abstract. Public spaces are interactive by the very fact that they are designed to be looked at, walked around, and used by multitudes of people on a daily basis. Architects design such spaces to create physical scenarios for people to interact with, but this interaction will usually be one sided- the physical space does not usually change or react. In this paper we present three interaction design projects which add reactive dynamics into objects located in public spaces and in the process enhance the forms of interaction possible with them.
1 Introduction Public spaces are an inseparable part of the urban fabric. Such spaces function mostly as passageways from place to place, or for short term activities. A public park, for example, where people spend relatively more time, contains a larger percentage of non built areas, which create areas for sitting, resting, and playing. A city center's squares and boulevards allocate a smaller percentage of their areas for benches and other places to hang around in, while leaving larger areas for human pedestrian travel which takes place on sidewalks and walkways. The design of these spaces will affect the types of activities that will be held in them – exposed places will encourage public interaction, whereas enclosed spaces will enable more intimate social interactions. Both of these models are important for a city since they satisfy people’s basic needs for public or intimate social encounters, and even enable a form of urban escapism. 1.1 Playful Interactivity We define playful interactivity as any human computer interaction that has at its core no pragmatic goals- in such situations users are more interested in enjoying themselves than they are in achieving a specific task. Using this definition, most computer games are not examples of playful interactivity since the people playing them are usually very goal oriented in their efforts to beat another player or to gain more points. A good example of playful interactivity is a music generation game such as Stella and the Star-Tones [6] in which the user explores ways to create nice sounding music. When playing the game, users are not competing with anyone, but are rather in a state of “doodling” with music and visuals and enjoying themselves. The process M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 225 – 229, 2005. © Springer-Verlag Berlin Heidelberg 2005
226
A. Dekel et al.
usually lasts no more than a few minutes, after which one may get bored or simply have to return to work. In the projects outlined below, the goal was to add a playful interactive layer to objects that people are used to seeing in public spaces: water falls, rows of sculptural blocks, and benches. The motivation was to see how effective such playful interactivity would be, as well as to explore how its addition will affect the way people interact with objects they already know very well. The work builds on previous work on the creation of social interaction in public spaces by Yoon et. al. [8] and Rogers & Brignull [1], but differs in that it does not necessitate the use of high cost projection devices.
2 The Sonic Waterfall The Sonic Waterfall (designed and built by Yitzhak Simon) explores the effect of adding interactive aural elements to a public waterfall. A prototype was designed and built which included a pumped water system running over an ultrasonic sensor system aimed parallel to the water. As visitors ran their hands through the water they broke one of the hidden wireless sensing circuits which activated a sensor. An attached microcontroller (MC) communicated which sensor was activated to a nearby workstation. An application on the workstation parsed and used this data and combined it with a decision algorithm to play sounds or music in the vicinity of the waterfall.
Fig. 1. The Sonic Waterfall
The sonic waterfall was installed in one of the entrance corridors of the Bezalel Academy of Art and Design. Our observations found that the interactive audio element caused the waterfall to be extremely enticing and resulted in people standing to watch, waiting for their turn to play. One shortcoming we quickly noticed was that only one person could use the waterfall at a time if the desired outcome was to be aesthetically pleasing music. When we used non-musical audio, a more freeform interaction could take place, and more than one person could interact without downgrading the aural experience of the piece.
3 Musical Chairs The musical chairs project explores the addition of aural and visual interaction into a group of blocks set in a public space. In this project, designed and built by Yitzhak
Adding Playful Interaction to Public Spaces
227
Simon and Oren Rabinowitz, visitors see a row of stool high blocks. Sitting on a block creates a short audio-visual reaction. The real action starts when someone sits on another bloc; when that happens all the blocks in between erupt in a wave of audio-visual action. This wave creates a visual animation effect accompanied by sounds which together cause a musical and playful effect. If a “circuit” is cut by someone sitting on a seat in between the current ones being occupied, the wave now moves between the first seat and the new seat. The second seat is now left out of the loop, forcing that person to move to another block in order to get back into the action.
Fig. 2. Musical Chairs
Our observations show that when people first sit on a block, they are surprised by the audio-visual reaction, but subsequently start exploring by moving to another seat to cause a reaction. At this point a second person usually joins in, sitting on a second block in unison. The ensuing audio visual “wave” surprises the participants and causes them to start exploring. One of them gets up, then the other, and then they both sit down or get up together. In this process they learn the vocabulary of the system while enjoying themselves. While they are doing this, a crowd forms around them and then a third person joins the fun, cutting off the wave in the middle and exploring further options.
4 The Intimate Bench One of the most ubiquitous elements in any public space is the bench. People sit on them to read the morning newspaper, watch their children play, eat a sandwich, wait for someone, and even to sleep. An additional scenario that sometimes takes place on the bench is that of a couple who use it as a semi-intimate place to be together. The Intimate Bench project, designed and built by Yoav Sterman, is meant to explore this social scenario. The bench (seen in figure 3 below) looks and feels like an ordinary park bench, but embedded into it are seat sensors which are connected to an embedded MC. When someone sits on the bench it reacts by turning on embedded lights in various configurations which are sensitive to the position of the people sitting on it. Thus, when people sit far from each other (on both ends), it reacts in ways which try to get them to communicate and come closer together. When they are close together the Bench reacts in ways that are meant to strengthen the situation. The lights in the bench cause visual shapes (hearts and triangles in red and yellow) to appear, with the hope that the
228
A. Dekel et al.
shapes will subtly encroach into the participants interaction (if at first by surprise and delight, then by humor and a breaking of the social “ice”). The prototype chair has not been tested yet in public, but initial presentations found that onlookers seemed very enticed by the sudden appearance of shapes in the actual wood (which until then were completely invisible). Further testing is planned.
Fig. 3. The Intimate Bench: Left: Photo of the Bench with lights on. Right: System Schematic.
5 Conclusions and Summary This paper presented design projects in the Texperience (Technology for Experience) course1 given at the Bezalel Academy of Art and Design in Jerusalem in the Master of Design program in Industrial Design. The main goal of these projects was to add a playful interactive layer to objects that people are used to seeing in public spaces. Our motivation was to see how effective such playful interactivity would be next to exploring how its addition will affect the way people view and interact with objects they already know very well. Our observations brought us to the realization that as long as the interactive vocabulary of such systems is simple enough to be understood quickly by viewers and participants, then they can be effective. This is a clear barrier- a system that is too complicated to understand within a 30 second window will go unused in most cases. We view effectiveness in this context as a situation in which casual participants, who do not have much time on their hands, will successfully negotiate and understand the interaction model of the object and thus enter into an enjoyable and playful session with it. We also understood that the content and dynamics of the reactive objects must be designed to take all the possible interaction scenarios into consideration in order to be successful. As we saw in the Sonic Waterfall project, the successful creation of music was only possible when done by a single participant. A simple change in the sounds generated by the system (from music to sound effects) enabled multiple users to generate a pleasing sonic texture. As for how the addition of interactivity affects the way people view and interact with well known objects, we observed that the objects, that in many cases have become almost invisible in the background, suddenly gained peoples’ attention. People could not walk by without at least noticing that something unique and interesting was happening, and in many cases they stood around to get a chance to interact with the 1
The course was developed and taught by Amnon Dekel, Hila Dar and Ezri Tarazi.
Adding Playful Interaction to Public Spaces
229
objects. In some ways we could say that adding interactivity to such objects can bring them back to life, although we by no means should be understood as stating that interactivity should be added everywhere. All we hope to achieve is the opening of a discourse centered on the place of interactivity in the design of public spaces. Are such objects ready for prime time? We feel that more research and exploration must be done to explore how such interaction models will weather the passage of time- will they age elegantly, and take their place as a new form of public object? Or will they become aural and visual noise generators that annoy passers by, ultimately suffering damage by angry onlookers or turned off by city officials. Only field tests and time will tell.
References 1. Brignull, H, Rogers, Y.: Enticing people to interact with large public displays in public spaces. INTERACT'03. M. Rauterberg et al. (Eds.) Published by IOS Press, (c) IFIP, 2003, pp. 17-24 2. Carmona, M., Heath, T., Oc, T., Tiesdel, S: Public places – Urban Spaces, The dimensions of Urban Design, Architectural Press 2003 3. Lebesch, B., Sherman,C., Williams, G.: Stella and the Star-tones, Bohem Interactive, 1996 4. Paradiso, J. New Sensor Architectures for Responsive Environments. Talk given at Perdue University on October 13, 2004. See online PDF at http://www.iee.org/OnComms/PN/ controlauto/Paradiso%20paper.pdf] 5. Paradiso, J.A., Hsiao, K., Strickon, J., Lifton, J. and Adler, A., “Sensor Systems for Interactive Surfaces,” IBM Systems Journal, Vol. 39, No. 3&4, October 2000, pp. 892- 914 6. Stuart Reeves, Steve Benford, Claire O'Malley, Mike Fraser: Public life: Designing the spectator experience. Proceedings of the SIGCHI conference on Human factors in computing systems. 2005 7. Van Kleek, M: Intelligent Environments for Informal Public Spaces: the Ki/o Kiosk Platform. MIT Masters thesis, 2003. 8. Jennifer Yoon, Jun Oishi, Jason Nawyn, Kazue Kobayashi, Neeti Gupta. FishPong: Encouraging Human-to-Human Interaction in Informal Social Environments, CSCW’04, November 6-10, 2004
Report on a Museum Tour Report Dina Goren-Bar and Michela Prete ITC-irst, via Sommarive 18, 38050 Povo, Italy {gorenbar, prete}@itc.it
Abstract. A simulation study about some basic dimensions of adaptivity that guided the development of the personalized summary reports about museum visits, as part of PEACH are presented. Each participant was exposed to three simulated tour reports that realized a sequential adaptive, a thematic adaptive and a non-adaptive version, respectively, and subsequently on each of the dimensions investigated. Results were unexpected. The possible reasons are discussed and conditions under which personalized report generators can be preferred over non personalized ones are proposed.
1 Introduction The PEACH report generator [1, 5] allows visitors to continue interacting with exhibits after they have left the museum by creating a written report. The report includes features such as: a basic narration of their tour, the items and relationships they found most interesting – with images placed near the text to let them recall the artwork –, links to additional information on the internet, as a natural follow-up for a specific visitor’s interests, and even plan future visits at the current or at other museums. Based on prior studies on the PEACH guide [2], prior to performing a behavioral user study with the prototype, we aimed at understanding the perceived value of the adaptivity dimensions that should lead the final development of the real system, in this case the adaptive tour report, by means of an attitudinal study. In order to assess the adaptivity dimensions reflected in the reports, we wanted the stimulus situations to be abstracted away as much as possible from implementation details. In our case, we used three different reports just to demonstrate the adaptive dimensions. Then we interviewed the subjects on the generic character of those dimensions and not on the specific implementation of them. Our major concern in the present study was: do users perceive the differences between the summary reports in order to be able to evaluate what report they prefer? And more specifically, do they perceive the differences between the adaptivity dimensions that should lead the development of the different reports?
2 Dimensions of Adaptivity Participants compared at the beginning three types of summaries of museum tours two adaptive vs. one non adaptive): Personal-Sequential Report, Personal-Thematic Report and Non-Personal Generic Report. The difference between the adaptive M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 230 – 234, 2005. © Springer-Verlag Berlin Heidelberg 2005
Report on a Museum Tour Report
231
reports relies on the viewpoint the reports are generated with. The User Model Component that supports adaptation is described in [3]. The Non-Personal Generic Report presented a generic tour summary following the order of all the frescos shown in Torre Aquila, regardless of which have been actually viewed by the visitor. The three whole reports included all the adaptive/non-adaptive dimensions tested. Then, on each adaptive dimension, we stressed the contrast between two different possibilities. Following are the five dimensions of adaptivity that were identified for guiding the generation of an adaptive report: 1. Sequential vs. Thematic: In the Sequential Report the text was generated on the basis of the physical path of the visitor, while the Thematic Report referred to topics found interesting by him. 2. Personal Reference vs. Non-Personal Reference: The adaptive text related to topics that had interested the visitor and also had personal reference to him/her; for example: “Afterwards you have seen…”or “You were very interested in…” The Non-Personal Reference text didn’t make any personal reference to the visitor’s interest but presented a generic summary of all the frescos exhibited in Torre Aquila. 3. Personal Content vs. Non-Personal Content: The adaptive text conveyed information specifically to the topic that had most interested the user. The Non-Personal Content text related to all the frescos in Torre Aquila. 4. Personal Suggestion inside vs. General Suggestion Inside: The adaptive text made reference to related topics in unseen frescos in Torre Aquila, based on the inferred visitors’ interests. The generic text presents a generic report on all frescos in Torre Aquila including those not seen in the current visit. 5. Personal Suggestion Outside vs. GeneralPersonal Suggestion Outside: The adaptive text recommended visiting other sites based on the visitors’ inferred personal interests. The generic suggestion just recommended visiting other museums in the surroundings (Figure 1 shows an example of dimensions 3 and 5).
Fig. 1. An example of reference to related topics in other museums and sites and to the most interesting topic in an adaptive visit summary report
232
D. Goren-Bar and M. Prete
3 The Study This study had two main goals: (1) to assess if users perceived the difference between the reports (Personal-Sequence, Personal-Content, Non-Personal) and, if positive, to understand their attitude towards the reports; (2) to deeply investigate each dimension of adaptivity on which the reports are generated. Pilot Studies. We first ran two pilot studies in order to asses the evaluation metho dology. This step was crucial before engaging in the full fledged user study aimed at evaluating user attitudes towards the adaptive dimensions that should lead the generation of the different reports. Attitudinal Study. A within subjects design was adopted: each participant read all the reports and answered all the questions. Six different presentation orders of the reports were created (permuting the order of the Personal-Sequential Report (PS), PersonalThematic Report (PC-ontent) and Non-Personal Generic Report (NP)) in order to eliminate any possible order effect. Forty two (42 - 19 males and 23 females) students from the University of Trieste (Italy) volunteered for this study. They were randomly assigned to each condition, with a total of 7 subjects per condition. The results of the attitudinal study didn’t meet our expectations. The results of the second pilot study, beyond improving the methodology indicated that those users preferred the adaptive reports. However, this was not the case when we engaged in a larger study. Table 1 summarizes the results for the overall preferences. Our main result is that the participants preferred the Non Adaptive versions of the museum report. This was also confirmed when we compared the various adaptive dimensions. As [4], we will discuss the possible reasons for the unexpected results. Different populations participated in the pilots and in the attitudinal study. While in the pilot the participants had technological background, in the attitudinal study they were students with social sciences background. This is an important difference that should be taken into account since probably most of the visitors in museums are more interested in arts and humanities, probably lacking technological background. We performed content analysis by classifying the reasons assigned for each preference. Participants liked short, simple and schematic text. Learnable and easy (to read and remember) text should be given to visitors, without sophisticated or difficult words. They also expected not too many details. The attitude towards Personal Reference and adaptive dimensions was interesting. Participants perceived the Personal Reference as "being judged" by the system. Special care should be taken about the words and phrases chosen when generating a report, especially when it is performed by an automatic language generator. Words like: "just", “probably" or "you have…" might have an adverse meaning for the visitor, while might be part of basic elements in an automatic language generator. Moreover, several participants related to the “Big Brother effect”: people don’t want to be observed. While the capability of logging the visitor's actions and its context is crucial for a User Model Component enabling inference and adaptation, its effects towards the user should be controlled in an adaptive report generator. We assume that is better to allow visitors to be active and contribute during the visit to the
Report on a Museum Tour Report
233
report generator, for instance by enabling them to “save” during the tour some frescos/pictures/sculptures they prefer and, at the end, have a report with a picture on those works of art they had previously chose, instead of generating a personalized report based on the users tour. Table 1. Chi-Square for overall preferences (N and frequencies)
Total N= 42
N -%
N -%
Differences
Pleasant
YES 40 - 95 PS 13 - 31
NO 2 - 5 PC 9 - 21
NP 20 - 48
4.43
Unpleasant
21 - 50
13 - 31
8 - 19
6.14*
Useful
12 - 29
7 - 16
23 - 55
18.33**
I dimension
S 24 - 57 PR 12 - 29 PC 10 - 24 PSI 23 - 55 PSO 23 - 55
TH 18 - 43 NPR 30 - 71 NPC 32 - 76 GSI 19 - 45 GSO 19 - 45
II dimension III dimension IV dimension V dimension
N -%
ChiSquare 34.38**
0.86 7.71* 11.52** 0.38 0.38
** = significant (p < .01) * = significant (p < .05)
The personalization might even be more active to allow users to enrich themselves from this automatic capability, without being concerned about "being watched". It might be that the idea of “report or summary” does not elicit a personal experience. The idea of “my memories of the tour”, “my diary”, where the visitor could also add his/her impressions might arouse a positive context for an adaptive tour report. In our study, some of the participants perceived the idea of personalization as a limitation. We assume that probably the context was wrong. The Torre Aquila was a too limited scenario, it would be better to have “Louvre”, “MoMA” or “Uffizi” as a scenario; in these cases it’s impossible to receive a report on all the works of art, also because a visitor probably doesn't see all the artwork at once. This differentiation will be also tested in a future study. The idea of reference to unseen frescos should be put in context. In the context of Torre Aquila where all frescos relate to the same context maybe this capability is of limited value but in a large museum this dimension maybe very useful. We received similar reaction about recommending other museums, unless it
234
D. Goren-Bar and M. Prete
is related to other specific exhibits in other museums. The reasons for that seem to be that it doesn't differentiate much between the adaptive and the non adaptive reports. Reference to the most interesting scene seems not to be a dimension that can be easily tested in the context of Torre Aquila exhibition. All the frescos relate to the cycle of the months in the medieval period in Trento, the distinction between the frescos is based on the subtopics that they present. Again, it might be an important dimension for a larger exhibition.
Acknowledgments This work was conducted in the context of the PEACH project funded by the Autonomous Province of Trento.
References 1. Callaway, C., Kuflik, T., Not, E., Novello, A., Stock O. and Zancanaro M. Personal Reporting of a Museum Visit as an Entrypoint to Future Cultural Experience. In Proceedings of Intelligent User Interfaces IUI’05. San Diego, January (2005) 2. Goren-Bar D., Graziola I., Pianesi F. and Zancanaro M. Dimensions of Adaptivity in Mobile Systems: Personality and People’s Attitudes. In Proceedings of Intelligent User Interfaces IUI’05. San Diego, CA. January (2005) 3. Kuflik, T., Callaway, Ch., Goren-Bar, D., Rocchi, C., Stock, O. and Zancanaro, M. NonIntrusive User Modeling for a Multimedia Museum Visitors Guide System. UM 2005 UserModeling: The Proceedings of the Tenth International Conference. Edinburgh, Scotland. July 24-30 (2005) 4. Reiter, E., Robertson, R. and Osman. L Lessons from a Failure: Generating Tailored Smoking Cessation Letters. Artificial Intelligence 144 (2003) 41-58 5. Rocchi, C., Stock, O., Zancanaro, M., Kruppa, M. and Krueger, A. The Museum Visit: Generating Seamless Personalized Presentations on Multiple Devices. In Proceedings of Intelligent User Interfaces IUI-2004, Madeira (2004)
A Ubiquitous and Interactive Zoo Guide System Helmut Hlavacs1 , Franziska Gelies2 , Daniel Blossey2 , and Bernhard Klein1 1
2
Institute of Distributed and Multimedia Systems, University of Vienna, Lenaug. 2/8, 1080 Vienna, Austria {helmut.hlavacs, bernhard.klein}@univie.ac.at Otto-von-Guericke University Magdeburg, Fakult¨ at f¨ ur Informatik/IWS, Universit¨ atsplatz 2, D-39112 Magdeburg, Germany
Abstract. We describe a new prototype for a zoo information system. The system is based on RFID and allows to retrieve information about the zoo animals in a quick and easy way. RFID tags identifying the respective animals are placed near the animal habitats. Zoo visitors are equipped with PDAs containing RFID readers and WLAN cards. The PDAs may then read the RFID tag IDs and retrieve respective HTMLdocuments from a zoo Web server showing information about the animals at various levels of detail and languages. Additionally, the system contains a JXTA and XML based peer-to-peer subsystem, enabling zoos to share their content with other zoos in an easy way. This way, the effort for creating multimedia content can be reduced drastically.
1
Introduction
Personal digital assistents (PDAs) based on standard operating systems have become popular as basis for mobile information systems. Mobile information systems are used for instance by visitors of museums or other exhibitions, visitors of large companies or universities, or in the context of mobile learning. They may provide general information about the institution the visitor is in, about displayed items, or may provide navigational information about specific tours the visitor may follow. When presenting information about displayed items, the PDA usually is used for showing multimedia information, like text, pictures, or audio and video presentations. Such a visitor information system for exhibitions thus must master two tasks: (i) identify the exhibited item (or equivalently, determine the exact position of the PDA, and based on this information, the nearest shown item), and (ii) present the available multimedia information on the PDA. For (ii), the multimedia content may either be stored directly on the PDA, or may be fetched on-demand from a server via wireless LAN (WLAN).
2
Related Work
Dedicated visitor information systems based on PDAs have been implemented mostly for museums, but also other institutions. The most important difference M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 235–239, 2005. c Springer-Verlag Berlin Heidelberg 2005
236
H. Hlavacs et al.
is given by the technology for identifying the displayed items.1 One way is given by manually entering an identification number into the PDA, which is used for instance with the Personal Art Assistant (PAA).2 Another approach for determining the position of a visitor is to use a WLAN based positioning engine, for example the Ekahau engine3 , as for instance done in the Zoological Museum of the University of Munich [3]. In the CoBIT system [4], a light beam mounted on top of each exhibit sends information and energy to a head-mounted audio system. Similarly, the Musical Instrument Museum in Brussels uses infraredcontrolled headphones, which automatically play prerecorded tunes.4 Using GPS is a straight forward idea. However, GPS generally provides an accuracy of only about 3 m. Thus, GPS is mainly used outdoor for applications which do not need high positioning accuracy, like car navigation or city tourist guides [5,1]. In our system we use the RFID technology for identifying the exhibits. Here, an RFID tag is placed near the exhibit, and a PDA holding an RFID reader is placed near the tag to read the tag’s unique ID.5 For passive tags, the distance between reader and tag usually must be in the order of 10 cm or below. However simple the technical solution is, all multimedia presentations must be created by the respective institutions for presenting their usually unique exhibits. Creating the multimedia content requires a lot of effort, a fact that may prevent institutions from doing so. We present a system and a case study, where the effort for the production of multimedia content can be decreased dramatically. This is achieved by sharing content with other institutions, a scenario that makes sense, for instance, in the context of zoos.
3
The Ubiquitous Zoo Information System Prototype
Zoos have to provide information about every single animal they show. In general, most zoos offer information boards that are either affixed to the animal cages or next to them. Unfortunately this information is often sparse and limited to one or two languages. We therefore propose an interactive zoo guide which lets visitors obtain animal specific information right on their mobile devices. The requirements of such an interactive zoo guide include multi-language support, rich multimedia like audio commentary or selfexplaining video sequences, and on-demand information composition to different levels of details. The production of such detailed animal information however can result in an overwhelming effort for a single zoo. Therefore we propose the zoo to be connected to a global peer-to-peer system through which zoos share their animal descriptions and download missing information from other zoos to reduce time and costs for additional authoring in many languages. 1 2 3 4 5
http://www.cimi.org/whitesite/index.html http://www.pocket.at/business/kunstforum.htm http://www.ekahau.com/ http://www.mim.fgov.be/home uk.htm http://rfidjournal.com/article/articleview/1110/1/1
A Ubiquitous and Interactive Zoo Guide System
237
In the beginning the system administrator would setup the animal categories and assign RFID identifications for the animals. Then presentations and metadata is created for several (but not necessarily all) animals, at least one language, and various degrees of detail. To complement missing presentations, system administrators may periodically start a process within the zoo system, which searches for missing animal presentations and languages within the zoo community P2P system. Received results are stored locally in the zoo information base and made available to the visitor’s PDA. Zoo visitors are then equipped with PDAs having WLAN access to the zoo Web server, together with an RFID reader. When starting the zoo tour, visitors may first select their preferred language. Then, by holding the PDA close to an RFID tag which is attached to a specific cage, the PDA downloads the multimedia presentation for the animal living in the cage, and presents it on the PDA screen (Figure 1).
Fig. 1. Animal presentations on the PDA
4
RFID for PocketPC
On the client side we use a Toshiba e800 (PocketPC 2003) being equipped with a SanDisk WLAN (IEEE 802.11b) card plugged into the SDIO slot and an TAGflash RFID reader from TAGnology6 for the CF II slot. A Java class called PerformRFID monitors the RFID reader, the used Java VM is IBM’s J9 included into the WebSphere Device Developer Studio 5.5.0. The PerformRFID class calls 6
http://www.tagnology.com/
238
H. Hlavacs et al.
two native libaries via the Java Native Interface (JNI). The first is a DLL for reading data from the RFID reader via the serial port COM 5.7 Once a valid ID has been read, the PerfomRFID class calls a method from the native library IELaunch.DLL8 , which starts the Pocket PC’s Internet Explorer, together with a URL pointing to the zoo Web server servlet RequestInfo and the read ID. If a presentation for the ID for the selected type and language is available on the Web server, the servlet returns it to be displayed on the PDA’s Internet Explorer.
5
The Zoo Web Server
The core modules of the Web server include the information usage module to give visitors access to zoo relevant information, the information creation module enabling content creators to publish animal related information in a standardized way, and the information sharing and requesting module for enabling system administrators to download missing animal information from other zoos. As Web server we use an ordinary Apache Tomcat running on a Linux machine. A Web application called pda has been created and put into Tomcat’s webapps directory. Finding the right presentation for a given ID is implemented into the RequestInfo servlet. The read tag ID is passed to the servlet using a parameter called rfid. The RequestInfo servlet then opens the MySQL database table animal and looks for this ID in order to find the Latin name of the animal. For identifying each animal race, we chose the respective Latin name, which should be unique across the world. The Latin name, together with the language abbreviation and the content type are then used to construct a file name for an XML file, which should contain the requested animal presentation. Finally, if found, this XML file is turned into an HTML file which is returened to the PDA.
6
The Peer-to-Peer Backend System
The prototype’s information sharing and requesting module is based on JXTA9 , a widely used and mature peer-to-peer platform. Simple methods exist for making files available to other peers, for searching for files, and for downloading files [2]. To establish a zoo content management system (CMS), first a peer group like ”jxta-ZooGroup” for the zoo community should be created to restrict discovery queries automatically to members of the zoo community. During the CMS startup content advertisements are created for all shared files. These content advertisements contain information about the files, e.g., filename, unique content ID (cid), length in bytes, and for our system additionally the language and level of detail (level ”a” for a short description, level ”b” for a 7 8
9
http://www.ulrich-roehr.de/software/ipaq/serial/serial.html http://www.pocketpccity.com/software/pocketpc/IELaunch-2001-12-13-cepocketpc.html http://www.sun.com/software/jxta/
A Ubiquitous and Interactive Zoo Guide System
239
detailed presentation, and level ”c” for information concerning a specific zoo or zoo animal, which should not be shared with other zoos), to enable other peers to find these files in the JXTA network. Each peer stores shared content in the share folder. The share folder should be located within the Web server, such that all shared files are immediately available for download to the PDAs. All peers in the jxta-ZooGroup in this demonstration receive the discovery request of the query. Then, the remote peers look into their cached advertisements, and if they find the appropriate content advertisements, reply with a discovery response message which contains a list of the found content. After receiving the search results, the information sharing and requesting module downloads the corresponding XML file containing the animal presentation, together with additional graphics files. After the download is completed, the module then adds entries for the respective animals, languages and levels of detail to the corresponding tables in the MySQL database, thus enabling the Web server to send the new content to the zoo visitors.
7
Conclusion
In this paper we present a ubiquitous information system for zoos. Visitors are handed over a PDA with WLAN and RFID reader, by holding the PDA near an RFID tag, the visitor is presented multimedia information about the zoo animals. The main innovation of our prototype is the use of a peer-to-peer system, which enables the sharing of animal presentations between zoos on a global scale. Especially the exchange of presentations in various languages is of great interest for zoo visitors. This sharing decreases the effort for creating content significantly. Of course, sharing only makes sense if zoos worldwide show animals of similar race, an assumption which is valid for sure for many zoos and animals.
References 1. G. Benelli, A. Bianchi, P. Marti, D. Sennati, and E. Not. Hips: Hyper-interaction within physical space. In IEEE International Conference on Multimedia Computing and Systems Volume II-Number 2, 1999. 2. J.D. Gradecki. Mastering JXTA: Building Java Peer-to-Peer Applications. John Wiley & Sons, 2002. 3. B. Inderbitzin. Digitale annotation realer objekte aufgrund ortsabhngiger daten am zoologischen museum zrich. Master’s thesis, Universitt Zrich, Institut fr Informatik, January 2004. 4. T. Nishimura, H. Itoh, Y. Nakamura, and H. Nakashima. A compact battery-less information terminal for interactive information support. In The Fifth Annual Conference on Ubiquitous Computing (Ubicomp 2003), Workshop: Multi-Device Interfaces for Ubiquitous Peripheral Interaction, 2003. 5. G. Pospischil, H. Kunczier, and A. Kuchar. Lol@ - a umts location based service. In International Symposion on 3rd Generation Infrastructure and Services, July 2001.
Styling and Real-Time Simulation of Human Hair Yvonne Jung and Christian Knöpfle Fraunhofer-IGD, Fraunhoferstr. 5, 64283 Darmstadt / Germany
[email protected],
[email protected] Abstract. We present a method for realistic, real-time simulation of human hair, which is suitable for the use in complex virtual reality applications. The core idea is to reduce the enormous amount of hair on a human head by combining neighbored hair into wisps and use our cantilever beam algorithm to simulate them. The final rendering of these wisps is done using special hardware accelerated shaders, which deliver high visual accuracy. Furthermore we present our first attempt for interactive hair styling.
1 Introduction There is a big trend towards the use of virtual characters (virtual humans, embodied agents) for novel user interfaces. Within several application areas, visual realism of such characters plays an important role, especially when non-verbal communication gets important. To achieve this realism the visual appearance and movement of the character must be very close to reality, embracing the skin, the eyes, the hair, clothing, gestures, locomotion etc. Modern film productions already show that it is possible to simulate virtual characters which look completely real. But there are two major drawbacks of the technologies used in this area: The algorithms are fairly slow, thus simulation and rendering of a single frame takes minutes and hours. Highly reactive user interfaces must need real-time performance with an update rate of 30 frames per second. Thus animation and rendering must be very fast and should still leave a decent amount of processing power to the application. Creation of realistic characters and its animation is a very time consuming task and can be carried out by specialists only. For a major break-through of realistic virtual characters for novel user interfaces it is inevitable that their creation gets simplified and animation as well as rendering performance significantly improved. Within this paper we will focus on the realistic simulation and rendering of human hair, offering a solution for one of the important topics towards realistic virtual characters. The simulation of human hair is still an open area of research. As of today the simulation and rendering of each single hair, together with all other aspects which make up a realistic virtual character, overstrains any common PC platform. Many simplifications have to be made to be able to perform all these tasks together in realtime. In this paper we propose a wisp hair model based on quad strips, which is animated by a modified cantilever beam simulation. Rendering is done with GLSL shaders and includes amongst other things anisotropic reflection and a second specular highlight, which is characteristic for light colored hair. M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 240 – 245, 2005. © Springer-Verlag Berlin Heidelberg 2005
Styling and Real-Time Simulation of Human Hair
241
2 Related Work In order to create convincing human hair there are basically four problems to solve: modeling, hair dynamics, collision detection and response and finally rendering. Presently a seamless transition between these categories is still problematic because the fewest hair simulation systems are self contained and they all differ in their geometrical representations, animation methods, and lighting models. First a short review of common hair models is necessary, because not every hair model is suitable for every animation and rendering method. Thalmann et al. [1] classify hair models into several categories. The first one contains explicit models, which are relatively intuitive but computationally expensive, because every single hair strand is considered individually. The next category is the cluster models, which utilizes the fact that neighboring hairs have similar properties and tend to group together. They can be further divided into hierarchical (e.g. [2]) and flat models. More common are non-hierarchical schemes in which hair clusters are represented by generalized cylinders [3] or polygon strips. Volumetric textures are used for very short hair. The latest category regards hair as a fluid. In [4] a method is suggested for adding curliness by modulating the hair strands with offset functions. Animation can be done via key-framing on the one hand and numerical simulations on the other hand. A computationally cheap animation method also based on differential equations is the cantilever beam simulation originally proposed by Anjyo et al. [5] for computing the hair bending of smooth hairstyles during the hair shape modeling process. The main drawback of most simulation methods is the fact, that the simulators often can't guarantee for the hair style's preservation and they mostly neglect the existence of head and body movement in collision detection. But collision detection and response are a fundamental part of hair animation. Whereas hair-hair collision for interactive applications is mostly ignored or quite coarsely approximated (e.g. [6]), the treatment of hair-head collision is absolutely inevitable. Whilst geometry traversal, or grid based schemes offer more precision, for real time applications a head can be approximated sufficiently with the help of spheres or ellipsoids. Last but not least rendering itself covers the full range from drawing single-colored poly-lines and alpha-textured polygonal patches over heuristic local lighting models for anisotropic materials (e.g. [7]) up to physically and physiologically correct illumination solutions (e.g. [8]).
3 Geometric Model and Dynamic Simulation In order to reduce the geometric complexity and to avoid rendering problems we model hair wisps as relatively small quad strips (fig. 1, left), which are appropriately layered on top of the scalp mesh for providing an impression of volumetric qualities. They can consist of a variable number of segments along the direction of hair growth, which is specified by the position of the hair root and an initial hair tangent vector. After having specified all necessary parameters like hair distribution, width, height and number of segments the hair style is generated conforming to these parameters.
242
Y. Jung and C. Knöpfle
Fig. 1. Quad strip based hair model (left) and cantilever beam simulation (right)
Our hair simulation is derived from the cantilever beam method. Internally it works on a cinematic multi-body chain, as illustrated in fig. 1, right. Compared to mass spring approaches, which are used for hair dynamics quite frequently, but not necessarily do converge if forces are too strong or the time steps are too big, our cantilever beam method provides a much faster, numerically simpler and visually more convincing way to simulate hair. Besides this the initial distance between connected vertices can be fully conserved. The nodes of the multi-body chain are defined by the vertices of the original geometry and can be seen as joints connecting the edges between them. Two different types are distinguished, anchors and free moving vertices. Anchors, resembling the hair roots, are connected to the scalp and serve as the attachment point of the chain, whereas all the other vertices in the chain are only moved due to external forces, the bending forces caused by their connected neighbor vertices and by applying the length conservation constraint. An external force F, which is acting on a chain link, results in a bending moment M, which causes a deflection of the actual segment along the direction of F. This calculation is simplified by means of a heuristic approach: instead of calculating the torques, all forces, which are acting on the succeeding segments, are expressed by adding some offset vectors. Then length conservation is achieved by scaling the resulting vector back to the rest length. Besides a convincing simulation method a natural behavior in case of collisions is also required. Collision detection can be divided up into hair-body and hair-hair interaction. Because of the large amount of hair the trade-off between quality and speed of the collision detection has to be taken into account. Collisions with head or body must be treated explicitly. Because here users usually take no notice of a relatively low accuracy, internally the head is approximated by using collision objects like spheres, for which intersection tests can be handled quite efficiently. Hair-hair collision can't be handled easily in real-time. Thus the interpenetration of wisps is avoided by arranging the hair strips on top of the scalp in different layers. For keeping this up during dynamics, each vertex is assigned a virtual collision sphere with a different radius. This also has the nice side effect that implicitly collisions of hair segments with the head are handled too. Furthermore the problem is alleviated by using a slightly different bending factor for every chain, based on the position of its respective anchor. This layered collision avoidance structure is illustrated in fig. 2.
Styling and Real-Time Simulation of Human Hair
243
Fig. 2. Collision handling
4 Rendering The rendering is divided up into a CPU based part, in which the primitive sorting is done, and into a GPU based part, in which the lighting calculations are done. To further improve the visual appearance we implemented realistic specular highlights, shadowing, ambient and diffuse lighting for the hair rendering. For optimal performance we used hardware accelerated shaders. For more detailed information see [9].
5 Hair Styling To allow users modeling different hair styles we developed a hair creation plug in for the 3D modeling software Cinema4D (see fig. 3). It allows loading a human head and then placing interactively hair strands on top of the head. The base color, length and direction of each hair strand can be adapted individually. Since the hair styling highly
Fig. 3. Cinema4D plug in for hair rendering
244
Y. Jung and C. Knöpfle
depends on the dynamic simulation, the whole simulation system was embedded in the plug in, thus the user can immediately experience the results of his modeling and how it will look like in the virtual reality environment. The final result of the modeling can be exported in an X3D compatible file format.
6 Results and Future Work Our heuristically motivated hair simulation system is computationally cheap because of its special chainlike structure and properties like pliancy can be easily included. Moreover the system runs numerically very stable, is fast and looks convincing. By adopting approaches from physically based rendering to simpler GPU-based methods, we are still working on a phenomenological level. However, the result is very close to photorealism even for light colored hair. In fig. 4 (left) different states of the simulation can be seen. This hairstyle has 2388 wisps, and runs at 24 fps on a Pentium IV PC with 2.8 GHz, 1 GB RAM and a GeForce 6800 GT graphics card. Currently we are investigating the possibility of extending our system for the simulation of other hair styles, because without any further extensions the simulator is only suitable for smooth hair styles Furthermore we will improve our first attempt for hair styling and improve the usability and flexibility of the Cinema 4D plug in. The goal is that users are able to create realistic hair styles within a couple of minutes. Beside hair, realistic skin rendering is another major topic we will work in the future.
Fig. 4. Snapshots taken from a running simulation (left) and freaky hair (right)
References [1] N.Magnenat-Thalmann and S.Hadap and P.Kalra: “State of the Art in Hair Simulation”, in Proceedings of International Workshop on Human Modeling and Animation, Korea Computer Graphics Society, 2002 [2] Kelly Ward and Ming C. Lin: “Adaptive Grouping and Subdivision for Simulating Hair Dynamics “, in Proc. of the 11th Pacific Conf. on CG a. Applications, 2003 [3] Tae-Yong Kim and Ulrich Neumann: “Interactive multiresolution hair modeling and editing”, in Proceedings of SIGGRAPH '02, 2002 [4] Y. Yu: “Modeling Realistic Virtual Hairstyles”, Proceedings of Pacific Graphics, 2001 [5] Ken-Ichi Anjyo and Yoshiaki Usami and Tsuneya Kurihara: “A simple method for extracting the natural beauty of hair”, in Proceedings of SIGGRAPH '92 [6] Dehui Kong and Wenjun Lao and Baocai Yin: “An Improved Algorithm for Hairstyle Dynamics”, Proceedings of the 4th IEEE Int. Conference on Multimodal Interfaces, 2002
Styling and Real-Time Simulation of Human Hair
245
[7] J. T. Kajiya and T. L. Kay: “Rendering fur with three dimensional textures”, in Proceedings of SIGGRAPH ’89 [8] Stephen R. Marschner and Henrik Wann Jensen: „Light scattering from human hair fibers“, ACM Trans. Graph., Volume 22, 3/2003 [9] Yvonne Jung, Alexander Rettig, Oliver Klar, Timo Lehr: „Realistic Real-Time Hair Simulation and Rendering“, in Proceedings of Vision, Video and Graphics, 2005
Motivational Strategies for an Intelligent Chess Tutoring System Bruno Lepri, Cesare Rocchi, and Massimo Zancanaro ITC-irst, 38050 Trento, Italy {lepri, rocchi, zancana}@itc.it Abstract. The recognition of student’s motivational states and the adaptation of instructions to the student’s motivations are hot topics in the field of intelligent tutoring systems. In this paper, we describe a prototype of an Intelligent Chess Tutoring System based on a set of motivational strategies borrowed from Dweck’s theory. The main objectives of the prototype are to teach some chess tactics to middle-level players and to help them to avoid helpless reactions after their errors. The prototype was implemented using Flash Mx 2004. The graphical user interface encompasses a life-like character functioning as tutor.
1 Introduction In recent years, the recognition of learner’s motivational states and the adaptation of instructions to the learner’s motivations have become hot topics in the field of intelligent tutoring systems. In this research area we can identify two different approaches to motivational issues. The first approach aims at improving the student’s motivational state: it aims at designing and constructing systems able to enjoy, involve and motivate the students [8]. The second approach considers the motivation as a factor that influences the learning. Such approach tries to link student’s behavior to motivation and consequently to influence the student learning [1]. In our work we extended the latter approach emphasizing the individuation and the implementation of several motivational pedagogical strategies useful to influence the student’s behaviors. In this paper we describe a Motivational Intelligent Tutoring System that explains and teaches three different chess tactics (mate in one, pink and fork) and helps students to not de-motivate themselves for errors made in problem solving activity1. The choice of chess tactics learning as field of application for the prototype is due to the foremost importance that student’s reactions to errors play in chess learning. Although it is very easy to make an error during a chess game, students may fruitfully use errors to review their game plans and to find new tactics and strategies to apply.
2 Implicit Self-theories As a first step toward a motivational-oriented tutoring system, we need a theory that explains why some students have helpless reactions after their errors and some 1
We want to thank Shay Bushinsky for the helpful insights about chess and chess tutoring and Oliviero Stock for the fruitful discussions on intelligent tutoring systems.
M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 246 – 250, 2005. © Springer-Verlag Berlin Heidelberg 2005
Motivational Strategies for an Intelligent Chess Tutoring System
247
students do not. In the design and in the implementation of our prototype we were inspired by a model of psychology of motivation elaborated by Dweck called “Implicit Self-Theories” [4]. Dweck individuates the cause of the helpless reactions in the theory that the students have of their own intelligence and asserts that it is possible to act pedagogically on such idea in order to help students to not de-motivate themselves. According to Dweck’s model, students develop either an entity or an incremental theory of their own intelligence. Students with an entity theory have helpless reactions (worsening of the performance, de-motivation) after mistakes; whereas students with an incremental theory are more keen to self-motivate themselves. Therefore, it is pedagogically important to help students to develop an incremental theory of their own intelligence. Dweck suggests some strategies to obtain this goal: (i) to give negative feedbacks not regarding the person or the result but on the commitment put in the performance; (ii) to assign complex tasks and challenges to solve to students with high performances; (iii) to show videos in which actors play as if they were incremental students (modeling); (v) to reject the student’s help request if s/he demonstrated little commitment in the problem solving activity.
3 Student Modeling We consider three student’s characteristics: (i) the theory the student has about his own intelligence; (ii) the level of the student’s commitment; (iii) the student’s knowledge of each one of three chess tactics. The theory the student has about his own intelligence is inferred from a questionnaire submitted to the students the first time [3]. In order to model the student’s chess tactic knowledge, we used the knowledge tracing method. It allows estimating the probability that the student masters a specific skill observing his attempts in applying it [2]. The probability that the student masters the tactic “mate in one” after n completed problems is the sum of two probabilities: (i) the revisited estimation of the probability that the tactic was already known on the basis of the evidence (problem completed successfully or unsuccessfully); (ii) the probability that the student has learned the tactic after completing problem n. In Dweck’s theory the feedback on the student’s commitment is very important. In our prototype, we assumed that the persistence in the problem solving activity is an indicator of the student’s commitment and that it can be measured considering the number of attempts and the time put into the problem solving activity. The threshold value that distinguishes between many and few attempts and between much and little time depends on the type of task (difficult/simple). It has also been assumed that a student who frequently asks for help is less committed that a student who tries to solve the problem by himself [5, 7]. Finally, the outcome of the problem solving activity is another relevant factor: a student who gives up the problem usually does not exhibit much commitment. So, to model the student’s commitment we implemented 24 production rules obtained combining the values of four conditions: (i) the time a student uses to solve a problem ( “much” or “little”), (ii) student’s attempts to solve a problem (“many” or “few”), (iii) help (“requested” or “not requested”), (iv) problem’s outcome (“solved successfully”, “not solved” or “abandoned”).
248
B. Lepri, C. Rocchi, and M. Zancanaro
4 The Prototype The prototype was implemented using Flash Mx 2004. The graphical user interface encompasses a life-like character functioning as tutor. The character is able to accomplish some deictic gestures, e. g. indicate the chessboard, and to perform precodified utterances stored as external mp3 files [see Fig. 1]. Although the character is simple, we decided to use it as tutor to make more appealing the interaction between the prototype and the student. In fact, some experiments highlight the students are more motivated using tutoring systems with life-like characters (the persona effect) [6]. The pedagogical module is given by two sets of production rules. The first set encompasses 48 production rules and determines the tutor feedback on the student’s problem solving activity. The feedback is the set of actions that are carried out by the tutor after the student’s problem solution namely the comment on problem outcome, the comment on the student’s commitment, the choice of the new problem and the use of several motivational strategies [see Sect. 2].
Fig. 1. A snapshot of the Graphical Interface (left) and a sketch of the architecture (right)
Four conditions are considered: (i) student’s chess tactic knowledge (“mastered” or “not mastered”); (ii) student’s commitment (“null”, “low”, “medium” or “high”); (iii) the theory the student has about his own intelligence (“incremental” or “entity”); (iv) problem outcome (“wrong”, “abandoned” or “solved successfully”). An example of such a rule is the following: if the student is an incremental one, shows little commitment, solved the last problem and does not master the current tactic then the character selects a harder new problem and says: “You solved this problem, but you did not show much commitment. Are you sure of your move or are you just lucky? Let see how you will manage this new problem”. The second set, 16 production rules, determines the answer the tutor gives to a student’s help request. The 16 production rules are obtained combining the values of three different conditions: student’s chess tactic knowledge; student’s commitment; the theory the student has about his own intelligence. As an example, consider the following rule: if the student is an entity one, shows high commitment and knows the current tactic then the character gives an advice to the student indicating the chessboard and saying “ Uhm, do not ask for help just at the first trouble. Make some attempts and do not fear mistakes. Anyway, do not mind the position of your knight”.
Motivational Strategies for an Intelligent Chess Tutoring System
249
Let’s follow a couple of examples. John is a middle-level chess player and he is an entity student. He proceeds to solve problems excitedly until he makes an error. His initial confidence is shaken after his error. John asks for help to the character without making any attempts to solve the problem. The problem is adequate to his “mate in one” knowledge, so the character recognizes the student’s behaviour as an example of helpless reaction and says: “Do not let it daunt you. All chess players make errors, even chess masters do. Yet errors can be used to devise new tactics or strategies to solve the problem. Try again! By the way, what do you think of the black knight?” Mark is a middle-level chess player too and he is an incremental student. He wants to improve his “mate in one” tactic knowledge and he is sure to learn it attempting to solve the problems given by the tutor. While Mark is solving many problems, the tutor does not praise his intelligence but rather his commitment and gives him more and more difficult problems. When Mark makes an error, the tutor chooses to challenge him: “I can’t believe you are not able to solve this problem. Maybe, you were not concentrated enough. Do you think you are an expert? You’ll need much more commitment to become a good chess player. Try to look at the problem again”.
5 Future Work The next step in our work will be a full-fledged evaluation of the prototype. In this regard, we will evaluate how many entity students will develop an incremental theory of their own intelligence and to what extent the system succeeded in avoiding helpless reactions after errors. A further evaluation will be the assessment of the role of the different components of the prototype, for example the use of a life-like character as tutor, in achieving the motivational goals.
References 1. Baker, R.S., Corbett, A.T., Koedinger, K.: Detecting Student Misuse of Intelligent Tutoring Systems. In: Proceedings of the 7th International Conference on Intelligent Tutoring Systems. Lecture Notes in Computer Science, Vol. 3220. Springer-Verlag, Berlin Heidelberg New York (2004) 531-540 2. Corbett, A.T., Anderson, J.R.: Knowledge Tracing: Modeling the Acquisition of Procedural Knowledge. User Modeling and User-Adapted Interaction, Vol. 4. Kluwer Academic Publishers, The Netherlands (1995) 3. De Beni, R., Moè, A., Cornoldi, C.: AMOS. Abilità e Motivazione allo studio: Prove di valutazione e di orientamento. Erikson, Trento (2003) 4. Dweck, C.S.: Self-Theories: Their Role in Motivation, Personality and Development. Psychology Press, Philadelphia (1999) 5. Karabenick, S. (ed.): Strategic Help Seeking for Learning and Teaching. Mahwah, NJ: Erlbaum (1998) 6. Lester, J. C., Converse, S. A., Khaler, S. E., Barlow, S. T., Stone, B. A., Bhogal, R. S.: The persona effect: Affective impact of animated pedagogical agents. In: Proceedings of the Conference on Human Factors in Computing Systems. ACM Press, New York (1997) 359366
250
B. Lepri, C. Rocchi, and M. Zancanaro
7. del Soldato, T., du Boulay, B.: Implementation of motivational tactics in tutoring systems. Journal of Artificial Intelligence in Education, Vol. 6. (1995) 337-378 8. de Vicente, A., Pain, H.: Informing the detection of the student’s motivational state: an empirical study. In: Proceedings of the 6th International Conference on Intelligent Tutoring Systems. Lecture Notes in Computer Science, Vol. 2363. Springer-Verlag, Berlin Heidelberg New York (2002) 933-943
Balancing Narrative Control and Autonomy for Virtual Characters in a Game Scenario Markus L¨ockelt, Elsa Pecourt, and Norbert Pfleger DFKI GmbH, Stuhlsatzenhausweg 3, 66123 Saarbr¨ ucken, Germany {loeckelt, pecourt, pfleger}@dfki.de
Abstract. We report on an effort to combine methods from storytelling and multimodal dialogue systems research to achieve flexible and immersive performances involving believable virtual characters conversing with human users. The trade-off between narrative control and character autonomy is exemplified in a game scenario.
1
Introduction
Story elements in computer games are increasingly used to enhance the playing experience. Even simple contemporary action games often incorporate intricate narrative scenes (in-game or cinematic) involving other characters during actual gameplay to explain the player’s progress. Most of the time, however, these scenes allow little to no direct player interaction. On the other hand, there are games that allow the user conversations with game characters that change the game world (e. g. the Infocom series, or The Sims 2). But this is often not guided by a real story, but limited to moving around various locations collecting clues and solving puzzles (in adventures), or “living a life” without an overarching story or goal (in the person simulator case). We look into the possibilities offered by combining components from multimodal dialogue system and storytelling research to achieve immersive and believable interaction for human users in a virtual world with a narrative background. A narrative component directs agents of the dialogue system representing virtual characters and sets them individual goals that let them enact a coherent story. There are difficulties with this approach, also pointed out in e. g. [5]. It is vital to balance story control and character autonomy in a way to achieve the narrative goals while retaining an interesting variety of character behaviors. If the user is given an extended choice of interactions with real consequences in the story world, it must be possible to tackle situations where the narrative could be jeopardized by user actions, the other characters must be responsive to the user, as well as believable and entertaining.
2
Autonomy Versus Control
We use different components sharing the same ontological knowledge representation to supervise the narrative, deliberative and reactive aspects of character M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 251–255, 2005. c Springer-Verlag Berlin Heidelberg 2005
252
M. L¨ ockelt, E. Pecourt, and N. Pfleger
behavior. The main objective is to keep the story going without destroying the perception that the user is dealing with characters acting by their own will. The control levels synchronize by exchanging constraint and feedback messages in addition to effecting actions: the autonomy of each component can be constrained by the others. There are different narrative goal granularities, from direct presentations (e. g. camera movements) over atomic actions up to complex activities that can be achieved in relatively autonomous manner by the characters. 2.1
Knowledge Resources
The knowledge resources are modeled using an ontology containing all relevant concepts and their relationships. Ir comprises three sections representing different aspects of our storytelling environment: (1) Story World: A representation of the stages, characters, and props populating the story, including the relationships and the possible interactions between them, their roles, etc. It also includes the events and processes that can happen in the story, their participants, preconditions and postconditions, causal relations, timing, etc. (2) Fictional mediation: This section contains data relevant for the control of the overall story structure and allows the author to model her desired story by instantiating concepts of the ontology. Narrative units are modeled at different abstraction levels, and include slots that represent connectivity data (e. g. temporal and causal relations with other units, timing restrictions, participants). (3) Inter-Agent communication: This section provides concepts used for inter-agent communication and the representation of user contributions. Any action performed by a participant is an instance of a subclass of Act, such as DialogAct, NonVerbalAct, or PhysicalAct. These concepts themselves introduce subclasses describing more specific subtypes, e. g., Statement as subconcept of DialogAct. 2.2
Narrative Control
Current interactive storytelling often follows a branching structure representing the possible paths the user can take to achieve a well-formed story. The user is given a repertoire of possible actions, which can be represented as a graph. This allows control over the resulting narrative but the freedom of action is very restricted. Providing a wider range of possible actions implies an combinatorial explosion of branches in the graph, beyond manageability by a human author. In order to enhance the interaction possibilities the story graph has to be made implicit by specifying a story policy, which is used for the selection of the next story event, instead of manually linking the nodes of the graph. We use a specialized module, a narration component or director, which controls the progress of the story from a high level of abstraction. The author must then specify: (1) a representation of the story, (2) a collection of story events at different granularities: beat, scene, etc. and (3) an algorithm that, given the current state of the story, the model of the desired story, and the player actions, decides the next story event. Story guidance happens at certain points of the story and leaves the user free space to interact in the remaining time.
Balancing Narrative Control and Autonomy for Virtual Characters
253
A story grammar contains rules governing the sequencing of story events to build a well-formed story. The story events hold narrative goals that the characters search to achieve in an autonomous way. Story events can be defined at different granularities and levels of abstraction. The greater the level of abstraction, the lower the narrative control will be. Dramatic arcs represent the course of the desired story by means of value points that have to be achieved at determined points of the story. The narrative control selects the story event that increases the probability of achieving the next point most. For example, a suspense arc determines how much suspense the story has to show at determined time points. Story value arcs are coupled with declarative knowledge attached to the narrative events, which includes preconditions over facts from the story history, priority tests, weight tests, and postconditions (the consequences of the execution of the event in the story world). 2.3
Action Planning: Dialogue Games Under Narrative Control
The action planner devises and executes a combination of planned and scripted activities geared towards achieving the goals of the characters. Activities are ontological objects parameterized by role values and triggered by an individual module for each character. During the execution of an activity, the world state is altered by dialogue with other characters and physical actions. Interactions can be atomic (acts) or composite (dialogue games). The narrative goals represent the smallest externally controllable units of value change in the story world. Dialogue games describe dialogue act exchanges with other participants in terms of rule-governed move sequences. A move has a set of preconditions to be legal, and it promises to produce postconditions. A precondition for a statement could be e. g. that the initiator believes some fact F and wants the addressee to also believe F ; a postcondition could be that the addressee knows that the initiator believes F , and has the (social) obligation to give feedback whether it accepts F . Games also have preconditions and postconditions; composite games can be formed by combining games and acts. Following [3], the games are used to coordinate joint character actions; a character assumes that others fulfill obligations. However, a grumpy character can obviate these obligations by e. g. choosing not to answer a question. [6] gives a more detailed description of the planning and execution process. The narrative control can directly affect the game world to effect changes in scene or world state (e. g. asserting a door is closed). It can also set goals for individual characters. A goal is an ontological object associated with a procedural meaning . Slots of the object correspond to roles of the goal. There are several ways to provide narrative feedback. One is to attach invariants that, if violated, will cause the goal to fail and report back the cause (e. g. having tried something three times without success, or timeouts). The narrative component selects role values to be returned after a goal ends to use them in deciding what to do next. Lastly, it can specify a set of events that will generate a feedback message when they occur (e. g. a character enters the room). Several goals can be active simultaneously in a character; the narrative control can let one have
254
M. L¨ ockelt, E. Pecourt, and N. Pfleger
precedence by assigning it a higher priority. Goals with low priority act as longtime motivations of characters, an example in our scenario would be examining the surroundings to gather evidence when not executing a high-priority goal. 2.4
Reactive Layer: Action / Perception in a Dynamic Environment
As there is no predetermined script and the behavior of human users is often unpredictable, the virtual characters have to deal with a constantly changing environment. This means that every participant who wants to utter a statement or to perform a physical action needs to synchronize its actions with other participants and events that take place in the physical environment. A multimodal generation component translates the abstract dialogue acts from the action planner into sequences of verbal and nonverbal actions (such as gestures, gaze, head-nods, etc.). This process needs to take the immediate situational and conversational context into account. Human participants in a dialogue display signals that permit co-interactors to infer at least parts of the prospected plans with respect to the flow of the interaction [2]. A speaker who wants to hand over the floor indicates this to his audience by displaying turn signals (e. g., completion of a grammatical clause, sociocentric sequences like “you know”, etc.) without any accompanying hand gestures. and wants to take over can respond by indicating a speaker-state signal while already starting to speak. Eventually, the first speaker has to yield the turn. In total, a sequence of three distinct actions has to take place in order to conduct this transition of the turn between speaker and hearer (for details see [7]). Within our framework of autonomous conversational dialogue engines we model these aspects of multiparty interaction by means of a reactive layer that comprises two components: (i) a fusion and discourse engine (FADE) that maintains an extended contextual representation of the ongoing interaction (see [7]) and (ii) a multimodal generation component (MMGEN) that converts the abstract dialogue acts into sequences of suitable verbal and nonverbal actions (see [4]). Reactive behavior depends on an extended notion of conversational context within which individual actions take place. Before a character can exhibit an action, it verifies whether it is feasible to act that way given the situation.
3
Taking the User Character into Account
The inherent autonomy of a human user threatens the outcome of a story, since she should be able to make unexpected dialogue contributions and action at any point. The user interacts with the story through physical actions and dialogue contributions. The virtual characters prefer talking with the user about discoveries to actively searching for them. Also, user contributions must be welcomed by the system and generate an appropriate response, since the user’s immersion experience is the top priority for the system. Since we can not offer perfect ASR, the user needs to be given an intuition about what she can say and expect to be understood. This can be facilitated by letting the characters generate utterances
Balancing Narrative Control and Autonomy for Virtual Characters
255
that could also be parsed if they were spoken by the user. (Recognized speech is classified as dialogue acts from a set defined in the ontology.) We distinguish three principal techniques to make the user contribute to rather than threaten the narrative: (1) Assuming the user is cooperative, the story can guide the user towards sensible actions. This can be done by an offscreen soliloquist speaker giving hints in a manner known from many adventure games, e. g. a comment, or other characters can talk to the user about it. (2) Actions available to the user can be restricted by removing destructive options. Doors can be locked, characters can refuse to talk about certain subjects, physical interactions can be made unavailable. The latter possibility must be used with care, due to its potential to disrupt the perception of agency. (3) The narration engine can instruct characters to rectify unwanted world states (e. g. putting the lights back on), or prevent them (especially if they are irreversible) by manipulating action effects.
4
Conclusion
We outlined how existing storytelling models and components from a multimodal dialogue system (developed in the VirtualHuman project) can be integrated in order to provide an convincing and immersive environment to tell an interactive story. We are currently in the process of finishing the demonstrator for a scenario based on the mystery game Cluedo. An interesting question is how the success in balancing control and autonomy can be quantified in, e. g., a user study. This work is part of the research done in the VirtualHuman and Inscape projects. We thank our project partners for their invaluable contributions.
References 1. Charles, F., Mead, S. and Cavazza, C.: User Intervention in Virtual Interactive Storytelling, Proceedings of the VRIC, Laval, France (2001) 2. Duncan, S.: Some Signals and Rules for Taking Speaking Turns in Conversations. Journal of Personality and Social Psychology, 23(2):283–292 (1972) 3. Hulstijn, J.: Dialogue Games are Recipes for Joint Action, Proceedings of Gotalog Workshop on the Semantics and Pragmatics of Dialogues, Gothenburg (2000) 4. Kempe, B., L¨ ockelt, M. and Pfleger, N.: Generating Verbal and Nonverbal Behavior for Virtual Characters, Proceedings of International Conference for Virtual Storytelling ’05, Strasbourg (2005) 5. Mateas, M.: Interactive Drama, Art, and Artificial Intelligence, PhD Thesis, School of Computer Science, Carnegie Mellon University, Pittsburgh, USA (2002) 6. L¨ ockelt, M.: Action Planning for Virtual Human Performances, Proceedings of the International Conference on Virtual Storytelling ’05, Strasbourg (2005) 7. Pfleger, N.: FADE - An Integrated Approach to Multimodal Fusion and Discourse Processing. In Proceedings of the Doctoral Spotlight Session of the International Conference on Multimodal Interfaces) ICMI’05, State College, PA (2005) 8. Pfleger, N. and L¨ ockelt, M.: Synchronizing Dialogue Contributions of Human Users and Virtual Characters in a Virtual Reality Environment. In: Processing of Interspeech ’05, Lisboa (2005)
Web Content Transformed into Humorous Dialogue-Based TV-Program-Like Content Akiyo Nadamoto1 , Adam Jatowt1 , Masaki Hayashi2 , and Katsumi Tanaka3,1 1
National Institute of Information and Communications Technology, 3-5 Hikaridai Seika-chyo, Kyoto, Japan {nadamoto,adam}@nict.go.jp 2 NHK Science & Technical Research Laboratories, Kinuta, Setagaya-ku, Tokyo, Japan
[email protected] 3 Graduate School of Informatics,Kyoto University, Yoshida Honmachi, Sakyo, Kyoto, Japan
[email protected] Abstract. A browsing system is described for transforming declarative web content into humorous-dialogue TV-program-like content that is presented through character agent animation and synthesized speech. We call this system Web2Talkshow which enables users to obtain web content in a manner similar to watching TV. Web content is transformed into humorous dialogues based on the keywords-set of the web page. By using Web2Talkshow, users will be able to watch and listen to desired web content in an easy, pleasant, and userfriendly way, like watching a comedy show.
1
Introduction
The current web browsing environment typically demands that users engage in active operations such as reading, scrolling, and clicking. That is, to search among the many sources of information on the Internet, we have to use a web browser that requires our active attention and requires us to repeatedly perform many operations. When we are browsing web content, we usually browse in the same way, silently and alone. Why not browse and view web content in a more entertaining way? Furthermore, disabled persons, the elderly, and the young children, often have trouble operating computers, and it is difficult for them to browse and search web content. In contrast, we can passively obtain information from TV by simply watching and listening. This ease of getting information means that we can watch TV programs with our family or friends while talking, relaxing, and laughing together, and we can share the same content. Likewise, disabled persons, the elderly, and the young children can easily obtain information by watching and listening. We also believe that presenting web content in a humorous way, similar to the style of TV comedy shows, can help to develop interest in the Internet. We are now developing a passive-manner (audio-visual) browser that will provide web M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 256–261, 2005. c Springer-Verlag Berlin Heidelberg 2005
Web Content Transformed into Humorous Dialogue-Based TV-Program
257
Fig. 1. Typical display and system architecture of Web2Talkshow
content without requiring the user to perform active operations. We call this system Web2Talkshow. Web2Talkshow transforms web content into TV-program-like content, which is presented using humorous dialogue, character agent animation, and synthesized speech, with the result that the web content resembles a TV program. While there are many kinds of TV programs, we believe that comedy programs provide one of the most entertaining, most interesting, and easiest ways to get and remember new information. In Japan, there is a traditional form of comedy called ”manzai”. Manzai typically consists of two or three comedians participating in a humorous dialogue. It is similar to ”stand-up comedy” in the U.S., or ”xiang sheng” in China. If there are two performers, one is the ”boke”, the clown, and the other is the ”tsukkomi”, the straight man. The boke says things that are stupid, silly, or out of line, sometimes using puns, while the tsukkomi delivers quick, witty, and often harsh responses. We use the manzai style in Web2Talkshow. There have been also some attempts for computational approach to humor[1][2], most of the research has focused on analyzing humorous texts instead of creating humor. Web2Talkshow, however, transforms declarative sentences on the web into humorous dialogues. Figure 1(a) shows a typical Web2Talkshow display. The remainder of this paper is organized as follows: Section 2 explains the basic concept of Web2Talkshow, Section 3 explains the transformation of web page content into humorous dialogue, Finally, we conclude in Section 4.
2
Basic Concept
We formalize Web2Talkshow here to show how it can automatically transform web content into humorous TV-program-like content. Web2Talkshow consists of a pre-processing part, a scenario part, and a direction part. Pre-processing When we create TV-program-like content, we have to set up the type of character agents, studio sets, characters’ properties, and the role of agents. In the pre-processing part, we set up these elements in advance.
258
A. Nadamoto et al.
Scenario The scenario is the main part of the transformation of web content into TVprogram-like content. The scenario consists of an introduction, a body, and a conclusion. The introduction consists of several short dialogues. It includes a greeting and a mention of the theme of the original web content. The body consists of multiple dialogues based on the keywords in the original web content. Each dialogue has semantic sets of lines. The semantics of a dialogue are given by the keywords in the original web content. The conclusion consists of farewell and the final laughing point a pun taken from a pun dictionary that is related to the web page’s theme. Direction The direction part defines the character agent animations, camera views, lighting, and background music. This is particularly important for Web2Talkshow because the TV-program-like content is audio-visual content. Figure 1(b) shows system flow of Web2Talkshow.
3
Transformation of Web Content into Humorous Dialogue
We transform web content into humorous dialogue based on keywords sets. We use the topic structure[3] which consists of subject term and content terms as the keywords sets. In the real world, humorous dialogue is often based on exaggeration, deliberate mistakes or misunderstandings. In this paper, we describe how to create mistake or exaggeration dialogue, as a first step of transformation into humorous dialogue. 3.1
Mistake and Misunderstanding Techniques
In the manzai form of comedy, when the boke says an incorrect word or phrase, the audience generally laughs. There are five types of mistake and misunderstanding techniques. Using different topic structure When a sentence in the original content includes a subject term(si ) and content term(s)(cij ) of a topic structure(ti ), the system applies the different topic structure technique. Content terms cooccur with the subject term; that is, content terms are terms ordinarily used with the subject term. We believe that if the system deliberately uses mistaken topic structure sets consisting of incorrect content term(s) and a subject term to transform the content into a dialogue, the system can present the content in a funnier way. We extract an incorrect topic structure from the page. If the correct topic structures for a page are t1 [= (s1 , c11 ) = (Ichiro, M ariners)] and t2 [= (s2 , c21 ) = (Rangers, T exas)], we can create incorrect topic structure it1 [= (s1 , c21 ) = (Ichiro, T exas)]. The resulting dialogue might take Figure 2(a).
Web Content Transformed into Humorous Dialogue-Based TV-Program
259
Using related word When a sentence includes only a subject word or a content word and it is a common noun, the system applies the related words technique. The system replaces the common noun with a different word that has the same ontology as the common noun by using an ontology dictionary. For example, for the original sentence, ”Yesterday, the baseball player hit a homerun”, the system replaces ”baseball” with ”soccer”, and the dialogue might take Figure 2(b). Using antonym The system changes an adjective that has a dependency relation with a subject term or content term into an antonym. Figure 2(c) shows example of using antonym. Inserting opposite conjunction When two consecutive sentences do not include subject term(s) or content term(s) and there is no conjunction between them, the system inserts a conjunction such as ”but” or ”however” that indicates opposition. Figure 2(d) shows example. Cutting in When the tsukkomi says a subject or content term from an extracted sentence, the boke cuts in and finishes it so that a different word is created with a different meaning. Figure 2(e) shows example. 3.2
Exaggeration Techniques
The boke might exaggerate a word or phrase on purpose to surprise the audiences. We apply three types of exaggeration techniques. Exaggerated number If the sentence includes a number, the system increases or decreases the number substantially. For example, if the sentence is, ”Today, I picked up $1”, the exaggerated dialogue might take Figure 2(f). Exaggerated time expression If the sentence includes a word that specifies time, the system exaggerates the time expression. For example, if the sentence is, ”Yesterday, I watched the Yankees game”, the exaggerated sentence might take ”One hundred years ago, I watched the Yankees game.” Exaggerated adjective or adverb If the sentence includes an adjective or adverb, the system inserts a word or phrase like ”very”, ”many”, ”much”, or ”many times” or it changes the degree of the adjective or adverb. For example, if the sentence is, ”Yesterday, a small bank in Tokyo was robbed”, the exaggerated sentence might take ”Yesterday, the smallest bank in Tokyo was robbed many times.” 3.3
Pre-scenario
It is difficult to completely automatically transform dialogue and obtain a perfect result. Thus, we manually create dialogue frameworks in a pre-scenario file in
260
A. Nadamoto et al.
Fig. 2. Example of each dialogue
XML. The pre-scenario consists of structure tags, content tags, and direction tags. For the Figure 2(a), we would write framework dialogues as Figure 2(g). In this case, ”type” specifies the different topic structure technique, ”key” specifies the who question type, ”num” specifies the variation number of the dialogue, variable $S1 is a subject term of topic 1, $C21 is a content term of topic 2, and $sentence is a sentence in the original content. The system selects the dialogue type from the pre-scenario file depending on the topic structure and sentence type in the original content. If there are several possible transformations, the system applies one randomly.
4
Conclusion
We proposed a new web browsing system called ”Web2Talkshow”, which transforms declarative web content into humorous dialog-based TV-program-like content that is presented through character agent animation and synthesized speech. The transformation is based on topic structure and is done using several transformation techniques. We plan to work on improving the grammar of the transformations, on transforming the content of several pages simultaneously, and on introducing links in the transformed content to reflect the link structure of the web content.
References 1. Mulder, M. P. and Nijholt, A., ”Humor research: State of the Art”, CTIT technical reports series(13813625) vol.02 nr.34, 2002. 2. Ritchie, G. ”Current Directions in Computational Humor”, Artificial Intelligence Review, 16(2), pp.119-135, 2001.
Web Content Transformed into Humorous Dialogue-Based TV-Program
261
3. Akiyo Nadamoto, Ma Qiang, and Katsumi Tanaka, ”Concurrent Browsing of Bilingual Web Sites By Content-Synchronization and Difference-Detection”, Proceedings of the 4th International Conference on Web Information Systems Engineering(WISE2003), pp.189-199, Roma, Italy, Dec 2003
Content Adaptation for Gradual Web Rendering Satoshi Nakamura1, Mitsuru Minakuchi1, and Katsumi Tanaka1,2 1
National Institute of Information and Communications, Japan, 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0289, Japan {gon, mmina}@nict.go.jp http://www2.nict.go.jp/jt/a133/gon/index.html 2 Graduate School of Informatics, Kyoto University, Yoshida Honmachi, Sakyo, Kyoto 606-8501, Japan
[email protected] http://www.dl.kuis.kyoto-u.ac.jp/~tanaka/index_j.html
Abstract. We previously proposed a gradual Web rendering system. This system rendered Web content incrementally according to the context of the user and the environment, enabling casual Web browsing. Unfortunately, it had low levels of readability and enjoyment. In this paper, we describe the problems with it and introduce content adaptation mechanisms to solve these problems.
1 Introduction We can now browse Web pages at any time and from anywhere, by using portable computers or mobile phones, thanks to advances in downsized computer hardware, improved performance, and the popularity of wireless networks. Also, ubiquitous computing and wearable computers are now becoming a reality. Thus, the Web is no longer confined to offices or studies but is instead being widely applied to various scenarios in our daily lives. Expanding the use of the Web in everyday life could also increase the provision of services based on coordination between Web content and external factors, such as the status of the user and the environment. Context-aware information services are an example of possible developments. Existing approaches, however, which assume active utilization on the user’s part, are unsuitable for many daily uses. Therefore, by extending the concept of time-based representation of content, we developed a gradual Web rendering system based on abstract parameters, which could be connected to inputs indicating various status values (e.g., temperature, time, illumination, and energy consumption). This approach enables casual browsing. In gradual Web rendering, the system first acquires the target Web page from the WWW (World Wide Web). Next, it divides the page into several parts and serializes them. It also monitors status values from sensors or input devices and adds them to the content of the page. Finally, it renders the different parts of the Web page. People can change their reading (browsing, rendering) speed by using certain parameters. In addition, we previously implemented the EnergyBrowser [2] and AmbientBrowser [3] systems. EnergyBrowser was designed to encourage the user and makes M. Maybury et al. (Eds.): INTETAIN 2005, LNAI 3814, pp. 262 – 266, 2005. © Springer-Verlag Berlin Heidelberg 2005
Content Adaptation for Gradual Web Rendering
263
exercising more enjoyable and interesting. AmbientBrowser was designed to provide a huge variety of peripheral information via ubiquitous displays in kitchens, bathrooms, bedrooms, studies, and streets. People can thus acquire knowledge by viewing information from ubiquitous displays, with only minimal interaction. We found that the gradual Web rendering system could increase legibility, because it directed the user’s viewpoint so that he or she could read more easily. We therefore concluded that the gradual Web rendering/browsing mechanism would be suitable for everyday use. Unfortunately, we also found problems with the system, as the following summary of user comments reveals. z z z z
It was frustrating for users to have to read/browse unnecessary parts of pages, such as menus, links, and advertisements on news sites. It was difficult for the elderly to read/browse Web pages that were designed with small fonts. Most users became bored with reading/browsing very long texts. Just browsing through image content was not sufficiently interesting because nothing was left to the imagination.
The users’ comments emphasized the importance of adapting Web pages to the gradual Web rendering system. In addition, we concluded that the system should remove unnecessary parts for readers in order to increase readability. The presentation of Web pages is also important. Large fonts and appropriate typefaces are required for legibility. We found that some Web pages that could typographically adjust the pace and volume of speech were very popular with users, because the incremental rendering made it appear as though the speech was actually being spoken. Adapting these content presentation styles to a Web page consisting of long passages of text would increase users’ enjoyment. On the other hand, we should also introduce gradual image rendering mechanisms, because some users indicated that just browsing through image content was not interesting. In this paper, based on these considerations, we introduce a content adaptation mechanism that removes unnecessary parts of Web pages and increases readability by changing the sizes of fonts, images, and tables. In addition, we introduce a simple presentation mechanism that uses the context of text and gradual image rendering to increase users’ enjoyment of content.
2 Content Adaptation 2.1 Removing Non-essential Parts To remove the non-essential parts of a Web page accurately, we normally would have to first analyze the structure of the page and calculate which parts are essential. In a gradual Web browsing system, however, accuracy is less important, because it is not used for searching. The system thus uses prepared patterns and pattern matching to remove non-essential parts. It first loads rules specifying essential and non-essential parts. The rules consist of flags for essential or non-essential content, the names of the rules, the parts of the
264
S. Nakamura, M. Minakuchi, and K. Tanaka
URL required to recognize the target Web content, and the start and end tag patterns (see Fig. 1). The system applies all rules to the target Web content. If the URL pattern for the rule corresponds to the URL for the target content, the system detects essential/nonessential parts by applying the matching rules. If a matched part is non-essential, the system removes it. In general, the system removes everything but matched parts. It gives preference to non-essential rules over essential ones when targeting content. The system then detects non-essential parts by using part of the URL. If there is a URL for an advertisement in area part, it removes this part as an advertisement. In addition, the system counts the number of links in one part, such as a table. If the number of links is above a preset value, it removes this part as a menu. In addition to these processes, other non-essential parts of Web pages, such as frames and banner advertisements, are removed. Users can then read/browse only the target content. Name: Remove Sample Advertisement Flag: Non-essential URL: http://news.sample.site/ Start:
