Facets of Virtual Environments: First International Conference, FaVE 2009, Berlin, Germany, July 27-29, 2009, Revised Selected Papers

Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering Editorial Bo...

Author: Fritz Lehmann-grube | Jan Sablatnig

58 downloads 1112 Views 12MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering Editorial Board Ozgur Akan Middle East Technical University, Ankara, Turkey Paolo Bellavista University of Bologna, Italy Jiannong Cao Hong Kong Polytechnic University, Hong Kong Falko Dressler University of Erlangen, Germany Domenico Ferrari Università Cattolica Piacenza, Italy Mario Gerla UCLA, USA Hisashi Kobayashi Princeton University, USA Sergio Palazzo University of Catania, Italy Sartaj Sahni University of Florida, USA Xuemin (Sherman) Shen University of Waterloo, Canada Mircea Stan University of Virginia, USA Jia Xiaohua City University of Hong Kong, Hong Kong Albert Zomaya University of Sydney, Australia Geoffrey Coulson Lancaster University, UK

33

Fritz Lehmann-Grube Jan Sablatnig (Eds.)

Facets of Virtual Environments First International Conference, FaVE 2009 Berlin, Germany, July 27-29, 2009 Revised Selected Papers

13

Volume Editors Fritz Lehmann-Grube Technische Universität Berlin Center for Multimedia in Education and Research (MuLF) Straße des 17. Juni 136 10623 Berlin, Germany E-mail: [email protected] Jan Sablatnig Technische Universität Berlin Institute of Mathematics Straße des 17. Juni 136 10623 Berlin, Germany E-mail: [email protected]

Library of Congress Control Number: 2009943510 CR Subject Classification (1998): K.8, I.2.1, K.4.2, K.3, J.4, I.3.7 ISSN ISBN-10 ISBN-13

1867-8211 3-642-11742-2 Springer Berlin Heidelberg New York 978-3-642-11742-8 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © ICST Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering 2010 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12985821 06/3180 543210

Preface

In recent years, the popularity of virtual worlds has increased significantly and they have consequently come under closer academic scrutiny. Papers about virtual worlds are typically published at conferences or in journals that specialize in something entirely different, related to some secondary aspect of the research. Thus a paper discussing legal aspects of virtual worlds may be published in a law journal, while a psychologist's analysis of situation awareness may appear at a psychology conference. The downside of this is that if you publish a virtual worlds paper at an unrelated conference in this manner you are likely to be one of only a handful of attendees working in the area. You will not, therefore, achieve the most important goal of attending conferences: meeting and conversing with like-minded colleagues from the academic community of your field of study. Virtual worlds touch on many well-established themes in other areas of science. Researchers from all these fields will therefore be looking at this new, interesting, and growing field. However, to do effective research related to these complex constructs, researchers need to take into account many of the other facets from other fields that impact virtual worlds. Only by being familiar with and paying attention to all these different aspects can virtual worlds be properly understood. We therefore believe that the study of virtual worlds has become a research field in its own right. To date, this research field can claim only a relatively small community, because interested researchers from more established fields largely keep to themselves. FaVE was born to change that. We wanted to start creating a multidisciplinary community of academic researchers all interested in virtual worlds and their applications; and we wanted everyone to talk to each other, regardless of their original field, because we do believe that every one of these researchers has something to say that will be of interest to the rest. After much organizational work and with lots of help from collaborators all over the world (and of course some sleepless nights), the conference was finally held during July 27–29, 2009. The tracks and sessions were organized with our multidisciplinary goal in mind: that is, we attempted to create sessions with a combination of presenters who are working on similar subjects, albeit perhaps coming from different angles. Over the course of the conference, our attendees did indeed see the advantages of the format. By the end of the conference, there were vivid and vibrant discussions going on, bringing all the diverse viewpoints to the table––surprisingly similar in some cases and surprisingly different in others. The first set of papers presented at the conference talked about the application of virtual worlds to science, both for research and for education. Virtual worlds are seen as a means to solve problems that have been known to science for a while, but which are expected to become more pronounced in the near future––such as data visualization and extending the reach of scientific teaching. The following papers were presented:

VI

Preface

• “Exploring the Use of Virtual Worlds as a Scientific Research Platform: The Meta-Institute for Computational Astrophysics (MICA)” by S. G. Djorgovski, P. Hut, S. McMillan, E. Vesperini, R. Knop, W. Farr, M. J. Graham • “Dual Reality: Merging the Real and Virtual” by Joshua Lifton and Joseph A. Paradiso • “Development of Virtual Geographic Environments and Geography Research” by Fengru Huang, Hui Lin, Bin Chen The next few papers addressed how people behave and react in existing virtual worlds. This not only characterized how people move and navigate, but also included very tangible advice on how one might improve the usability and acceptance of virtual worlds, such as by adding landmarks and improving the virtual weather. These papers comprised: • • •

“Landmarks and Time-Pressure in Virtual Navigation: Towards Designing Gender-Neutral Virtual Environments” by Elena Gavrielidou and Maarten H. Lamers “Characterizing Mobility and Contact Networks in Virtual Worlds” by Felipe Machado, Matheus Santos, Virgilio Almeida, and Dorgival Guedes “The Effects of Virtual Weather on Presence” by Bartholomäus Wissmath, David Weibel, Fred W. Mast

Next, we took a look at what can be done to make virtual worlds easier to use for the end user. This ranged from a shop assistant who attempts to understand typed speech, through a visualization plug-in architecture, to an analysis of current virtual worlds' Terms of Service and how those may be improved. The papers here were: • • •

“The Role of Semantics in Next-Generation Online Virtual World-Based Retail Store” by Geetika Sharma, C. Anantaram, and Hiranmay Ghosh “Complexity of Virtual Worlds' Terms of Service” by Holger M. Kienle, Andreas Lober, Crina A. Vasiliu, Hausi A. Müller “StellarSim: A Plug-in Architecture for Scientific Visualizations in Virtual Worlds” by Amy Henckel and Cristina V. Lopes

We subsequently discussed the theory and practice of collaboration in virtual worlds. A formal description of virtual world collaboration was developed that may be used to describe workflow in a virtual world setting. Also, an actual workflow was studied experimentally and some requirements for characters controlled by artificial intelligences in interacting efficiently with human users were set out. The papers were: • “Formalizing and Promoting Collaboration in 3D Virtual Environments - A Blueprint for the Creation of Group Interaction Patterns” by Andreas Schmeil and Martin J. Eppler • “Usability Issues of an Augmented Virtuality Environment for Design” by Xiangyu Wang and Irene Rui Chen • “Conceptual Design Scheme for Virtual Characters” by Gino Brunetti and Rocco Servidio

Preface

VII

Finally, we focused on the social aspects of using virtual worlds. While in traditional media the media produces content and consumers consume it, these lines are blurred in virtual worlds. This touches on many important questions such as ownership and rights. Does a user of a virtual world even have rights? The mixing of play and work that is becoming noticeable in many virtual worlds was also explored. The Papers were: • “The Managed Hearthstone: Labor and Emotional Work in the Online Community of World of Warcraft” by Andras Lukacs, David Embrick, and Talmadge Wright • “Human Rights and Private Ordering in Virtual Worlds” by Olivier Oosterbaan • “Investigating the Concept of Consumers as Producers in Virtual Worlds: Looking Through Social, Technical, Economic, and Legal Lenses” by Holger M. Kienle, Andreas Lober, Crina A. Vasiliu, Hausi A. Müller The papers are an interesting read and we hope that you take the time to peruse a few that may not be quite in your area of research.

Organization

Steering Committee Imrich Chlamtac Sabine Cikic Viktor Mayer-Schönberger

Create-Net, Italy Technische Universität Berlin, Germany Harvard University, USA

General Conference Chair Richard A. Bartle

University of Essex, UK

General Conference Vice Chair Sven Grottke

University of Stuttgart, Germany

Technical Program Chair Jan Sablatnig

Technische Universität Berlin, Germany

Workshops Chair Fritz Lehmann-Grube

Panels Chair Julian R. Kücklich

University of Arts London, UK

Local Arrangements Chair Sabine Cikic


Publicity Chair Sebastian Deterding

Publications Chair Fritz Lehmann-Grube

Utrecht University, The Netherlands

X

Organization

Web Chair Sharon Boensch


Sponsorship Chair Sabina Jeschke

University of Stuttgart, Germany

Conference Coordinator Gabriella Magyar

ICST

Program Committee Katharina-Maria Behr Anja Beyer Sabine Cikic Julian Dibbell Sebastian Deterding Martin Dodge Sean Duncan David England James Grimmelmann Sven Grottke Shun-Yun Hu Jesper Juul Fritz Lehmann-Grube Andreas Lober Claudia Loroff Viktor Meyer-Schönberger Claudia Müller Heike Pethe Thomas Richter Albert 'Skip' Rizzo Jan Sablatnig Uwe Sinha Matthew Sorell Marc Swerts Anton van den Hengel Xiangyu Wang Marc Wilke Leticia Wilke Theodor G. Wyeld Tal Zarsky

Hamburg Media School, Germany Ilmenau University of Technology, Germany Technische Universität Berlin, Germany Utrecht University, The Netherlands University of Manchester, UK University of Wisoconsin-Madison, USA Liverpool John Moores University, UK New York Law School, USA University of Stuttgart, Germany National Central University Taiwan Singapore-MIT GAMBIT Game Lab, Singapore Technische Universität Berlin, Germany Schulte Riesenkampff, Lawyers Institut für Innovation und Technik, Germany Harvard University, USA University of Stuttgart, Germany University of Amsterdam, The Netherlands University of Stuttgart, Germany University of Southern California, USA Technische Universität Berlin, Germany Technische Universität Berlin, Germany University of Adelaide, Australia Tilburg University, The Netherlands Australian Centre for Visual Technologies, Australia The University of Sydney, Australia University of Stuttgart, Germany University of Stuttgart, Germany Flinders University Adelaide, Australia University of Haifa, Israel

Table of Contents

FaVE 2009 – Track 1 Development of Virtual Geographic Environments and Geography Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fengru Huang, Hui Lin, and Bin Chen Dual Reality: Merging the Real and Virtual . . . . . . . . . . . . . . . . . . . . . . . . . Joshua Lifton and Joseph A. Paradiso Exploring the Use of Virtual Worlds as a Scientific Research Platform: The Meta-Institute for Computational Astrophysics (MICA) . . . . . . . . . . S. George Djorgovski, Piet Hut, Steve McMillan, Enrico Vesperini, Rob Knop, Will Farr, and Matthew J. Graham

1 12

29

FaVE 2009 – Track 2 Characterizing Mobility and Contact Networks in Virtual Worlds . . . . . . Felipe Machado, Matheus Santos, Virg´ılio Almeida, and Dorgival Guedes Landmarks and Time-Pressure in Virtual Navigation: Towards Designing Gender-Neutral Virtual Environments . . . . . . . . . . . . . . . . . . . . . Elena Gavrielidou and Maarten H. Lamers The Effects of Virtual Weather on Presence . . . . . . . . . . . . . . . . . . . . . . . . . Bartholom¨ aus Wissmath, David Weibel, and Fred W. Mast

44

60 68

FaVE 2009 – Track 3 Complexity of Virtual Worlds’ Terms of Service . . . . . . . . . . . . . . . . . . . . . . Holger M. Kienle, Andreas Lober, Crina A. Vasiliu, and Hausi A. M¨ uller

79

The Role of Semantics in Next-Generation Online Virtual World-Based Retail Store . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Geetika Sharma, C. Anantaram, and Hiranmay Ghosh

91

StellarSim: A Plug-In Architecture for Scientific Visualizations in Virtual Worlds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Amy Henckel and Cristina V. Lopes

106

XII

Table of Contents

FaVE 2009 – Track 4 Formalizing and Promoting Collaboration in 3D Virtual Environments – A Blueprint for the Creation of Group Interaction Patterns . . . . . . . . . . Andreas Schmeil and Martin J. Eppler

121

Conceptual Design Scheme for Virtual Characters . . . . . . . . . . . . . . . . . . . . Gino Brunetti and Rocco Servidio

135

Usability Issues of an Augmented Virtuality Environment for Design . . . Xiangyu Wang and Irene Rui Chen

151

FaVE 2009 – Track 5 The Managed Hearthstone: Labor and Emotional Work in the Online Community of World of Warcraft . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Andras Lukacs, David G. Embrick, and Talmadge Wright Human Rights and Private Ordering in Virtual Worlds . . . . . . . . . . . . . . . Olivier Oosterbaan

165 178

Investigating the Concept of Consumers as Producers in Virtual Worlds: Looking through Social, Technical, Economic, and Legal Lenses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Holger M. Kienle, Andreas Lober, Crina A. Vasiliu, and Hausi A. M¨ uller

187

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

203

Development of Virtual Geographic Environments and Geography Research Fengru Huang1, Hui Lin1, and Bin Chen2 1

Institute of Space and Earth Information Science, Chinese University of Hong Kong, Shatin, N.T., Hong Kong 2 Institute of Remote Sensing and Geographic Information System, Peking University, Beijing, China {huangfengru,huilin}@cuhk.edu.hk, [email protected]

Abstract. Geographic environment is a combination of natural and cultural environments under which humans survive. Virtual Geographic Environment (VGE) is a new multi-disciplinary initiative that links geosciences, geographic information sciences and information technologies. A VGE is a virtual representation of the natural world that enables a person to explore and interact with vast amounts of natural and cultural information on the physical and cultural environment in cyberspace. Virtual Geography and Experimental Geography are the two closest fields that associate with the development of VGE from the perspective of geography. This paper discusses the background of VGE, introduces its research progress, and addresses key issues of VGE research and the significance for geography research from Experimental Geography and Virtual Geography. VGE can be an extended research object for the research of Virtual Geography and enrich the contents of future geography, while VGE can also be an extended research method for Experimental Geography that geographers can operate virtual geographic experiments based on VGE platforms. Keywords: Virtual Environment, Virtual Geography, Experimental Geography, Virtual Geographic Experiment.

1 Introduction Geographic environment is a combination of natural and cultural environments under which humans survive, and traditional geography takes geographic environments in the real world as its study object. Geography aims to study the physical, chemical, biological and human processes of the geographic environment (the earth surface system), analyze the relationships between the interfaces of each geo-spheres, and interaction mechanisms between various natural and human processes, thus to explore the precepts of coordinative and sustainable development of resources, environments and human activities. As the development of information technologies such as Internet, Web and Virtual Reality goes further, both new opportunities and challenges are generated for the development of geographic information sciences and technologies, as well as for geography sciences. Virtual Geographic Environment (VGE) was first proposed in F. Lehmann-Grube and J. Sablatnig (Eds.): FaVE 2009, LNICST 33, pp. 1–11, 2010. © Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering 2010

2

F. Huang, H. Lin, and B. Chen

early 2000 by geography and geographic information science researchers [1, 2, 3, 4]. VGE is a new multi-disciplinary initiative that links geosciences, geographic information sciences and information technologies. A VGE is a virtual representation of the natural world that enables a person to explore and interact with vast amounts of natural and cultural information on the physical and cultural environment, in cyberspace. From the perspective of geography, VGE is an environment concerned with the relationship between avatar-based humans and 3-dimension (3D) virtual worlds. From the perspective of information systems, VGE is an advanced information system that combines GIS (Geographic Information System) with VR technology [1, 2, 3]. At present, there has launched much research into VGE theory, technology and applications [5, 6, 7, 8]. Those works focus on different aspects of VGE research and thus raise broader and more complicated research such as topics on geo-data, geo-models, geosciences knowledge acquisition, GeoComputation, geo-visualization, geocollaboration, interaction mode, virtual geographic experiments and Virtual Geography. To address this, this paper aims to discuss the background of VGE, introduce its research progress, and address key issues on VGE research and the significance for geography research from the perspectives of Experimental Geography and Virtual Geography. This paper is organized as follows. In section 2, we discuss background and research progress of VGE, as well as its research contents and key issues. In section 3, we present revolution of geography research method and geographic language. Section 4 and Section 5 discuss development of Virtual Geography and development of Experimental Geography, respectively. Section 6 contains some final discussion and remarks on VGE and geography research.

2 Background and Research Progress of VGE 2.1 What Is VGE? VGE was first proposed as a concept of a virtual world that was referenced to the real world, which had five types of space, namely Internet space, data space, 3D graphical space, personal perceptual and cognitive space, and social space [2]. To this concept regard, there are three stages in the evolutionary process of a VGE: virtual crowds, virtual villages and virtual cities. In this sense, VGE research focuses on the differences and extension of life content and life style from the real world to virtual worlds, or between the real world and a virtual world, and thus relate to research of Virtual Geography or other terms alike. To make emphasis on representation of geographic process and phenomena in the real world, such as visualization and simulation of geomodals in diverse geosciences, the concept of VGE has been supplemented as a new generation of information platform that can be used for geo-phenomena representation and simulation, and geo-knowledge publishing and sharing [9]. Such a VGE represents an ideal interface of geo-information scientists for geographic representation and research, that is ‘immersive experience and beyond the understanding of reality’. VGE systems have five characteristics:

Development of Virtual Geographic Environments and Geography Research

3

1, Integrated management and interoperation on geo-models and GIS data; 2, Multi-dimension geo-visualization, including visualization of geometric models (represent static objects) and geo-models (represent dynamic geographic processes); 3, Immersive virtual interaction: users can ‘step’ into the virtual geographic world and be a part of the environment, thus have an immersive interaction with the virtual environment. 4, Distributed geo-collaboration: geographic experts from different places/locations of the real world can carry out professional discussion and decision-making with the support of VGE platform; 5, Public participation: VGE emphasizes on the role of social public participation, so the users are not just experts and professional users, but also the general public. 2.2 Why VGE Rising? The rising of VGE has a profound background that includes not only development of geographic sciences, but also currently rapid development of computer technology, information technology and social sciences. The development of VGE is closely related to the development of Earth System Science and will ultimately serve the research of global environment change and human sustainable development. 1, Earth System Science research needs a new research tool and information platform in which scientific computation and virtual representation are the two important characteristics, to facilitate simulation and prediction on natural complex phenomena that can not be experimented in the real world conditions, such as prediction on the whole cycle of the Earth's atmosphere-ocean, global warming, Earth's crust change, earthquake occurrence, and human behavior simulation in emergency public accident or natural disasters, so as to help manage on environmental resources and human activities to achieve sustainable development. 2, Current rapid development of Earth information technologies provides technical support for the emergence of VGE. As the development of mathematical scientific methods (for example, scientific computation, cellular automation, fractal geometry, fuzzy mathematics, etc), and computer science and technologies (such as computer communication, networks, databases, distributed computing, artificial intelligence, human-computer interaction and virtual reality) goes further and is being applied to geographic science and Earth System Science, there has been continuous development from different angles in the field of Earth information technologies. This provides support for the rising and development of VGE, which integrates with Remote Sensing (RS), Global Position System (GPS), Geographic Information System (GIS), computer network, virtual reality technology, and other computer technologies. 3, The field of social and cultural sciences require a research platform or a window like VGE to learn about human development trends in the age of post-modernism. The style of post-modern society has the basic characteristics as "information age", "knowledge economy" and "learning society", and has actually penetrated into various aspects of contemporary human society, quickly and fully. In recent years, geography research activities and literature have been increasing with regard to the impact of modern information technology on geography. For example, Batty [10, 8] proposed "invisible cities", "Cyberspace Geography" and "Virtual Geography" in terms of geographic space–place, espace, cyberspace, and cyberplace. Increasing public is being

4


familiar with and a part of virtual environments, virtual earth, or virtual worlds. The new styles of learning, working and living, such as e-tourism, e-education, eshopping, virtual communities, virtual office, virtual banking, virtual stock market, virtual games, and virtual art appear in succession and show a strong vitality, and may represent human development trends and directions in the post-modern age. Therefore, from the perspective of social scientists who study socio-economic, political, legal, cultural, and human psychology, behavior and life styles of the post-modern age, something like VGE as a research window is needed to help to explore characteristics and development trends of human society of the post-modern age. 2.3 Related Work VGE is developed with the support of the advancement in computer science and technologies, geosciences, Geographic Information science and techniques. Only by combination of those theories and technologies to construct an integrated platform can we meet the need of the development of Earth System Science for global environmental change and sustainable development research. In recent years, much progress has been made on such a next-generation geographic information platform from different aspects. Chinese scholars have been engaging actively in relevant research since VGE was put forward a decade ago. Lin and Gong explored basic theory, technology and application of VGE through a series of academic work and papers [1, 2, 3, 4, 9, 11]. Tang et al. studied on visual geographic modeling and construction of VGE [12]. Researchers in the Electronic Visualization Laboratory (EVL) of The University of Illinois have focused on the development of tools, techniques and hardware to support real-time and highly interactive visualization [13], and the platform GeoWall [14] was developed with the characteristics of users’ immersive interaction with the virtual environment which was displayed to the big screen. MacEachren developed a system named Dialogue Assisted Visual Environment for Geoinformation (DAVE_G), in which the earlier multi-modal interface framework and two test-bed implementations: iMap and XISM [15] were built on and extended. Batty, M. established virtual city and explored Virtual Geography [8, 10, 16]. Yano built Virtual Kyoto through 4DGIS and Virtual Reality to show social customs and traditional culture in Japan [17]. Google, Microsoft, Linden Lab and other companies started to build community, city, region, or even global 3D virtual environments. Google developed Google Earth for public searching the high resolution digital map freely [18], and Google SketchUp [19] for 3D models building. Microsoft launched Virtual Earth project, which was built up by using photos and offered a higher sense of reality [20]. Linden Lab created and opened Second Life® to the public since 2003, and now it owns the largest amount of virtual residents and many kinds of applications such as virtual meeting, virtual class, virtual industry, etc., in its virtual world [21]. As one of the approaches of VGE application construction, some GIS-based multi-user virtual environment applications are being carrying out based on virtual world platforms such as Second Life®, OpenSimulator [22] or other similar projects. We can therefore see that, as a new generation of geographic information platform, VGE development has a broad prospect for geography research.


5

2.4 Research Contents and Key Issues of VGE In contrast to current data-centered GIS, a VGE is a human-centered environment. A VGE system can present immersive multi-dimension visualization, support multi-user collaborative work, and provide a natural way of perception and interaction between avatars or users, or between users and virtual environments. Thus, VGE can be an integrative innovation and its research contents may involve multi-discipline issues, such as geo-modeling, geographic simulation, GeoComputation, geo-visualization, computer network, geo-collaboration and interaction, geo-knowledge discovery and sharing, and virtual geographic experiments. Those are as well as the key issues of VGE research. On the other hand, VGE extends the research range of traditional geography with virtual extended geographic environments. Thus, the research contents of geography extend from place and space of real geographic environment to placespace and relationship in virtual environments or interaction between those two. This paper will discuss further on two extended research fields: Virtual Geography and Experimental Geography in the subsequent sections.

3 Revolution of Geography Research Methods and Geographic Languages There has always been a thread of research thoughts of "Pattern - Structure - Process -Mechanism" throughout geography studies. However, the research methods in traditional physical geography are mostly field-site inspection and the use of maps and data analysis. Geographer Baranshiy once said, "Map is the second language of geography". Using maps for thinking and analyzing is the most important research method that makes geography different from other subjects. Development of GIS is based on a combination of map, mathematical methods, and modern information technologies. To date, GIS has become the most common carrier and platform of geographic information. Chen argued "GIS is the third-generation language of geography" [23]. Along with constant improvement of ability and means to access digital spatial data and expansion of GIS applications, limitation of traditional GIS (map-centered and data-driven mechanism) has hindered the development of new methods in the field of geographic information representation and services. Virtual reality technology can be used as an immersive human-computer interface for 3D visualization, collaborative work and group decision making through integration with traditional GIS and 3D GIS. Thus, development of VGE can be seen as a higher level of GIS that integrates traditional GIS, virtual reality, network technology, geo-models, humancomputer interaction technology, and systematic methods. Lin argued VGE can be a new generation of geographic language in that VGE had the ability of abstract expression of multi-dimensional, multi-viewpoint, multiple details of multi-model visualization, supporting for a variety of natural interaction and multi-spatial cognition [4, 11]. Fig. 1 shows the developing process from map and GIS to VGE.

6


Field Survey and Mapping Geographic Science Earth System Science

Mathematical Methods Map Digitization

Computer Technologies

Map Systematic Methods

GIS 2D GIS

Network Technology

GIS visualization (2D->2.5D)

Virtual Reality Virtual Map

3D Virtual GIS

Network Based GIS

Geo-informatic Tupu Geo-spatial Cognition Geo-graphical Thinking Geo-Knowledge Reasoning ……

Distributed Computing Grid Computing

Collaborative Information Sysem

Distributed Collaborative 3D Virtual GIS Geographic Models and Geometry Spatial Database Integration

VGE

Fig. 1. Process from map and GIS to VGE

4 Development of Virtual Geography 4.1 VGE Extends Geographic Environment in the Real World Geography is the science of place and space [24]. Traditional geography focuses on place and space of geographic environment in the real world. However, information science and technology provide open and distributed environments like VGE in the Internet or in other cyberspace. In those information worlds, the importance of geographic distance and place has gradually decreased [2]. Online communities or virtual companies exist in cyberspace with virtual places in virtual environments, but with their locations at “elsewhere” or even nowhere in the real world. Thus, space-place becomes virtual space-place and this leads to a deep thinking and wide discussion for geographers in the context of future geography [25, 26, 27, 28]. Geography Research has extended from traditional geographic environment to virtual geographic environments that Virtual Geography focuses on.


7

4.2 Virtual Geography Virtual geography, cyber geography, and imagine geography are all the similar terms in the present literatures that show the impacts of modern technology on geography [2]. Batty proposed virtual geography and focused on the relationship and interaction between cyberspace and the real world, and argued that the boundary between space and place in cyberspace turned blurred, while Crang et al. examined virtual geography mainly from the aspect of complicated social relationships in virtual environments. Lin and Gong [1, 2] argued that virtual geography was a new dimension of geography studying the characteristics and laws involving VGE, and the relationship and interaction between VGE and real geographic environments. In comparison to traditional geography, research contents of this new initiative of geography may include: 1, cybercartography: this is to study the principles and methodology of cybermapping. 2, Development, planning and building of 3D virtual worlds. 3, Spatial perception, cognition and behavior of post-human in 3D virtual environments. 4, Issues in the evolution process of VGE, such as boundary and relationship among various 3D virtual worlds, mechanism of driving forces of evolution of VGE, etc. 5, Relationship and interaction between VGE and real geographic environments in population, landscape, social, political, and economic structures.

5 Development of Experimental Geography 5.1 Experimental Geography Experiment is an important feature as well as a symbol of development of modern science. That means a scientific experiment can be repeated and be verified. Experience, observation, practice, and experiment are of great importance in geography research. From 1950-1960, Chinese geographers have come to realize the importance of experiments for scientific theories and methods of geography development. Huang Bingwei, a modern Geography pioneer of China, pointed out that, the old methods such as empirical and descriptive study in geography research were inanimate, and Experimental Geography was a major development direction of the forward-looking geography [29]. Experimental Geography applies specific experimental ideas, experimental methods, observation equipments and instruments to learn about the spatial structure, time series, human-earth relationship of geographic environment, discover the basic law of geography information accumulation and provide evidence to form a measurable, comparable, controllable geographic system. Therefore, experimental design and experimental execution theories and methods together constitute the research contents of Experimental Geography. The purpose of all experimental work is to identify geographical relations through accessing geographic information by the means of an extension of human senses.

8


Traditional methods used in Experimental Geography include field experiments and indoor physical modeling and experiments. However, those traditional experimental methods show much limitation when its processed object, geographic system, is a complex giant system with geographic issues of multi-dimension, multi-scale, ambiguity and uncertainty. At present, geographic mathematical modeling, remote sensing information modeling and computer simulation calculation and experimental methods are various and complicated, thus, an organic integration of those modern methods from the perspective of Experimental Geography is needed to be achieved. 5.2 Virtual Geographic Experiments for Experimental Geography Virtual experiments are defined as digital and virtual environments to carry out scientific experiments with the support of computer and network technologies. As development of information technology and simulation technology goes further, currently, virtual experiments are applied to a large number of research areas, including biology, chemistry, physics, human motion, and manufacturing, and has become a hot issue in those research fields. However, virtual experiment applications in geosciences are relatively few due to the giant system and highly complex nature of geographic environment. In recent years, as development of VGE and related research that has been carried out, as well as learning from virtual experiment applications in experimental economics, experimental medicine and other areas, virtual geographic experiment has gradually formed a new direction of research methods for Experimental Geography. 5.3 VGE as a Virtual Geographic Experiment Platform We argue that VGE, a virtual geographic world, can be a virtual laboratory in which Virtual Geographic Experiment can be carried out. Virtual Geographic experiment aims to establish and visualize geographic models to verify and represent geographic phenomena and processes by calculation, simulation, visualization, real-time human participation, interaction and manipulation based on geosciences data. It may correspond to geographic positioning field experiments, or indoor physical modeling experiments. It may also be some virtually constructed experiments based on specific geographic features, phenomena and laws that are difficult to be carried out as physical experiments in the real world. Virtual Geographic Experiment can be widely used not only in traditional experimental geography focused research areas of physical geography, but also in economic geography and human geography as a major research method. With the support of such an integration platform of interactive and collaborative work and geographic experimental environment provided by VGE, geographers can analyze the represented geographic phenomena and processes and carry out joint research, knowledge discovery, communication and decision-making in its immersive way. Thus, VGE extends the research methods of Experimental Geography (Fig. 2).


Methods of Experimental Geography

9

Virtual Geographic Environment Virtual geographic experiment

Field investigate

Geo-knowledge discovery and sharing

Field observation and survey

Geo-collaboration interaction Computer simulation experiment Interior experiment and analysis Interior physical simulation experiment

Remote sensing information modeling

and

Multi-D geo-visualization Geo-system Simulation Scientific geo-computation

Mathematical Geographic Modeling

Geo-modeling

Fig. 2. VGE extends the research methods of Experimental Geography

6 Discussion and Conclusion In recent years, multi-user virtual environments have come into widespread use on the Internet. Virtual environment technologies and virtual world platforms (e.g. the classical virtual world "Second Life"®) are used not only for games but also for various non-game purpose applications [30]. Moreover, Roush argued that the World Wide Web will soon be absorbed into the World Wide Sim: an immersive, 3D visual environment combining elements of social virtual worlds ( e.g. Second Life®) and mapping applications (e.g. Google Earth), and what’s coming is a larger digital environment-a 3D Internet [31]. Many relevant issues are being developed or need to be developed to explore both on theory, technology and various applications on those subjects. VGE combines elements of all these technologies and research on relevant frontier issues from the perspective of geography. However, current VGE research focuses more on geometry modeling and visualization or realistic representation that inherits and extends from 2D GIS functionalities, there are limitations with VGE but are important aspects of VGE are dynamic geographic processes modeling and visualization, geo-collaboration, interaction under a 3D virtual environments that support for the capability of people to better understand the real geographic environment. Virtual Geography and Experimental Geography are two closest fields that associate with the development of VGE. Virtual Geography has VGEs as its research object and extend geographic issues from traditional geographic environment to virtual environments and the spaces, places, avatars, and all the other elements and relations in it. Experimental Geography might have VGE as a new medium to establish virtual experiments on geographic processes with a way of immersive visualization, geocollaboration and natural interaction. Development of VGE represents a new field in

10


geographic information and geographic research in the coming 3D Internet age. Much work should be developed from different aspects of this new field.

Acknowledgements This research is partially supported by The National “863”High Technology Research and Development Program of China (No. 2006AA12Z207, 2007AA120502), and Direct Grant from CUHK (No. 2020967). We would also like to thank the three anonymous reviewers for their valuable suggestions on previous version of this paper.

References [1] Gong, J., Lin, H.: Virtual Geographic Environments—A Geographic Perspective on Online Virtual Reality. High Education Press, Beijing (2001) [2] Lin, H., Gong, J.: Exploring Virtual Geographic Environments. Geographic Information Sciences 7(1), 1–7 (2001) [3] Lin, H., Gong, J.: On Virtual Geographic Environments. Acta Geodaetica et Cartographica Sinica 31(1), 1–6 (2002) [4] Lin, H., Gong, J., Shi, J.: From Maps to GIS and VGE-A Discussion on the Evolution of the Geographic Language. Geography and Geo-Information Science 19(4), 18–23 (2003) [5] Jiulin, S.: An Exploration of Virtual Recreation Environment on Resources and Environment Sciences. Resources Science 21(1), 1–8 (1999) [6] Jun, G., Yunjun, X., Xiong, Y.: Application of Virtual Reality in Terrain Environment Simulation. People’s Liberation Army Press, Beijing [7] Dykes, J., Moore, K., Wood, J.: Virtual environments for student field work using network components. International Journal of Geographical Information Science 13(4), 397– 416 (1999) [8] Batty, M., Smith, A.: Virtuality and Cities: Definitions, Geographies, Designs. In: Fisher, P.F., Unwin, D.B. (eds.) Virtual Reality in Geography, pp. 270–291. Taylor and Francis, Abington (2002) [9] Lin, H., Xu, B.: Some Thoughts on Virtual Geographic Environments. Geography and Geo-Information Science 23(2), 1–7 (2007) [10] Batty, M.: Virtual Geography. Future 29(4/5), 337–352 (1997) [11] Lin, H., Zhu, Q.: The Linguistic Characteristica of Virtual Geographic Environments. Journal of Remote Sensing 9(2), 158–165 (2005) [12] Tang, W., Lv, G., Wen, Y., et al.: Study of Visual Geographic Modeling Framework for Virtual Geographic Environment. Geo-information Science 9(2), 78–84 (2007) [13] Jeong, B., Renambot, L., Jagodic, R., Singh, R., Aguilera, J., Johnson, A., Leigh, J.: High-Performance Dynamic Graphics Streaming for Scalable Adaptive Graphics Environment. In: Proceedings of SC 2006, Tampa, FL, November 11-17 (2006) [14] Johnson, A., Leigh, J., Morin, P., Van Keken, P.: GeoWall: Stereoscopic Visualization for Geoscience Research and Education. IEEE Computer Graphics and Applications (11/01/2006 - 12/31/2006) [15] MacEachren, A., Cai, G., Sharma, R., Rauschert, I., Brewer, I., Bolelli, L., Shaparenko, B., Fuhrmann, S., Wang, H.: Enabling collaborative Geoinformation access and decisionmaking through a natural, multimodal interface. International Journal of Geographical Information Science 19(3), 293–317 (2005)


11

[16] Smith, H., Evanss, Batty, M.: Building the virtual city: Public participation through edemocracy. Knowledge, Technology & Policy 18(1), 62–85 (2005) [17] Keiji, Y.: Virtual Kyoto through 4D-GIS and Virtual Reality, http://www.ritsumei.ac.jp/eng/newsletter/winter2006/gis.shtml [18] http://www.Earth.google.com [19] http://www.sketchup.google.com [20] http://www.preview.local.live.com [21] http://www.secondlife.com [22] http://www.opensimulator.org [23] Chen, S.: Geographic Information System Exploration and Experiments. Scientia Geographica Sinica 3(4), 287–302 (1983) [24] AAG, What is a geography (2001), http://www.aag.org/ [25] Couclelis, H.: he Death of Distance. Environment and Planning B: Planning and Design 23, 387–398 (1996) [26] NCGIA, Project Varenius (1998), http://www.ncgis.ucsb.edu/varenius/ [27] Crang, M., Crang, P., May, J.: Introduction in Virtual Geography: Bodies, Space, and Relations. In: Crang, M., Crang, P., May, J. (eds.), pp. 1–20. Routledge, London (1999) [28] Dodge, M.: Cybergeography. Environment and Planning B: Planning and Design 28, 1–2 (2001) [29] Tang, D.: Experimental Geography and Geographical Engineering. Geographical Research 16(1), 1–10 (1997) [30] Quinn, B.: Immersive 3D Simulator-based GIS. Bay Area Automated Mapping Association, 3–16 (2009) [31] Roush, W.: Second Earth. Technology Review 7/8, 39–48 (2007)

Dual Reality: Merging the Real and Virtual Joshua Lifton and Joseph A. Paradiso MIT Media Lab

Abstract. This paper proposes the convergence of sensor networks and virtual worlds not only as a possible solution to their respective limitations, but also as the beginning of a new creative medium. In such a “dual reality,” both real and virtual worlds are complete unto themselves, but also enhanced by the ability to mutually reflect, influence, and merge by means of sensor/actuator networks deeply embedded in everyday environments. This paper describes a full implementation of a dual reality system using a popular online virtual world and a humancentric sensor network designed around a common electrical power strip. Example applications (e.g., browsing sensor networks in online virtual worlds), interaction techniques, and design strategies for the dual reality domain are demonstrated and discussed. Keywords: dual reality, virtual worlds, sensor network.

1

Introduction

At the heart of this paper is the concept of “dual reality,” which is defined as an environment resulting from the interplay between the real world and the virtual world, as mediated by networks of sensors and actuators. While both worlds are complete unto themselves, they are also enriched by their ability to mutually reflect, influence, and merge into one another. The dual reality concept, in turn, incorporates two key ideas – that data streams from real-world sensor networks are the raw materials that will fuel creative representations via interactive media that will be commonly experienced, and that online 3D virtual worlds are an ideal venue for the manifestation and interactive browsing of the content generated from such sensor data streams. In essence, sensor networks will turn the physical world into a palette, virtual worlds will provide the canvas on which the palette is used, and the mappings between the two are what will make their combination, dual reality, an art rather than an exact science. Of course, dual reality media will complement rather than replace other forms of media. Indeed, the end product, that which can be consumed and shared, is unlikely to outwardly resemble current forms of media, even if it is just as varied. Browsing the real world in a metaphorical virtual universe driven by a ubiquitous sensor network and unconstrained by physical boundaries approaches the concept of a digital “omniscience,” where users can fluidly explore phenomena at different locations and scales, perhaps also interacting with reality through distributed displays and actuators. Indeed, a complete consideration of dual reality must also include the possibility of “sensor” data from the F. Lehmann-Grube and J. Sablatnig (Eds.): FaVE 2009, LNICST 33, pp. 12–28, 2010. c Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering 2010

Dual Reality: Merging the Real and Virtual

13

Fig. 1. An environmental taxonomy as viewed on the real-virtual axis (left). Sensor networks seamlessly merge real and virtual to form dual reality (right).

virtual world embodied in the real world. Insofar as technically feasible, dual reality is bi-directional – just as sensed data from the real world can be used to enrich the virtual world, so too can sensed data from the virtual world be used to enrich the real world. Of the many axes along which various virtual worlds can be compared, the most relevant for this work is the real-virtual axis, which indicates how much of the constructed world is real and how much virtual. See Figure 1. A rough taxonomy can further compartmentalize the real-virtual axis into reality, which is simply life in the absence of virtual representations of the world; augmented reality, which has all aspects of reality, as well as an “information prosthetic” which overlays normally invisible information onto real objects [1,2]; mixed reality, which would be incomplete without both its real and virtual components, such as the partially built houses made complete with blue screen effects for use in military training exercises [3]; and virtual reality, which contains only elements generated by a computer in an attempt to mimic aspects of the real world, as exemplified in some popular computer games [4]. Contrast this with the taxonomy given by Milgram and Kishino in [5]. Each of these environments represents what is supposed to be a single, complete, and consistent world, regardless of which components are real or virtual. Although this taxonomy can be successfully applied to most enhanced reality efforts, it does not address well the concept of dual reality, which comprises a complete reality and a complete virtual reality, both of which are enhanced by their ability to mutually reflect, influence, and merge into each other by means of deeply embedded sensor/actuator networks. See Figure 1.

2

Background

By their nature, sensor networks augment our ability to understand the physical world in ways beyond our innate capabilities. With sensor networks and a record of the data they generate, our senses are expanded in space, time, and modality. As with previous expansions of our ability to perceive the world, some of the first and perhaps in the long run most important upshots will be the stimulation of new creative media as artists working in dual reality strive to express sensed phenomena into strong virtual experiences. The work described

14

J. Lifton and J.A. Paradiso

here begins to explore directions for such self-expression as it takes shape in the interplay between sensor networks and virtual worlds. There is no definition of online virtual worlds that is both agreed upon and useful. The term itself is vague enough to encompass a full spectrum of technologies, from text-based multiple user domains (MUDs) originating in the late 1970s [6] to visually immersive online 3D games commercially available today [7,8]. This work primarily focuses on the concept of virtual world as introduced in science fiction works by authors such as William Gibson [9] and Neil Stephenson [10]. This type of online virtual world is characterized by an immersive 3D environment, fluid interactions among inhabitants, and some level of ability for inhabitants to shape their environment. The goal may not be, and probably should not be, to replicate all aspects of the real world, but rather only those that facilitate the interaction in a virtual environment. In light of this, imbuing virtual worlds with the ability to sense aspects of the real world is a technique with significant potential. The real world portions of this work use the 35-node Plug sensor network described in [11,12,13] and reviewed in a later section. The virtual world portions of this work focus exclusively on Second Life, an online virtual world launched in 2003 and today still maintained by Linden Lab [14]. A comprehensive review of all online virtual worlds is beyond the scope of this work and better left to the several websites that specialize in such comparisons [7,8,15]. Second Life was chosen because of its technical and other advantages in implementing many of the dual reality ideas explored here. For a more detailed introduction to Second Life, see Linden Lab’s official guide book and the Second Life website [16,14]. 2.1

Self-expression in Virtual Worlds

Virtual worlds today are largely social in nature – people enter these worlds in order to meet other people and build connections with them through shared experiences. As in the real world, social interactions in virtual worlds revolve around self-expression. Taking Second Life as a representative example of the state-of-the-art in this respect, a resident of Second Life can express herself via the appearance and name of her avatar, the information revealed in her avatar’s profile (favorite places, preferences, etc.), her avatar’s scripted or explicitly triggered actions (dancing, laughing, running, etc.), text chat on public channels (received only by those nearby in the virtual world), text chat on private channels (received by a user-determined list of people regardless of their location in the virtual world), and live voice chat using a headset. A typical encounter when meeting another person for the first time, especially someone new to Second Life, revolves around explanations of how names and appearances were chosen, elaborations of details in avatar profiles, and exhibitions of clothing or animations. A less explicit although arguably more compelling form of self-expression in Second Life is the ability to build objects, from necklaces to cars to castles, and imbue them with a wide range of behaviors. The skill level needed to do so, however, is on par with that needed to build compelling web sites. As such, this form of self-expression is limited to a small proportion of the total virtual


15

world demographic. However, those who can build and script in Second Life can express themselves to a far wider audience than those who cannot. Compared to the real world, self-expression in Second Life and other virtual worlds is limited; missing are rich sources of information taken for granted in the real world, such as scent, body language, and the telltale signs of daily wear and tear. It’s not that these sources of information were forgotten, just that they are difficult to emulate in meaningful ways in the virtual world. For example, virtual wind causes virtual trees to sway, a virtual sun and moon rise and set periodically, and virtual clouds form and disperse in Second Life, but there is no meaning or cause behind any of these phenomena and their effect on the virtual world is superficial at best. Overall, the demand for richer forms of selfexpression in virtual worlds is apparent. Data collected from real-world sensor networks can help meet this demand by importing into the virtual world the inherent expressiveness of the real world. 2.2

The Vacancy Problem

The vacancy problem is the noticeable and profound absence of a person from one world, either real or virtual, while they are participating in the other. Simply put, the vacancy problem arises because people do not currently have the means to be in more than one place (reality) at a time. In the real world, the vacancy problem takes the form of people appearing completely absorbed in themselves, ignoring everything else. In the virtual world, the vacancy problem takes the form of virtual metropolises appearing nearly empty because there are not enough avatars to fill them. In part, this virtual vacancy is due to technical barriers preventing large numbers (hundreds) of people from interacting within the same virtual space. However, the vacancy problem will remain, even as processor speeds, network bandwidth, and graphics fidelity increase to overcome these technical difficulties. In a world nearly unconstrained by geography or physics, the currency of choice is people rather than real estate or possessions. As of this writing, there are over 10 million registered Second Life accounts, but only about 50,000 users logged into Second Life at any given time [17], providing a population density of 10 people per square kilometer (vs. over 18,000 for real-world Manhattan). The vacancy problem is a fundamental characteristic of today’s virtual worlds. More closely linking the real world with the virtual world, as the dual reality concept suggests, can work to mitigate the vacancy problem – just as real cities require special infrastructure to allow for a high population density, so too will virtual cities. We can envision people continuously straddling the boundary between real and virtual through “scalable virtuality”, where they are never truly offline, as sensor networks and mobile devices serve to maintain a continuous background inter-world connection (an early exploration of this idea was given in [18]). This can be tenuous, with virtual avatars passively representing some idea of the user’s location and activity and the virtual world manifesting into reality through ambient display, or immersive, with the user fully engaged in manipulating their virtual presence.

16

2.3


Mapping between Realities

There are numerous challenges in designing exactly how the real and virtual will interact and map onto each other. A direct mapping of the real to virtual and virtual to real may not be the most appropriate. For example, the sensor data streams collected from a real person may be better mapped to the virtual land the person’s avatar owns rather than to the avatar itself. One possible mapping strategy is to shape the virtual world according to our subjective perceptions of the real world. In essence, the virtual world would be a reflection of reality distorted to match our mind’s eye impressions as discerned by a network of sensors. For example, the buildings on a virtual campus could change in size according to the number of inhabitants and virtual corridors could widen or lengthen according to their actual throughput. 2.4

Related Work

Work that couples the real world with virtual worlds falls into several broad categories. There are several efforts to bring a virtual world into the real world by using positioning and proximity systems to cast real people as the actors of an otherwise virtual world, such as Human Pacman [19], Pac Manhattan [20], ARQuake [21], and DynaDOOM [22]. Such work remains almost exclusively within the realm of converting video games into live action games and, aside from location awareness, does not incorporate other sensing modalities. Magerkurth et al. provide a good overview of this genre of pervasive games, as well as other more sensor-rich but physically confined games [23]. In an attempt to make Second Life more pervasive in the real world, Comverse has created a limited Second Life interface for cell phones [24]. Virtual worlds are being used to involve citizens in the collaborative planning of real urban areas [25], although this type of system relies more on GIS data than sensor networks embedded in the environment. More advanced and correspondingly more expensive systems are used for military training [26]. Most of the systems mentioned above support only a handful of simultaneous users. Among efforts to bring the real world into the virtual world, it is standard practice to stream audio and video from live real events, such as conferences and concerts, into Second Life spaces built specifically for those events [27]. More ambitious and not as readily supported by existing technologies is the IBM UK Laboratories initiative in which the state of light switches, motorized blinds, the building’s electricity meter, and the like in a real lab space are directly reflected and can be controlled in a Second Life replication [28]. Similar efforts on a smaller scale include a general-purpose control panel that can be manipulated from both the real world and Second Life [29], and a homebrewed virtual reality wearable computer made specifically to interface to Second Life [30]. The convergence of Second Life, or something like it, with popular real-world mapping software to form a “Second Earth” has been broadly predicted [31]. Uses of such a “hyper reality” include analyzing real-world data (“reality mining”), as was done in the Economic Weather Map project [32]. Such ideas have appeared


17

before as interactive art pieces. For example, the Mixed Realities juried art competition organized by Turbulence (a net art commissioning organization [33]) in collaboration with Ars Virtua (a media center and gallery within Second Life [34]) recognizes projects that mix various aspects of the real and virtual [35]. Sensor network-enabled dual realities may naturally merge with or evolve from the life logging work pioneered by Gordon Bell [36,37] and popularized by web applications such as MySpace [38], Facebook [39], and Twitter [40]. Central to the dual reality concept is the expressive and social intent of the participants, which separates dual reality from the broader field of information visualization [41,42]. For example, consider services like Google Maps [43] and Traffic.com [44], that visualizes traffic congestion in a large metropolitan area. Traffic information might be gathered from numerous sources, such as cell towers, arial imagery, or user input, and displayed in a variety of ways, such as on the web, in a 3D virtual environment, or text messaging. The primary use of this service is to allow participants to intelligently plan their daily commute. Although hardly social by most standards, this service does form a social feedback loop; a user of the service will change her route according to the data presented and in doing so change the nature of the data presented to the next user. However, the motivation or intent of the service is entirely devoid of self-expression, and therefore does not readily fall under the rubric of dual reality. Closer to dual reality is VRcontext’s ProcessLife technology [45], which uses high-fidelity 3D virtual replicas of real environments to visualize and remotely influence industrial processes in real-time, though the potential for social interaction and rich metaphor appears low, as does the granularity of the sensor data visualizations.

3 3.1

Design and Implementation Real World Implementation

This work utilizes the previously developed “Plug” sensor network comprising 35 nodes modeled on a common electrical power outlet strip and designed specifically for ubiquitous computing environments [11,12,13]. A Plug offers four standard US electrical outlets, each augmented with a precision transformer for sensing the electrical current and a digitally controlled switch for quickly turning the power on or off. The voltage coming into the Plug is also sensed. In addition to its electrical power sensing and control features, each Plug is equipped with two LEDs, a push button, small speaker, analog volume knob, piezo vibration sensor, microphone, light sensor, 2.4GHz low-power wireless transceiver, and USB 2.0 port. An external expansion port features a passive infrared (PIR) sensor motion sensor, SD removable memory card, and temperature sensor. All the Plug’s peripherals are monitored and controlled by an Atmel AT91SAM7S64 microcontroller, which is based on the 32-bit ARM7 core, runs at 48MHz, and comes with 16KB of SRAM and 64KB of internal flash memory. Figure 2 shows Plug node with and without the external expansion. An extensive library of modular firmware can be pieced together into applications at compile time.

18


Fig. 2. A Plug sensor node with (right) and without (left) an external expansion

3.2

Virtual World Implementation

The following sections describe objects or effects in the Second Life virtual world that were designed as an example of interfacing with the real world through sensor networks. Everything in Second Life exists as some combination of land, avatars, objects, and scripts. Land in Second Life is mapped directly to Linden Lab server resources, such as computing cycles, memory, and bandwidth. Avatars are the virtual manifestation of real people using Second Life. Objects are built from one or more primitive three-dimensional solids (“prims”), such as spheres, cubes, tori, and cones. A script is a program written in the Linden Scripting Language (LSL) and placed in an object to affect the object’s behavior. Data Ponds. A single “data pond” is meant to be an easily distinguishable, locally confined representation of the sensor data from a single Plug node. See Figure 3. The data pond design consists of a cluster of waving stalks growing out of a puddle of water and an ethereal foxfire rising from among the stalks, as might be found in a fantastic swamp. The mapping between a Plug’s sensor data and its corresponding data pond is easily understood once explained, but still interesting even without the benefit of the explanation. The particular mapping used is detailed in Table 1. The data ponds allowed sensed phenomena in the physical world to be efficiently browsed virtually, and proved effective, for example, in seeing at a glance which areas of our lab were more active than others. A real version of the data pond complements the virtual version. The real version follows the virtual’s tentacle aesthetic by using a standard desk fan shrouded in a lightweight, polka dotted sheet of plastic. The air flow through the shroud and therefore the height, sound, and other idiosyncrasies of the shroud can be finely controlled by plugging the fan into the outlet of a Plug device and pulse width modulating the supply voltage accordingly. See Figure 3. Virtual Sensing. Whereas real sensor networks capture the low-level nuance of the real world, virtual sensor networks capture the high-level context of the


19

Fig. 3. A virtual data pond reflects real data near a virtual wall (left) and a real data pond reflects virtual data near a real wall (right)

Table 1. The mapping from a real-world Plug’s sensor data to its corresponding virtual data pond Plug Sensor Data Pond Mapping Modality Attribute light stalk length the stalk height is proportional to the maximum light level over the most recent one-second window temperature stalk color the color of the stalks varies linearly from blue to yellow to red from 18◦ C to 29◦ C motion stalk motion the stalks sway gently when no motion is detected and excitedly when motion is detected over the most recent one-second window sound puddle size the diameter of the water puddle is proportional to the maximum sound level over the most recent one-second window electrical current fire intensity the height and intensity of the fire is proportional to the total average absolute value of the electrical current over the most recent one-second window

20


Fig. 4. Side view of the final implementation of Shadow Lab, which includes data ponds. A human-sized avatar is standing in the foreground - our particular labspace is rendered in detail, while the rest of the building was represented by a map. In the background are buildings belonging to unrelated neighbors.

virtual world. For example, in reality, there are literally an infinite number of ways a person can touch a table, but in Second Life, there is exactly one. This work uses embedded and wearable virtual sensing schemes. The embedded sensing scheme entails seeding every object of interest in the virtual environment to be sensed with a script that detects when an avatar touches or otherwise interacts with the object and then reports back to a server external to Second Life with a full description of the interaction, including avatar position, speed, rotation, and identity. The wearable sensing scheme requires each avatar in the region of interest to wear a sensing bracelet. The sensing bracelet reports back to the same external server every five seconds with a full description of its avatar’s location, motion, and public channel chat. As incentive for avatars to wear the sensing bracelet, the bracelet also serves as an access token without which the avatar will be ejected from the region being sensed. Shadow Lab. Shadow Lab is a space in Second Life modeled after our real lab in which the Plug sensor network is deployed and exemplifies our real space to virtual space mapping. The primary feature of Shadow Lab is the to-scale two-dimensional floor plan of the third floor of our building. Only a small portion of the entire space is modeled in three dimensions. In part, this is due to the difficulty and resource drain of modeling everything in three dimensions. However, it is also a design decision reflecting the difficulty in maneuvering an avatar in a to-scale three dimensional space, which invariably feels too confining


21

Fig. 5. Avatar metamorphosis (left to right) as real-world activity increases

due to wide camera angles, quick movements, and the coarseness of the avatar movement controls in Second Life. Moreover, the two-dimensional design lends itself more readily to viewing the entire space at once and drawing attention to what few three-dimensional objects inhabit it. Figure 4 shows the latest version of Shadow Lab, which consists of the map of the lab, approximately 30 data ponds positioned on the map according to the positions of their corresponding Plugs in the real lab, and a video screen displaying a live video stream, when available, from a next-generation Tricorder [13] device equipped with a camera. Metamorphosis. The only unintentional body language exhibited in Second Life is the typing gesture avatars make when the user is typing a chat message, the slumped over sleeping stance assumed when the user’s mouse and keyboard have been inactive for a preset amount of time, automatically turning to look at nearby avatars who have just spoken, and a series of stances randomly triggered when not otherwise moving, such as hands on hips and a bored slouch. All other body language and avatar actions must be intentionally chosen by the user. Clearly, there is room for improvement. Metamorphosis explores mapping real space to a virtual person. See Figure 5. In this prototype, the avatar begins as a typical human and transforms into a Lovecraftian alien according to several parameters drawn from the sensor streams of the Plug sensor network spread throughout the real building. While this particular example is outlandish and grotesque, in practice the mapping used in a metamorphosis is arbitrary, which is exactly its appeal as a method of self-expression – metamorphosis can be mapped to other arbitrary stimuli and unfold in any fashion. Virtual Atrium. The translation of our lab’s atrium into Second Life attempts to retain that which is iconic about the original and at the same time take advantage of the freedom of the virtual world. See Figure 6. The virtual atrium is defined by the intersection of two perpendicular walls of tile, one representing the total activity level of the real world as sensed by the Plug network and the one representing the total activity of the virtual world as sensed by the virtual sensing systems mentioned above. The physical extent and color scheme of the virtual atrium walls change accordingly. Each tile has a blank white front face, four colored sides, and a black back face. Touching a tile will cause it to flip over, at which point the black back face comes to the front and changes to reveal a

22


Fig. 6. The real lab atrium (left) and the virtual version (right). A real person and an avatar show their respective scales.

Fig. 7. Side view of the Ruthenium region

hidden movie or image. All tiles in a given wall share the same image or movie when flipped, although the exact image or movie displayed is variable. Dual Reality Open House. At the time of this writing, the state-of-the-art in large events that bridge the real and virtual worlds amounts to what is essentially video conferencing between a real auditorium and a virtual auditorium [46]. As a prototype demonstration of moving beyond this by employing sensor networks, a dual reality open house was constructed to introduce residents of Second Life to the lab and visitors of the lab to Second Life. The dual reality open house premiered at a one-day technical symposium and held in the atrium of our lab [47]. The real portion of the event consisted of talks and panel discussions in the building’s main auditorium, interspersed with coffee breaks and standup meals in the atrium among tables manned by lab students demonstrating various lab projects related to virtual worlds. The virtual portion of the open house was located in a typical 256-meter by 256-meter region of Second Life [48] called “Ruthenium.” The server running the Ruthenium region is limited to 40 simultaneous avatars and 15,000 simultaneous prims. In preparation for the open


23

house, Ruthenium was terraformed and filled with static information kiosks and live demonstrations of various projects from around the lab. More details about the projects displayed can be found in [11]. The virtual atrium described in 3.2 framed the space where the virtual portion of our event took place. Data ponds and an avatar metamorphosis were featured as well. See Figure 7. The entire Ruthenium region employs the virtual sensing schemes described earlier.

4

Dual Reality Event and Discussion

The dual reality open house described earlier has the potential to explore the real data and virtual data collection systems. (See [12,11] for more detailed evaluations of the Plug sensor network.) Sensor data from both the real world and virtual world were collected during the day-long event. The real-world data originated from the Plug sensor nodes used throughout the real lab atrium at the various open house demo stations. Motion, sound, and electrical current data from a typical Plug are shown in Figure 8. Also collected but not shown here are data for each Plug’s light, voltage, vibration, and temperature sensors. The virtual-world data originated from the virtual sensing system previously detailed as deployed throughout the virtual portion of the dual reality open house described earlier. Such an extensive data set from a single event spread across both real and virtual worlds had not previously been collected. By the nature of the event and its presentation in each world, very little correlation between the real and virtual data was expected. However, each data set does speak to how people interact within each world separately and what the possibilities are for using data from one world in the other. The real-world sound and motion data shown in Figure 8 clearly follows the structure of the event as attendees alternate between the atrium during break times and the auditorium during the conference talks - the auditorium is noisier during breaks, during which demo equipment was also generally switched on and people are moving around the demos. On the other hand, the light data (not shown) indicate physical location more than attendee activity – direct sunlight versus fluorescent lights versus LCD projector light. See [11] for more detail. Of the various data collected from the virtual world during the day-long event, Figure 9 shows the distribution over time of touch events (avatars touching a virtual object equipped with the virtual embedded sensing system) and avatar movement events (the virtual wearable sensing system checks if its avatar is moving approximately once per second) collected from 22 avatars, of which 16 chose to wear the access bracelet virtual sensing system. Due to a network glitch, data collected from the virtual sensing system started being logged at approximately 11 AM rather than at 8 AM, when the event actually started. The spike of avatar movement at around noon is likely due to the pause in the live video stream from the auditorium when the talks broke for lunch, thus giving avatars watching the video stream incentive to move to another location to interact with other aspects of the virtual space. The relatively constant motion thereafter might indicate the exploratory nature of the participants and/or the space. Of all avatar-object

24


Fig. 8. Electrical current, sound level, and motion versus time from a typical Plug node during the dual reality open house

interactions, 83% were between an avatar and a virtual atrium wall tile, that displayed the live video feed from the real auditorium. This trial could have been improved in several respects. For example, the number of virtual attendees could have been increased with better advertising. Also (and most crucially), a stronger connection between real and virtual premises could have been made and “connectedness” metrics formulated and tested. These are being addressed in another dual reality event that we are hosting soon. 4.1

Discussion

In a completely fabricated virtual world, the entropy of a real-world data stream can dramatically alter the virtual ambiance. Certainly, a cleverly utilized pseudorandom number generator could do the same, but meaning derives more from perception than from the underlying mechanism, and it is much easier to weave a story from real data than from pseudo-random numbers. The act of weaving a story from sensor data is essentially the act of designing and implementing a mapping from data to a real or virtual manifestation of the data. A successful story must be meaningful to tell as well as to hear, and using sensor data grounded in either the real or virtual world helps achieve this. In essence, the act of creation must be as gratifying as the act of consumption. The creative aspects of dual reality, the mapping of real or virtual sensor data to some manifestation, will likely follow the trend of another recent medium – blogs. While blogs have allowed some creative geniuses an outlet and given them a wide, appreciative, and well-deserved audience, the quality of most blogs, at least as a consumptive medium, is far below previous mass media standards. Of course, their quality as a creative medium and the value they bring to their creators in that regard far exceed previous standards by virtue of their relatively low barrier to entry alone. These trends will be exaggerated in the context of dual reality for two reasons. First, the medium is much richer, involving virtual 3D worlds and complex social interactions and is therefore accessible to a wider


25

Fig. 9. Avatar movement and interaction during the dual reality open house

audience. Second, once the mapping of data to manifestation is set, the act of creation is nearly automatic (sitting somewhere between an interactive installation and a performance) and therefore a wider range of talent will participate. In short, the worst will be worse and the best will be better, a hallmark of successful mass media. As with other creative media, virtuosity will still play a critical role in dual reality, namely in the conception, implementation, and honing of the specific mappings between sensor data and their manifestations. These ideas are further discussed in [49]. While mapping sensor data to manifestation may be at the highest level of the dual reality creative process, once the mappings are in place, people can still intentionally express themselves in many ways, depending on the exact nature of the mapping. The evolution of emoticons in text messages is one example of such expression using a current technology. Another is the habit of maintaining an active online presence, such as used in Internet messaging clients, by jogging the computer’s mouse occasionally. In the same way, users of dual reality environments will modify their behavior so as to express themselves through the medium.

5

Conclusion

Various technologies have fundamentally altered our capacity to consume, share, and create media. Most notably, television and radio made consumption

26


widespread and the Internet made sharing widespread. In comparison, creation of media is still difficult and limited to a small subset of the population. The promise of dual reality is to use sensor/actuator networks as a generative tool in the process of transforming our everyday experiences in the real world into content shared and experienced in the virtual world. Just as the data created by a movie camera are shared and consumed in a theater, the data collected from sensor networks will be shared and consumed in virtual worlds. This holds the potential to revolutionize sensor network browsing, as participants fluidly explore metaphoric representations of sensor data - similarly, virtual denizens can manifest into real spaces through display and actuator networks. If sensor networks are the palette, then virtual worlds are the canvas that usher in a new form of mass media.

References 1. Feiner, S., et al.: Knowledge-based Augmented Reality. Comm. of the ACM 36(7), 53–62 (1993) 2. Sportvision. Virtual Yellow 1st and Ten (1998), http://www.sportvision.com/ 3. Dean Jr., F.S., et al.: Mixed Reality: A Tool for Integrating Live, Virtual & Constructive Domains to Support Training Transformation. In: Proc. of the Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) (2004) 4. Electronic Arts. SimCity (2007), http://simcity.ea.com/ 5. Milgram, P., Kishino, F.: A taxonomy of mixed reality visual displays. IEICE Trans. of Information Systems E77-D(12) (December 1994) 6. Rheingold, H.: The Virtual Community: Homesteading on the Electronic Frontier. Addison-Wesley, Reading (1993) 7. Good, R.: Online Virtual Worlds: A Mini-Guide (April 2007), http://www.masternewmedia.org/virtual reality/virtual-worlds/ virtual-immersive-3D-worlds-guide-20071004.htm 8. B. Book. Virtual Worlds Review (February 2006), http://www.virtualworldsreview.com/ 9. Gibson, W.: Neuromancer. Ace Books (1984) 10. Stephenson, N.: Snow Crash. Bantam Books (1992) 11. Lifton, J.: Dual Reality: An Emerging Medium. Ph.D. Dissertation, M.I.T., Dept. of Media Arts and Sciences (September 2007) 12. Lifton, J., et al.: A Platform for Ubiquitous Sensor Deployment in Occupational and Domestic Environments. In: Proc. of the Sixth Int’l Symposium on Information Processing in Sensor Networks (IPSN), April 2007, pp. 119–127 (2007) 13. Lifton, J., et al.: Tricorder: A mobile sensor network browser. In: Proc. of the ACM CHI 2007 Conference - Mobile Spatial Interaction Workshop (April 2007) 14. Linden Lab. Second Life (2003), http://www.secondlife.com 15. Lifton, J.: Technology Evaluation for Marketing & Entertainment Virtual Worlds. Electric Sheep Co. Report (2008), http://www.electricsheepcompany.com/publications/ 16. Rymaszewski, M., et al.: Second Life: The Official Guide. Wiley, Chichester (2007) 17. Linden Lab. Economic Statistics (2007), http://secondlife.com/whatis/economy_stats.php


27

18. Musolesi, M., et al.: The Second Life of a Sensor: Integrating Real-world Experience in Virtual Worlds using Mobile Phones. In: Fifth ACM Workshop on Embedded Networked Sensors (HotEmNets) (June 2008) 19. Cheok, A.D., et al.: Human Pacman: A Mobile Entertainment System with Ubiquitous Computing and Tangible Interaction over a Wide Outdoor Area. In: Fifth Int’l Symposium on Human Computer Interaction with Mobile Devices and Services (Mobile HCI), September 2003, pp. 209–223 (2003) 20. PacManhattan (2004), http://pacmanhattan.com 21. Thomas, B., et al.: ARQuake: An Outdoor/Indoor Augmented Reality First Person Application. In: Fourth Int’l Symposium on Wearable Computers (ISWC 2000) (2000) 22. Sukthankar, G.: The DynaDOOM Visualization Agent: A Handheld Interface for Live Action Gaming. In: Workshop on Ubiquitous Agents on Embedded, Wearable, and Mobile Devices (Conference on Intelligent Agents & Multiagent Systems) (July 2002) 23. Magerkurth, C., et al.: Pervasive Games: Bringing Computer Entertainment Back to the Real World. ACM Computers in Entertainment 3(3) (July 2005) 24. Roush, W.: New Portal to Second Life: Your Phone. Technology Review (2007), http://www.technologyreview.com/Infotech/18195/ 25. MacIntyre, J.: Sim Civics. Boston Globe (August 2005), http://www.boston.com/news/globe/ideas/articles/2005/08/07/ sim civics/ 26. Miller, W.: Dismounted Infantry Takes the Virtual High Ground. Military Training Technology 7(8) (December 2002) 27. Jansen, D.: Beyond Broadcast 2007 – The Conference Goes Virtual: Second Life (2006), http://www.beyondbroadcast.net/blog/?p=37 28. IBM. Hursley Island (2007), http://slurl.com/secondlife/Hursley/0/0/0/ 29. ciemaar. Real Life Control Panel for Second Life (2007), http://channel3b.wordpress.com/2007/01/24/ real-life-control-panel-for-second-life/ 30. Torrone, P.: My wearable computer – snowcrash (January 2006), http://www.flickr.com/photos/pmtorrone/sets/1710794/ 31. Roush, W.: Second Earth. Technology Review 110(4), 38–48 (2007) 32. Boone, G.: Reality Mining: Browsing Reality with Sensor Neworks. Sensors Magazine 21(9) (September 2004) 33. Turbulence (2007), http://www.turbulence.org/ 34. Ars Virtua (2007), http://arsvirtua.org/ 35. Turbulence. Mixed Realities Commissions (2007), http://transition.turbulence.org/comp_07/awards.html 36. Bell, G.: A Personal Digital Store. Comm. of the ACM 44(1), 86–91 (2001) 37. Gemmell, J., et al.: MyLifeBits: A Personal Database for Everything. Comm. of the ACM 49(1), 88–95 (2006) 38. MySpace (2007), http://www.myspace.com/ 39. Facebook (2007), http://www.facebook.com/ 40. Twitter (2007), http://twitter.com/ 41. Tufte, E.R.: The Visual Display of Quantitative Information. Graphics Press (1983) 42. Chen, C.: Information Visualisation and Virtual Environments. Springer, Heidelberg (1999)

28 43. 44. 45. 46.


Google. Google Maps (2007), http://maps.google.com Navteq. Traffic.com (2007), http://www.traffic.com VRcontext. ProcessLife (February 2009), http://www.vrcontext.com/ Verbeck, S.: Founder and CEO of The Electric Sheep Company. Personal comm. via e-mail, July 9 (2007) 47. IBM. Virtual Worlds: Where Business, Society, Technology, & Policy Converge (June 15, 2007), http://www.research.ibm.com/research/press/virtualworlds_agenda.shtml 48. Lifton, J.: Media Lab Dual Reality Open House (2007), http://slurl.com/secondlife/Ruthenium/0/0/0/ 49. Lifton, J., Laibowitz, M., Harry, D., Gong, N., Mittal, M., Paradiso, J.A.: Metaphor and Manifestation: Cross Reality with Ubiquitous Sensor/Actuator Networks. IEEE Pervasive Computing Magazine (Summer 2009)

Exploring the Use of Virtual Worlds as a Scientific Research Platform: The Meta-Institute for Computational Astrophysics (MICA) S.G. Djorgovski1,∗, P. Hut2,*, S. McMillan3,*, E. Vesperini3,*, R. Knop3,*, W. Farr4,*, and M. J. Graham1,* 1

California Institute of Technology, Pasadena, CA 91125, USA The Institute for Advanced Study, Princeton, NJ 08540, USA 3 Drexel University, Philadelphia, PA 19104, USA 4 Massachusetts Institute of Technology, Cambridge, MA 02139, USA [email protected] 2

Abstract. We describe the Meta-Institute for Computational Astrophysics (MICA), the first professional scientific organization based exclusively in virtual worlds (VWs). The goals of MICA are to explore the utility of the emerging VR and VWs technologies for scientific and scholarly work in general, and to facilitate and accelerate their adoption by the scientific research community. MICA itself is an experiment in academic and scientific practices enabled by the immersive VR technologies. We describe the current and planned activities and research directions of MICA, and offer some thoughts as to what the future developments in this arena may be. Keywords: Virtual Worlds; Astrophysics; Education; Scientific Collaboration and Communication; Data Visualization; Numerical Modeling.

1 Introduction Immersive virtual reality (VR), currently deployed in the form of on-line virtual worlds (VWs) is a rapidly developing set of technologies which may become the standard interface to the informational universe of the Web, and profoundly change the way humans interact with information constructs and with each other. Just as the Web and the browser technology has changed the world, and almost every aspect of modern society, including scientific research, education, and scholarship in general, a synthesis of the VR and the Web promises to continue this evolutionary process which intertwines humans and the world of information and knowledge they create. Yet, the scientific community at large seems to be at best poorly informed (if aware at all) of this technological emergence, let alone engaged in spearheading the developments of the new scientific, educational, and scholarly modalities enabled by these technologies, or even new ideas which may translate back into the better ways ∗

All authors are also associated with the Meta-Institute for Computational Astrophysics (MICA), http://mica-vw.org

F. Lehmann-Grube and J. Sablatnig (Eds.): FaVE 2009, LNICST 33, pp. 29–43, 2010. © Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering 2010

30

S.G. Djorgovski et al.

in which these technologies can be used for practical and commercial applications outside the world of academia. There has been a slowly growing interest and engagement of the academic community in the broad area of humanities and social sciences in this arena (see, e.g., [1, 2, 3, 4, 5], and references therein), but the “hard sciences” community has barely touched these important and potentially very powerful developments. While a few relatively isolated individuals are exploring the potential uses of VWs as a scholarly platform, the scientific/academic community as a whole has yet to react to these opportunities in a meaningful way. One reason for this negligence may be a lack of the real-life examples of the scientific utility of VWs. It is important to engage the scientific community in serious uses and developments of immersive VR technologies. With this growing set of needs and opportunities in mind, following some initial explorations of the VWs as a scholarly interaction and communication platform [6, 7], we formed the Meta-Institute for Computational Astrophysics (MICA) [8] in the spring of 2008. Here we describe the current status and activities of MICA, and its long-term goals.

2 The Meta-Institute for Computational Astrophysics (MICA) To the best of our knowledge, MICA is the first professional scientific organization based entirely in VWs. It is intended to serve as an experimental platform for science and scholarship in VWs, and it will be the organizing framework for the work proposed here. MICA is currently based in Second Life (SL) [9] (it initially used the VW of Qwaq [10]), but it will expand and migrate to other VWs and venues as appropriate. The charter goals of MICA are: 1. Exploration, development and promotion of VWs and VR technologies for professional research in astronomy and related fields. 2. To provide and develop novel social networking venues and mechanisms for scientific collaboration and communications, including professional meetings, effective telepresence, etc. 3. Use of VWs and VR technologies for education and public outreach. 4. To act as a forum for exchange of ideas and joint efforts with other scientific disciplines in promoting these goals for science and scholarship in general. To this effect, MICA conducts weekly professional seminars, bi-weekly popular lectures, and many other regularly scheduled and occasional professional discussions and public outreach events, all of them in SL. Professional members of MICA include scientists (faculty, staff scientists, postdocs, and graduate students), technologists, and professional educators; about 40 people as of this writing (March 2009). A broader group of MICA affiliates includes members of the general public interested in learning about astronomy and science in general; it currently consists of about 100 people (also as of March 2009). The membership of both groups is growing steadily. We have been very proactive in engaging both academic community (in real life and in SL) and general public, in the interests of our stated goals. Both our membership and activities are global in scope, with participants from all over the world, although a majority resides in the U.S.

Exploring the Use of Virtual Worlds as a Scientific Research Platform

31

MICA is thus a testbed and a foothold for science and scholarship in VWs, and we hope to make it both a leadership institution and a center of excellence in this arena, as well as an effective portal to VWs for the scientific community at large. While our focus is in astrophysics and related fields, where our professional expertise is, we see MICA in broader terms, and plan to interact with scientists and educators in other disciplines as well. We also plan to develop partnerships with the relevant industry laboratories, and conduct joint efforts in providing innovation in this emerging and transformative technology. The practical goals of MICA are two-fold. First, we wish to lead by example, and demonstrate the utility of VWs and immersive VR environments generally for scientific research in fields other than humanities and social sciences (where we believe the case is already strong). In that process, we hope to define the “best practices” and optimal use of VR tools in research and education, including scholarly communications. This is the kind of activity that we expect will engage a much broader segment of the academic community in exploration and use of VR technologies. Second, we hope to develop new research tools and techniques, and help lay the foundations of the informational environments for the next generation of VR-enabled Web. Specifically, we are working in the following directions: 2.1 Improving Scientific Collaboration and Communication Our experience is that an immediate benefit of VWs is as an effective scientific communication and collaboration platform. This includes individual, group, or collaboration meetings, seminars, and even full-scale conferences. You can interact with your colleagues as if they were in the same room, and yet they may be half way around the world. This is a technology which will finally make telecommuting viable, as it provides a key element that was missing from the flat-Web paradigm: the human interaction. We finally have a “virtual water cooler”, the collegial gathering work spaces to enhance and expand our cyber-workspaces. VWs are thus a very green technology: you can save your time, your money, and your planet by not traveling if you don't have to. This works well enough already, at almost no cost, and it will get better as the interfaces improve, driven by the games and entertainment industry, if nothing else. This shift to virtual meetings can potentially save millions of dollars of research funding, which could be used for more productive purposes than travel to collaboration or committee meetings, or to conferences of any kind. We have an active program of seminars, lectures, collaboration meetings, and freeform scholarly discussions within the auspices of MICA, and we are proactive in informing our real-life academic community about these possibilities. We offer coaching and mentoring for the novices, and share our experiences on how to best use immersive VR for scientific communication and collaboration with other researchers. In addition, starting in a near future, we plan to organize a series of topical workshops on various aspects of computational science (both general, and specific to astrophysics), as well as broader-base annual conferences on science and scholarship in VWs, including researchers, technologists, and educators from other disciplines. These meetings will be either entirely based in VWs (SL to start), or be in “mixed reality”, with both real-life and virtual environment gatherings simultaneously, connected by streaming media.

32


Fig. 1. MICA members attending a regular weekly astrophysics seminar, in this case by Dr. M. Trenti, given in the StellaNova sim in SL. Participants in these meetings are distributed worldwide, but share a common virtual space in which they interact.

Genuine interdisciplinary cross-fertilization is a much-neglected path to scientific progress. Given that many of the most important challenges facing us (e.g., the global climate change, energy, sustainability, etc.) are fundamentally interdisciplinary in nature, and not reducible to any given scientific discipline (physics, biology, etc.), the lack of effective and pervasive mechanisms for establishment of inter-, multi-, or cross-disciplinary interactions is a serious problem which affects us all. One reason for the pervasive academic inertia in really engaging in true and effective interdisciplinary activities is the lack of easy communication venues, intellectual melting pots where such encounters can occur and flourish. VWs as scientific interaction environments offer a great new opportunity to foster interdisciplinary meetings of the minds. They are easy, free, do not require travel, and the social barriers are very low and easily overcome (the ease and the speed of striking conversations and friendships is one of the more striking features of VWs). To this end, we will establish a series of broad-based scientific gatherings, from informal small group discussions, to full-size conferences. We note that once a VR environment is established, e.g., in a “sim” in SL, the cost (in both time and money) of organizing conferences is almost negligible, and the easy and instant worldwide access with no physical travel makes them easy to attend. Thus, we have developed a dedicated “MICA island” (sim), named StellaNova [11] within SL. This is intended to be the Institute’s home location in VWs; it is currently in SL as the most effective and convenient venue, but we will likely expand and migrate to other VW venues when that becomes viable and desirable. StellaNova is used as a staging area for most of our activities, including meetings, workshops, discussions, etc. It is intended to be a friendly and welcoming virtual environment for scholarly collaborations and discussions, very much in the tradition of academe of the golden age Athens.


33

A part of our exploration of VWs as scientific communication and collaboration platforms is an investigation in the mixed use of traditional Web (1.0, 2.0, … 3.0?) and VR tools; we are interested in optimizing the uses of information technology for scientific communications generally, and not just exclusively in a VR context, although a VR component would always be present. We plan to evaluate the relative merits of these technologies for different aspects of professional scientific and scholarly interaction and networking – while the Web mechanisms may be better for some things, VWs may be better for others. Finally, we intend to investigate the ways in which immersive VR can be used as a part of scientific publishing, either as an equivalent of the current practice of supplementing traditional papers with on-line material on the Web, or even as a primary publishing medium. Just as the Web offers new possibilities and modalities for scholarly publishing which do not simply mimic the age-old printed-paper media publishing, so we may find qualitatively novel uses of VWs as a publishing venue in their own right. After all, what is important is the content, and not the technical way in which the information is encoded; and some media are far more effective than others in conveying particular types of scholarly content. 2.2 A New Approach to Numerical Simulations Immersive VR environments open some intriguing novel possibilities in the ways in which scientists can set up, perform, modify, and examine the output of numerical simulations. In MICA, we use as our primary science environment the gravitational N-body problem, since that is where our professional expertise is concentrated [12, 13, 14, 15, 16, 17], but we expect that most of the features we develop will find much broader applicability in the visualization of more general scientific or abstract data sets. Our goal is to create virtual, collaborative visualization tools for use by computational scientists working in an arbitrary VW environment, including SL [9], OpenSim [18], etc. Here we address interactive and immersive visualization in the numerical modeling and simulations context; we address the more general issues of data visualization below. For an initial report, see [40]. We started our development of in-world visualization tools by creating scripts to display a set of related gravitational N-body experiments. The gravitational N-body problem is easy to state and hard to solve: given the masses, positions, and velocities of a collection of N bodies moving under the influence of their mutual Newtonian gravitational interactions, according to the laws of Newtonian mechanics, determine the bodies' positions and velocities at any subsequent time. In most cases, the motion has no analytic solution, and must be computed numerically. Both the character of the motion and the applicable numerical techniques depend on the scale of the system. Most of the essential features of the few-body problem can be grasped from studies of the motion of 3-5 body systems, in bound or scattering configurations. The physics and basic mathematics are elementary, and the required programming is straightforward. Yet, despite these modest foundations, such systems yield an extraordinarily rich spectrum of possible outcomes. The idea that simple deterministic systems can lead to complex, chaotic results is an important paradigm shift in many students' perception of physics. Few-body dynamics is also critically important in the determining

34


the evolution and appearance of many star clusters, as well as the stability of observed multiple stellar systems. These systems are small enough that the entire calculation could be done entirely within VWs, although we would wish to preserve the option of also importing data from external sources. This tests the basic capabilities of the visualization system – updating particles, possibly interpolating their motion, stopping, restarting, running backwards, resetting to arbitrary times, zooming in and out, etc. The next level of simulation involves broadening the context of our calculations to study systems containing several tens of particles, which will allow us to see both the few-body dynamics and how they affect the parent system. Specifically, the study of binary interactions and heating, and the response of the larger cluster, will illustrate the fundamental dynamical processes driving the evolution of most star clusters. We will study the dynamics of systems containing binary systems, a possible spectrum of stellar masses, and real (if simplified) stellar properties. These simulations are likely to lie at the high end of calculations that can be done entirely within the native VW environments, and much of the data may have to be imported. The capacity to identify, zoom in on, and follow interesting events, and to change the displayed attributes of stars on the fly will be key to the visualization experience at this level. The evolution of very large systems, such as galaxies, is governed mainly by largescale gravitational forces rather than by small-scale individual interactions, so studies of galaxy interactions highlight different physics and entail quite different numerical algorithms from the previous examples. It will not be feasible to do these calculations within the current generation of VWs, or to stream in data fast enough to allow for animation, so the goal in this case will be to import, render, and display a series of static 3-D frames, which will nevertheless be “live” in the sense that particles of different sorts (stars, gas, dark matter, etc.) or with other user-defined properties can be identified and highlighted appropriately. The choice of N ~ 50,000 is small compared to the number of stars in an actual galaxy, and it is more typical of a large star cluster. However, with suitable algorithms, galaxies can be adequately modeled by simulations on this scale, and this choice of N is typical of low-resolution calculations of galaxy dynamics, such as galaxy collisions and mergers, that are often used for pedagogical purposes. It also represents a compromise in the total amount of data that can be transferred into the virtual environment in a reasonable time. The intent here will be to allow users to visualize the often complex 3D geometries of these systems, and to explore some of their dynamical properties. This visualization effort in this case will depend on efficient two-way exchange of data between the in-world presentation and the external engine responsible for both the raw data and the computations underlying many aspects of the display. Our first goal is thus to explore the interactive visualization of simulations running within the VWs computational environments, thus offering better ways to understand the physics of the simulated processes – essentially the qualitative changes in the ways scientists would interact with their simulations. Our second goal is to explore the transition regime where the computation is actually done externally, on a powerful or specialized machine, but the results are imported into a VW environment, while the user feedback and control are exported back, and determine the practical guidelines as to how and when such a transition should be deployed in a real-life numerical study of astrophysical systems. The insights gained here would presumably be portable to


35

Fig. 2. A MICA astrophysicist immersed in, and interacting with, a gravitational N-body simulation using the OpenSim environment

36


other disciplines (e.g., biology, chemistry, other fields of physics, etc.) where numerical simulations are the only option of modeling of complex systems. 2.3 Immersive Multi-Dimensional Data Visualization In a more general context, VWs offer intriguing new possibilities for scientific visualization or “visual analytics” [19, 20]. As the size, and especially the complexity of scientific data sets increase, effective visualization becomes a key need for data analysis: it is a bridge between the quantitative information contained in complex scientific measurements, and the human intuition which is necessary for a true understanding of the phenomena in question. Most sciences are now drowning under the exponential growth of data sets, which are becoming increasingly more complex. For example, in astronomy we now get most of our data from large digital sky surveys, which may detect billions of sources and measure hundreds of attributes for each; and then we perform data fusion across different wavelengths, times, etc., increasing the data complexity even further. Likewise, numerical simulations also generate huge, multi-dimensional output, which must be interpreted and matched to equally large and complex sets of measurements. Examples include structure formation in the universe, modeling of supernova explosions, dense stellar systems, etc. This is an even larger problem in biological or environmental sciences, among others. We note that the same challenges apply to visualization of data from measurements, numerical simulations, or their combination. How do we visualize structures (clusters, multivariate correlations, patterns, anomalies...) present in our data, if they are intrinsically hyper-dimensional? This is one of the key problems in data-driven science and discovery today. And it is not just the data, but also complex mathematical or organizational structures or networks, which can be inherently and essentially multi-dimensional, with complex topologies, etc. Effective visualization of such complex and highly-dimensional data and theory structures is a fundamental challenge for the data-driven science of the 21st century, and these problems will grow ever sharper, as we move from Terascale to Petascale data sets of ever increasing complexity. VWs provide an easy, portable venue for pseudo-3D visualization, with various techniques and tricks to encode more parameter space dimensions, with an added benefit of being able to interact with the data and with your collaborators. While there are special facilities like “caves” for 3D data immersion, they usually require a room, expensive equipment, special goggles, and only one person at a time can benefit from the 3D view. With an immersive VW on your laptop or a desktop, you can do it for free, and share the experience with as many of your collaborators as you can squeeze in the data space you are displaying, in a shared, interactive environment. These are significant practical and conceptual advantages over the traditional graphics packages, and if VWs become the standard scientific interaction venue as we expect, then bringing the data to the scientists only makes sense. Immersing ourselves in our data may help us think differently about them, and about the patterns we see. With scientists immersed in their data sets, navigating around them, and interacting with both the data and each other, new approaches to data presentation and understanding may emerge.


37

Fig. 3. MICA scientists in an immersive data visualization experiment, developed by D. Enfield and S.G. Djorgovski. Data from a digital sky survey are represented in a 6-dimensional parameter space (XYZ coordinates, symbol sizes, shapes, and colors).

We have conducted some preliminary investigation of simple multi-dimensional data visualization scripting tools within SL. We find that we can encode data parameter spaces with up to a dozen dimensions in an interactive, immersive pseudo-3D display. At this point we run into the ability of the human mind to easily grasp the informational content thus encoded. A critical task is to experiment further in finding the specific encoding modalities that maximize our ability to perceive multiple data dimensions simultaneously, or selectively (e.g., by focusing on what may stand out as an anomalous pattern). One technical challenge is the number of data objects that can be displayed in a particular VW environment; SL is especially limiting in this regard. Our next step is to experiment with visualizations in custom VW environments, e.g., using OpenSim [18], which can offer scalable solutions needed for the modern large data sets. However, even an environment like SL can be used for experimentation with modest-scale data sets (e.g., up to ~ 104 data objects), and used to develop the methods for an optimal encoding of highly-dimensional information from the viewpoint of human perception and understanding. Additional questions requiring further research include studies of combined displays of data density fields, vector fields, and individual data point clouds, and the ways in which they can be used in the most effective way. This is a matter of optimizing human perception of visually displayed information, a problem we will tackle in a purely experimental fashion, using VWs as a platform. The next level of complexity and sophistication comes with introduction of the time element, i.e., sequential visualization of changing data spaces (an obvious example is the output of numerical simulations of gravitational N-body systems, discussed in the previous section). We are all familiar with digital movies displaying such information in a 2-D format. What we are talking about here is immersive 3-D data cinematography, a novel concept, and probably a key to a true virtualization of

38


scientific research. Learning how to explore dynamical data sets in this way may lead to some powerful new ways in which we extract knowledge and understanding from our data sets and simulations. Implementing such data visualization environment poses a number of technical challenges. We plan experiment with interfacing of the existing visualization tools and packages with VW platforms: effectively, importing the pseudo-3D visualization signal into VWs, but with a goal of embedding the user avatar in the displayed space. We may be able to adopt some emergent solutions of this problem from the games or entertainment industry, should any come up. Alternatively, we may attempt to encode a modest-scale prototype system within the VW computational environments themselves. A hybrid approach may be also possible. 2.4 Exploring the OpenGrid and OpenSim Technologies Most of the currently open VWs are based on proprietary software architectures, formats, or languages, and do not interoperate with each other; they are closed worlds, and thus probably dead ends. OpenSimulator (or OpenSim) [18] is a VW equivalent of the open source software movement. It is an open-source C# program which implements the SL VW server protocol, and which can be used to create a 3-D VW, and includes facilities for creating custom avatars, chatting with others in the VR environment, building 3-D content and creating complex 3-D applications in VW. It can also be extended via loadable modules or Web service interfaces to build more custom 3-D applications. OpenSim is released under a BSD license, making it both open source, and commercially friendly to embed in products. To demonstrate the feasibility of this approach, we have conducted some preliminary experiments in the uses of OpenSim for astrophysical N-body simulations, using a plugin, MICAsim [21, 22]. We have modified the standard OpenSim physics engine as a plugin, to run gravitational N-body experiments in this VW environment. We found that it's practical to run about 30 bodies in a gravitational cold-collapse model with force softening to avoid hard binary interactions in the simulator, where a few simulator seconds corresponds to a crossing time. We believe that we could get another factor of two in N from code optimizations in this setting. We will continue to explore actively the use of OpenSim for our work, and in particular in the arena or numerical simulations and visualization, and pay a close attention to the issues of avatar and inventory interoperability and portability. A start along these lines is ScienceSim [23]. Having an immersive VR environment on one’s own machine can bypass many of the limitations of the commercial VW grids, such as SL, especially in the numbers of data points that can be rendered. It is likely that the convergence of the Web and immersive VR would be in the form whereby one runs and manages their own VR environment in a way which is analogous to hosting and managing one’s own website today. OpenSim and its successors, along with a suitable standardization for interoperability, may provide a practical way forward; see also [24]. 2.5 Information Architectures for the Next Generation Web One plausible vision of the future is that there will be a synthesis of the Web, with its all-encompassing informational content, and the immersive VR as an interface to it,


39

since it is so well suited to the human sensory input mechanisms. One can think of immersive VR as the next generation browser technology, which will be as qualitatively different from the current, flat desktop and web page paradigm, as the current browsers were different from the older, terminal screen and file directory paradigm for information display and access. A question then naturally arises: what will be the newly enabled ways of interacting with the informational content of the Web, and how should we structure and architect the information so that it is optimally displayed and searched under the new paradigm? To this effect, what we plan to do is to investigate the ways in which large scientific databases and connections between them (e.g., in federated data grid frameworks, such as the Virtual Observatory [25, 26, 27]) can be optimally rendered in an immersive VR environment. This is of course a universal challenge, common to all sciences and indeed any informational holdings on the Web, beyond academia. Looking further ahead, many of the new scientific challenges and opportunities will be driven by the continuing exponential growth of data volumes, with the typical doubling times of ~ 1.5 years, driven by the Moore’s law which characterizes the technology which produces the data [35, 36]. An even greater set of challenges is presented by the growth of data complexity, especially as we are heading into the Petascale regime [37, 38, 39]. However, these issues are not limited to science: the growth of the Web constantly overwhelms the power of our search technologies, and brute-force approaches seldom work. Processing, storing, searching, and synthesizing data will require a scalable environment and approach, growing from the current “Cloud+Client” paradigm. Only by merging data and compute systems into a truly global or Web-scale environment – virtualizing the virtual – will sufficient computational and data storage capacity be available. A strong feature of such an environment will be high volume, frequent, low latency services built on message-oriented architectures as opposed to today’s serviceoriented architectures. There will be a heterogeneity of structured, semi-structured and unstructured data that will need to be persisted in an easily searchable manner. Atop of that, we will likely see a strong growth in semantic web technologies. This changing landscape of data growth and intelligent data discovery poses a slew of new challenges: we will need some qualitatively new and different ways of visualizing data spaces, data structures, and search results (here by “data” we mean any kind of informational objects – numerical, textual, images, video, etc.). Immersive VR may become a critical technology to confront these issues. Scientists will have to be increasingly immersed into their data and simulations, as well as the broader informational environment, i.e., the next generation Web, whatever its technological implementations are, simply for the sake of efficiency. However, the exponential growth of data volumes, diversity, and complexity already overwhelms the processing capacity of a single human mind, and it is inevitable that we will need some capable AI tools to aid us in exploring and understanding the data and the output of numerical models and simulations. Much of the data discovery and data analysis may be managed by intelligent agents residing in the computing/data environment, that have been programmed with our beliefs, desires and intents. They will serve both as proxies for us reacting to results and new data according to programmed criteria expressed in declarative logic languages and also as our interface point into the computing/data environment for

40


activities such as data visualization. Interacting with an agent will be a fully immersive experience combining elements of social networking with advances in virtual world software. Thus, we see a possible diversification of the concept of avatars – as they blend with intelligent software agents, possibly leading to new modalities of human and AI representation in virtual environments. Humans create technology, and technology changes us and our culture in unexpected ways; immersive VR represents an excellent example of an enabling cognitive technology [28, 29]. 2.6 Education and Public Outreach VWs are becoming another empowering, world-flattening educational technology, very much like as the Web has already done. Anyone from anywhere could attend a lecture in SL, whether they are a student or simply a science enthusiast. What VWs provide, extending the Web, is the human presence and interaction, which is an essential component of an effective learning process. That is what makes VWs such a powerful platform for any and all educational activities which involve direct human interactions (e.g., lectures, discussions, tutoring, etc.). In that, they complement and surpass the traditional Web, which is essentially a medium to convey pre-recorded lectures, as text, video, slides, etc. Beyond the direct mappings of traditional lecture formats, VWs can really enable novel collaborative learning and educational interactions. Since buildings, scenery, and props are cheap and easy to create, VWs are a great environment for situational training, exploration of scenarios, and such. Medical students can dissect virtual cadavers, and architects can play with innovative building designs, just moving the bits, without disturbing any atoms. Likewise, physicists can construct virtual replicas of an experimental apparatus, which students can examine, assemble, or take apart. There is already a vibrant, active community of educators in SL [30, 31], and many excellent outreach efforts are concentrated in the SL SciLands virtual continent [32]. MICA’s own efforts include a well-attended series of popular talks, “Dr.Knop talks astronomy” [33], which includes guest lecturers, as well as informal weekly “Ask an Astronomer” gatherings. We will continue with these efforts, and expand the range of our popular lectures. Under the auspices of MICA, we are starting to experiment with regularly scheduled classes and/or class discussions in SL, and we will explore such activities in other VW environments as well. These may include an introductory astronomy class, or an advanced topic seminar aimed at graduate students. We will also try a hybrid format, where the students would read the lecture materials on their own, and use the class time for an open discussion and explanations of difficult concepts in a VW setting. We also plan to conduct a series of international “summer schools” on the topics of numerical stellar dynamics, computational science, and possibly others, in an immersive and interactive VW venue.

3 Concluding Comments In MICA, we have started to build a new type of a scientific institution, dedicated to an exploration of immersive VR and VWs technologies for science, scholarship, and


41

education, aimed primarily at academics in physical and other natural sciences. MICA itself is an experiment in the new ways of conducting scholarly work, as well as a testbed for new ideas and research modalities. It is also intended to be a gateway for other scholars, new to VWs, to start to explore the potential and the practical uses of these technologies in an easy, welcoming, and collegial environment. MICA represents a multi-faceted effort aimed to develop new modalities of scientific research and communication using new technologies of immersive VR and VWs. We believe that they will enable and open qualitatively new ways in which scientists interact among themselves, with their data, and with their numerical simulations, and thus foster some genuine new “computational thinking” [34] approaches to science and scholarship. We use the VWs as a platform to conduct rigorous research activities in the fields of computational astrophysics and data-intensive astronomy, seeking to determine the potential of these new technologies, as well as to develop a new set of best practices for scholarly and research activities enabled by them, and by a combination of the existing Web-based and the new VR technologies. In that process, we may facilitate new astrophysical discoveries. We also hope to generate new ideas and methods which will in turn stimulate development of new technological capabilities in immersive VR and VWs, both as research and communication tools, and in the true sense of human-centered computational engineering. The central idea here is that immersive VR and VWs are potentially transformative technologies on par with the Web itself, which can and should be used for serious purposes, including science and scholarship; they are not just a form of games. By conveying this idea to professional scientists and scholars, and by leading by example, we hope to engage a much broader segment of the academic community in utilizing, and developing further these technologies. This evolutionary process may have an impact well beyond the academia, as these technologies blend with the cyber-world of the Web, and change the ways we interact with each other and with the informational content of the next generation Web. While at a minimum we expect to develop a set of “best practices” for the use of VR and VWs technologies in science and scholarship, it is also possible that practical and commercial applications may result or may be inspired by this work. If indeed immersive VR becomes a major new component of the modern society, as a platform for commerce, entertainment, etc., the potential impact may be very significant. In our work, we are assisted by a large number of volunteers, including scientists, technologists, and educators, most of them professional members of MICA. Some of them are actively engaged in the VWs development activities under the auspices of various governmental agencies, e.g., NASA. We have also established a strong network of international partnerships, including colleagues and institutions in the Netherlands, Italy, Japan, China, and Canada (a list which is bound to grow). We are also establishing collaborative partnerships with several groups in the IT industry, most notably Microsoft Research, and IBM, and we expect that this set of collaborations will also grow in time. This broad spectrum of professionally engaged parties showcases the growing interest in the area of scientific and scholarly uses of VWs, and their further developments for such purposes.

42


Acknowledgments. The work of MICA has been supported in part by the U.S. National Science Foundation grants AST-0407448 and HCC-0917817, and by the Ajax Foundation. We also acknowledge numerous volunteers who have contributed their time and talents to this organization, especially S. McPhee, S. Smith, K. Prowl, C. Woodland, D. Enfield, S. Cianciulli, T. McConaghy, W. Scotti, J. Ames, and C. White, among many others. We also thank the conference organizers for their interest and support. SGD also acknowledges the creative atmosphere of the Aspen Center for Physics, where this paper was completed.

References 1. Bainbridge, W.S.: The Scientific Research Potential of Virtual Worlds. Science 317, 472– 476 (2007) 2. Journal of Virtual Worlds Research, http://jvwresearch.org/ 3. TerraNova blog, various authors, http://terranova.blogs.com/ 4. Boellstorff, T.: Coming of Age in Second Life: An Anthropologist Explores the Virtually Human. Princeton University Press, Princeton (2008) 5. Convergence of the Real and the Virtual, the first scientific conference held inside World of Warcraft, May 9-11 (2008), http://mysite.verizon.net/wsbainbridge/convergence.htm 6. Hut, P.: Virtual Laboratories. Prog. Theor. Phys. Suppl. 164, 38–53 (2006) 7. Hut, P.: Virtual Laboratories and Virtual Worlds. In: Vesperini, E., et al. (eds.) Proc. IAU Symp. 246, Dynamical Evolution of Dense Stellar Systems, pp. 447–456. Cambridge University Press, Cambridge (2008) 8. The Meta-Institute for Computational Astrophysics (MICA), http://www.micavw.org/ 9. Second Life, http://secondlife.com/ 10. Qwaq Forums, http://www.qwaq.com/ 11. MICA SL island, StellaNova, http://slurl.com/secondlife/StellaNova/126/125/28 12. Hut, P., McMillan, S. (eds.): The Use of Supercomputers in Stellar Dynamics. Springer, New York (1986) 13. Hut, P., Makino, J., McMillan, S.: Modelling the Evolution of Globular Star Clusters. Nature 363, 31–35 (1988) 14. The Art of Computational Science, http://www.ArtCompSci.org 15. The Starlab Project, http://www.ids.ias.edu/~starlab 16. MUSE: a Multiscale Multiphysics Scientific Environment, http://muse.li 17. Hut, P., Mineshige, S., Heggie, D., Makino, J.: Modeling Dense Stellar Systems. Prog. Theor. Phys. 118, 187–209 (2007) 18. OpenSim project, http://opensimulator.org/ 19. SL Data Visualization wiki, http://sldataviz.pbwiki.com/ 20. Bourke, P.: Evaluating Second Life as a Tool for Collaborative Scientific Visualization. In: Computer Games and Allied Technology 2008 conf. (2008), http://local.wasp.uwa.edu.au/~pbourke/papers/cgat08/ 21. Johnson, A., Ames, J., Farr, W.: The MICASim plugin (2008), http://code.google.com/p/micasim/ 22. Farr, W., Hut, P., Johnson, A., Ames, J.: An Experiment in Using Virtual Worlds for Scientific Visualization (2009) (paper in prep.)

Exploring the Use of Virtual Worlds as a Scientific Research Platform 23. 24. 25. 26. 27. 28.

29. 30. 31. 32. 33. 34. 35. 36. 37.

38.

39. 40.

43

ScienceSim project wiki, http://sciencesim.com/ Virtual World Interoperability wiki, http://vwinterop.wikidot.com/ The U.S. National Virtual Observatory, http://www.us-vo.org/ The International Virtual Observatory Alliance, http://www.ivoa.net/ Djorgovski, S.G., Williams, R.: Virtual Observatory: From Concept to Implementation. ASP Conf. Ser. 345, 517–530 (2005) Roco, M., Bainbridge, W. (eds.): Converging Technologies for Improving Human Performance: Nanotechnology, Biotechnology, Information Technology, and Cognitive Science, NSF report. National Science Foundation, Arlington (2002) Bainbridge, W., Roco, M. (eds.): Managing Nano-Bio-Info-Cogno Innovations: Converging Technologies in Society, NSF report. National Science Foundation, Arlington (2005) Second Life Education Wiki, http://simteach.com/ The Immersive Education Initiative, http://immersiveeducation.org/ SciLands Virtual Continent, http://www.scilands.org/ MICA popular lectures series, http://mica-vw.org/wiki/index.php/Popular_Talks Wing, J.: Computational Thinking. Comm. ACM 49, 33–35 (2006) Szalay, A., Gray, J.: The World-Wide Telescope. Science 293, 2037–2040 (2001) Szalay, A., Gray, J.: Science in an Exponential World. Nature 440, 15–16 (2006) Emmott, S. (ed.): Towards 2020 Science. Microsoft Research Publ. (2006), http://research.microsoft.com/ en-us/um/cambridge/projects/towards2020science/ Djorgovski, S.G.: Virtual Astronomy, Information Technology, and the New Scientific Methodology. In: Di Gesu, V., Tegolo, D. (eds.) Proc. CAMP 2005: Computer Architectures for Machine Perception, IEEE Conf. Proc., pp. 125–132 (2005) Bell, G., Hey, T., Szalay, A.: Beyond the Data Deluge. Science 232, 1297–1298 (2009) Farr, W., Hut, P., Ames, J.: An Experiment in Using Virtual Worlds for Scientific Visualization of Self-Gravitating Systems. JVWR (in press, 2009)

Characterizing Mobility and Contact Networks in Virtual Worlds Felipe Machado, Matheus Santos, Virg´ılio Almeida, and Dorgival Guedes Department of Computer Science Federal University of Minas Gerais Belo Horizonte, MG, Brasil {felipemm,matheus,virgilio,dorgival}@dcc.ufmg.br

Abstract. Virtual worlds have recently gained wide recognition as an important field of study in Computer Science. In this work we present an analysis of the mobility and interactions among characters in World of Warcraft (WoW) and Second Life based on the contact opportunities extracted from actual user data in each of those domains. We analyze character contacts in terms of their spatial and temporal characteristics, as well as the social network derived from such contacts. Our results show that the contacts observed may be more influenced by the nature of the interactions and goals of the users in each situation than by the intrinsic structure of such worlds. In particular, observations from a city in WoW are closer to those of Second Life than to other areas in WoW itself. Keywords: Multi-player On-line Games, Virtual Worlds, social networks, complex networks, characterization.

1

Introduction

Virtual worlds are an important emerging form of social media that have recently caught the attention of the research community for their growth, their potential of applications and the new challenges they pose [1,2]. According to the companies responsible for those worlds, as of December 2008, World of Warcraft (WoW) is being played by more than 11.5 million subscribers worldwide and Second Life total residents are more than 16.5 million. Other data suggests that there are more than 16 million players of massively multi-player on-line games (MMOGs), where players control one or more characters in virtual worlds. Not only that, but users spend a significant amount of time on-line: in Q3/2008, residents spent 102.8 million hours in Second Life. Each virtual world fosters the creation of an active market both inside them and in other sites in the Internet, moving billions of dollars in the entertainment industry [3]. The environments provided by such virtual worlds are usually complex, providing a variety of opportunities for players to interact, fight and develop their characters. The virtual worlds are often divided in zones that may represent continents, islands, cities and buildings, where characters must move. Players may be forced to cooperate with others in order to achieve certain goals, and have to F. Lehmann-Grube and J. Sablatnig (Eds.): FaVE 2009, LNICST 33, pp. 44–59, 2010. c Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering 2010

Characterizing Mobility and Contact Networks in Virtual Worlds

45

fight elements of other groups according to the rules of each environment. Even Second Life can be analysed in such a manner, although in that world there are no explicit competitive situations other than those arising in usual social interactions. All the possibilities offered by those environments create a highly complex virtual reality where a variety of characters seek different goals. Although some aspects of the virtual worlds may be quite detached from reality (like the multitude of different forms of intelligent life and the presence of magic forces), other aspects can be quite similar to the real world. After all, characters are controlled by real people, and interactions are often based on rules also existing outside the virtual environments. Information extracted from such virtual worlds may be directly useful to understand the way users behave in them, but can also be applied to other problems. For example, information about user mobility may be used in studies of how viruses spread among people, how information disseminates through their contacts, or how malware may spread among wireless devices carried by them [4]. Our goal in this work is to provide a first analysis of those worlds in terms of the way players move through the game and how they interact. That is achieved through a spatio-temporal analysis of mobility patterns in both worlds. From those patterns, we derive the social networks based on the users’ contact patterns and study them considering the similarities and differences of the two environments. While in Second Life interactions are mostly cooperative, in WoW they also have a competitive nature, leading to mixed behaviors. That difference is visible in some of the results. As previously mentioned, the information we provide here can be useful for those interested in the development and analysis of virtual worlds, as well as an input for experiments that depend on movement and contact data for real people, such as in epidemiological studies or research on mobile networks, for example. In the Sections that follow, we start by discussing related work in Section 2. Section 3 provide a general description of the virtual worlds considered, while Section 4 discusses our approach to monitoring them and deriving the metrics we used. The subsequent Sections that follow present the results of our analysis in terms of mobility patterns and contact social networks. Finally, Section 7 provides some conclusions and discusses future work.

2

Related Work

Virtual worlds have recently become the focus of researchers looking for data that could be used to model real world mobility patterns. The Second Life virtual environment has been monitored to collect information about avatar movements to mirror movement in enclosed spaces [5]. Metrics used included time to first contact, contact time, inter-contact time, and covered distance, among others. They also analyzed the users’ contact network using complex networks metrics such as node degree, network diameter and clustering coefficients. We use similar metrics in this work.

46

F. Machado et al.

Characterization of on-line games has been an interest for some time now, but a lot of effort has been focused on studying the network traffic produced by them, not in understanding the mechanics of their virtual worlds [6,7,8]. In relation to the particular worlds considered in this study, there has been previous work characterizing Second Life and World of Warcraft from the point of view of the users, by collecting traffic in the client applications [9,10], but again with little insight into the virtual worlds themselves. With that in mind, this work is, to the best of our knowledge, the first one to consider two different virtual worlds with different interaction patterns and objectives. It is also the first one to consider the behavior of avatars in World of Warcraft from a social network perspective derived from their contacts.

3

Virtual Worlds: Background

Both environments considered can be seen as examples of massive multi-player on-line games (MMOGs) based on the Role-Playing Game model (RPG). In such games, players perform their roles through their characters in the game, which interact based on behavioral rules defined by the game environment. For the sake of completeness, this Section provides a brief description of both worlds. 3.1

Second Life

In Second Life, each user controls a virtual character (avatar) that can own objects, real estate, stores, etc. Usually there is no concept of game levels, since the game is entirely focused on social interactions. Hierarchies and class divisions are left for the players. Basically, an avatar sets itself apart from others based on its looks and its possessions. Differently from a traditional RPG, there are no clearly stated goals to Second Life, no missions or tasks defined by the game for the users to complete. The idea is just to allow users to interact socially, talking, performing collective activities, or trading, for example. Users can create virtual groups, which are just used to bring together users with common interests, like the appreciation for a certain location, the desire to meet other people or just as a means to make it simpler to keep contact over time. Avatars can become friends with others, leading to an underlying social network, although the environment does not offer tools to build such networks explicitly. The game territory is quite large, being composed by different continents and many islands. All of it is divided in smaller regions called lands, usually in the form of 256 meter-sided squares. Each land has a defined maximum occupancy and is kept associated with a specific server in order to make load distribution simpler. Management of user actions are therefore distributed among the servers. 3.2

World of Warcraft

World of Warcraft (WoW) adheres strongly to the concept of RPG. It takes place in a virtual world divided in large continents, each one with its special


47

characteristics and sub-divisions. In the game, each user can have multiple characters, but can control only one at a time. The goal of the game is, just like in most RPGs, to evolve the characters based on a hierarchy defined by the game and to defeat the enemy, which can be another player or a programmed entity running on the game servers. For that end there are different resources and possibilities, like items that characters can obtain during the game, their professions and special abilities they can develop. To help characters in their quests and facilitate interaction and trade among users, various cities exist in the territories offering supplies, shelter and training for characters. In WoW, each character belongs to one faction, race and class. They must belong to one of the two existing enemy factions, the Horde and the Alliance, bound to fight each other. For that reason, a meeting of characters of different factions cannot be collaborative, but instead must be surrounded by a clear form of dispute. Cities can belong to one of the factions or declare themselves neutral grounds, the only place where members of different factions can meet without open confrontation. The auction houses in such cities can mediate trade between the factions. Continents are divided in zones with different shapes larger but similar to Second Life’s lands in the way they restrict movements between them to a few points of transit. In that way, each zone can be controlled independently of the others. Eastern Kingdoms and Kalimdor are the older continents in the game, while Outlands is a newer continent added during an expansion named the Burning Crusade. There is also the concept of instances, regions of the map that are duplicated to restrict the occupancy to certain groups each time. If various groups go to a certain region to complete a mission, game servers instantiate one copy of that region for each group, in case the goal is to allow each group to work on the mission without affecting the others’ progress. That leads, in practice, to areas with externally controlled populations.

4

Methodology

In order to understand the behavior of characters in WoW and Second Life, we collected data from WoW at different levels, so we could analyse behavior in terms of the large continents, controlled regions (instances) and a city, which we expected to be a region with characteristics closer to those of an island in Second Life. Table 1 shows some general information about the data collected for each of the virtual regions we considered. The headers used for each of the first five columns refer to elements from WoW: main continents (Eastern Kingdoms, Kalimdor, and Outlands), an instance of a region (Instance 18 ), and a city (Stormwind ). The last column refers to Second Life. Rows show, for the duration of the logs, the total number of distinct characters seen in each region, the average and maximum number of concurrent users actually on-line, and the average session length in hours. The two worlds considered differ significantly in their operations, what led to the use of different data harvesting techniques. The details of each process are discussed next.

48

F. Machado et al. Table 1. General information about the collected data

E. K. Characters 1276 Avg. concurrent users 109 Max. concurrent users 340 Avg. session length (h) 1.4

WoW S.L. Kal. Inst18 Outl. SW. 1039 750 611 511 511 105 88 56 109 31 299 225 123 340 49 1.6 1.6 1.4 1.2 0.05

To collect data from Second Life we implemented a client for the game using the libsecondlife library1 . This automated client connects to the server as a player, interacting with the world following a pattern defined by the programmer. For this work, the client moved in large circles around the center or the territory, since it was found that a moving avatar draws less attention. Once the resulting avatar reaches one of the lands it begins receiving information about the general conditions of the land and all other characters in that region (their IDs, their position relative to the land and whether they are online or offline. The client stores that information once every five seconds in a record containing the number of online users in that land at the time, followed by a list with character ID and position for each avatar. The logs used in this work were selected to hold a continuous 24 hour period. The region used was the Dance Island2 , a popular location in Second Life which contains a dance floor and a bar, among other things. Besides the official World of Warcraft (WoW) game servers, there are currently other versions of those servers, developed through reverse engineering, maintained by users around the globe. For this work we used a message log obtained from one of those user-maintained servers for version 3.5 of the game. The log was created by instrumenting the private Mangos server to log every network message received or sent by it over a 24 hour period. That resulted in a 33 GB data log with more than one hundred million messages, being 15 million sent from clients to the server, and approximately 96 million sent by the server. If the server showed any interruption in its execution the period of the fault was removed from the logs and users returned to their activities where they had left them at the moment of the problem, avoiding any impact to the players movements. Coordinates in the WoW messages are relative to the main continents and instances the characters are in, so there is no global coordinate system that can be equally applied to all characters. To take that into account, all the following analysis considered each continent separately. As previously mentioned, we considered the continents Eastern Kingdoms, Kalimdor, and Outland. We also analysed separately one of the major cities in the game, Stormwind, to compare with the results from Second Life, since a city in WoW offered an area more similar to a land than a complete continent. Finally, we also added an instance of a replicated region of the game, identified as Instance 18, where the number of players was controlled by the game server. 1 2

http://www.libsecondlife.org/ http://slurl.com/secondlife/Dance%20Island/


49

An anomaly identified in the game, when compared to the real world, was the presence of different forms of teletransportation 3 . In some of the analysis, we experimented with removing that functionality from character behavior to try to get patterns closer to the real world, since teletransportation would allow them to travel unlimited distances in practically no time, something clearly impossible in the real world. To achieve that, each time a character used teletransportation, disappearing from one location and materializing at another one, we considered that the first character left the game at the earlier position and a new one entered the game at the materialization spot. We also analyzed the movements as they happened originally, with teletransportation. Once data was collected from WoW, we extracted from the log all messages carrying character positions with the ID of the character, its position and the message timestamp. That information was then processed to create a final log with the same format of that created for Second Life, with all active characters’ positions recorded every five seconds. After a single record format was available for both worlds they were processed using the same algorithms to derive information such as covered distances, demographic density and contact events. Contacts were considered to occur whenever two characters were closer than a certain distance r, considered 10 meters in this case. That definition allows us to consider not only direct character interaction but also close encounters, which have been identified in the literature as relevant for multiple purposes, such as epidemiological studies and wireless network interactions [11]. From the contact information we built the network of contacts, one of the main focus of this paper, and derived also a temporal analysis of contacts. For the temporal analysis, we computed time to first contact, the time it took characters to establish their first contact in the environment, contact time, the times characters spent in contact with others, and inter-contact time, the times between two successive contacts by each pair of characters. The results of the analysis of the metrics derived are discussed in the following Sections.

5

Spatio-temporal Analysis

5.1

Spatial Analysis

In this section we analyze and compare character movements in the two worlds, both in terms of distances traveled and demographic densities. Distances traveled. Figure 1 shows distances traveled (both as a probability density function, PDF, and a cumulative probability density function, CDF) for both worlds in log scale, with and without teletransportation in WoW. As expected, based on the dimensions of each area, probability of short travels is higher in Second Life, while distances in WoW with teletransportation may be significantly larger. 3

In Second Life avatars can also use teletransportation, but only between lands. Since we consider only one land, such events were seen as a user leaving the region.

50

F. Machado et al. 1

1

P[X x] (CCDF)

SL - Dance Stormwind instance 18 E Kingdoms Kalimdor Outland

1

0.01 0.001 0.0001

SL - Dance Stormwind instance 18 Kalimdor E Kingdoms Outland

1e-05 1e-06

1e-06 1e-07

51

1e-07 1

10

100

1000 Density

10000 100000

1

10

100 1000 10000 100000 Aggregate density

Fig. 2. Probability distribution, and complementary cumulative distribution of the aggregate demographic density

Figure 2 shows PDF and CCDF (complementary cumulative probability density function) of the aggregate density computed as the number of characters seen at each square. We can see that the PDF for Second Life stays constant for most of the densities, with some oscillation for lower concentrations. WoW, on the other hand, has a much more skewed distribution for all large areas, with a behavior close to a power law for most of the range considered. Stormwind, the city in Wow, being a restricted area, has a behavior closer to that of the Second Life land, although still closer to the general WoW pattern. From the CCDF, we can see that the three continents and the instance in WoW, being larger areas, spent most of the day with no visitors (about 1% of the area had at least one visitor during the period, except for Outland, in which case less than 0.5% of the area was visited. Even in Second Life, more than 50% of the area was not visited according to the log. Again, the curve for the city, Stormwind, is closer to that of Second Life. It might be the case that they would be even closer if their areas were more close to each other. 5.2

Temporal Analysis

To better understand the nature of the interactions in each world, we considered the temporal dynamic of the contacts. The metrics used, time to first contact, contact time and inter-contact time, were discussed in Section 4. Considering the strictly social nature of Second Live, time to first contact and inter-contact times should be shorter and contact time should be longer than for WoW. Second Life users enter the world mostly to socialize, so they seek other people as soon as they get on-line, reducing time to first contact. For the same reason, after they meet a character or a group, they tend to start a conversation in stead of just pass by and go somewhere else. That should be particularly true for Dance Island. As Table 2 and Figure 3 show, that is exactly the case, except in a few cases. Stormwind, being a city, again shares some of the characteristics of Second Life. Cities serve as temporary bases and support facilities, so people tend to

52

F. Machado et al.

Avg

Table 2. Contacts temporal metrics (averages in seconds)

E. K. First contact 2170 Contact time 89 Inter-contact time 384

WoW Kal. Inst. 18 Outland 1943 2695 520 170 316 128 405 435 112

S.L. S.W. 195 163 474 284 1222 387

seek populated places, like markets, banks and training sites once they reach them, leading to early contacts, so they have similar times to first contact. In the city, however, long sessions where players seek to improve their user experience (trading, grouping, training skills, seeking quests, chatting) seems to dominate contact times, making them even longer than for Second Life. Also, after characters part in Stormwind, they take much longer to meet again (if they ever do), as the average inter-contact time indicates. That was mostly due to the nature of the game: once characters part after training or conducting business they tend to leave the city for new quests, returning much later. Both features are also visible in Fig. 3, where we can see that approximately 50% of the intercontact times in Stormwind are longer than 100 seconds, against only 30% in Second Life, and also the longer contact times for Stormwind (roughly 5% are longer than 2.5 hours). Other elements of interest in Table 2 are the lower inter-contact time for Outland and high first-contact times and longer contact times in the Instance18. Those are also explained by the nature of the game. Outland is a continent visited by advanced characters in their quest to improve their rankings even further. In that condition, collaboration with other characters is important and they tend to meet often to exchange information, if for nothing else. That reduces inter-contact time. Instances are mostly places were collaborative game play is essential. Characters usually join outside an instance and enter them together. Once inside, they proceed together (getting closer or farther apart as the situation requires) but with no contacts with characters other than those in their group. We only registered the (eventual) moments when characters get more separated and then get closer again. On the other hand, contact times and inter-contact times capture the together-again-apart-again nature of the action. From Fig. 3 we see that Second Life has fewer short-lived contacts: characters tend to at least try to start a conversation each time they meet, so contacts tend to last at least a little longer (only 20% last less then 30 seconds). On the other hand, in WoW is more common for characters to just pass by others while en route to a farther destination, without ever stopping — although that is, again, a little less common for Outland and Stormwind, for the reasons discussed. In both, there are some short-lived contacts but also some long-lived ones. We can see basically five categories in terms of time to first contact in Fig. 3. Clearly Second Life is the one with lower values (almost 80% of the first contacts happen in less than 8 seconds, while the opposite is true for instance 18 (50% take longer than 4 minutes). Outland and Stormwind, since they have conditions

Characterizing Mobility and Contact Networks in Virtual Worlds 1

SL - Dance Outland Stormwind Kalimdor E Kingdoms instance 18 10

100 1000 10000 Time to first contact (sec)

E Kingdoms Kalimdor instance 18 Outland Stormwind SL - Dance

P[X of the query. The exact query fired is as shown below: Select = (“?f”) where m=GraphPattern([(“?a”,ds[prd1],ds[val1]), (“?b”,ds[prd2],“?f”), (“?a”, “?c”, “?a”) , (“?b”, “?c” ,“?a”)]) result = sparqlGr.query (select, where) where val1= camcorders and prd1= item name and prd2 = rebate. In case the query generation does not fetch an answer then the system traverses the RDF graph. Ontology traversal takes in concepts identified from the input sentence and determines which part of the ontology these concepts satisfy. That is, the concepts could be leaf nodes or some intermediate nodes in the ontology graph. Once this is established, the traversal tries to determine the relationship (direct or inherited) between the concepts identified in the graph structure. For example, if a user wants to know “what is common between DXG 3MP Digital Camcorder - DXG-301V and Apple iPod- 80 GB Video”, the query generation mechanism is

Virtual Retail Store

97

not going to give an answer as it cannot find out the commonality easily, whereas an ontology traversal would give the answer “Both are on discount”. 3.5

A Detailed Example

We consider a Retail Management System for a retail outlet that has a number of products; some promotion offers and that caters to various customer needs. Tables 1, 2, 3 and 4 show a sample data set. An example of the domain ontology follows: ds:Item ds:item id ds:5 ds:5 ds:item name ds:Aiptek IS-DV2 Digital Camcorder. ds:5 ds:item type ds:camcorder. ds:Item ds:item id ds:3 ds:3 ds:item name ds:Canon Digital Camera - SD900. ds:3 ds:item type ds:camera. Table 1. Item Store Item store ID Item ID Store ID Cost amt Discount 4 6 6 4 5 5 4

1 5 7 10 12 3 2

201 200 201 201 200 200 201

500 750 400 1150 2000 200 1600

0 25 15 30 35 10 25

Table 2. Item Item I 1 2 3 5 7 1 2

Item name

Item type

Panasonic Mini DZ Camcorder DXG 3MP Digital Camcorder - DXG-301V Canon Digital Camera - SD900 Aiptek IS-DV2 Digital Camcorder Apple iPod- 80 GB Video Panasonic Mini DV Camcorder Panasonic 2.8” LCD Camcorder SDR-S150 Table 3. Store Store ID 200 201

Store name Nicollete Mall PoundLand

Store addr A123 NYK Udyog Vihar

Camcorder Camcorder Camera Camcorder iPod Camcorder Camcorder

98

G. Sharma, C. Anantaram, and H. Ghosh Table 4. Department Department ID Dept desc 22 23

Electronics Apparel

The table name and the primary key form the subject (Item and 1 are the subjects), the fieldname forms the predicate whereas the values of the fields form the object in the ontology file. Let us assume that a user asks the question, “Which camcorders have more than 20% discount?”. The primary way to answer this question would be query formation and firing one of the seven query templates. In this example it is: Select= (”? f”) where.addPatterns([(”?a”,”?c”,”?a”),(”? a”,ds[prd],”?f”), (”?b”,”?c”,”?a”),(”?b”,”?d”,”?e”),(”?b”,ds[prd2], ds[va l2]), (”?e”,”?d”,”?e”),(”?e”,ds[prd1], ds[val1])]) result= self.sparqlGr.query(select,where) This query when fired fetches the appropriate answer: “The Camcorders are DXG 3MP Digital Camcorder - DXG-301V, Panasonic Mini DV Camcorder, Aiptek IS-DV2 Digital Camcorder, Panasonic 2.8” LCD Digital Camcorder with 3CCD Technology - Silver (SDR-S150)” This answer is then shown to the user.

4

Integrating Virtual Worlds and Business Applications

In this section, we describe how virtual worlds and business applications may be integrated. We use NATAS as an example interface. Virtual worlds and enterprise systems may also be integrated using any other similar interface. To the best of our knowledge, the kind of integration does not currently exist in any of the online virtual worlds. However, one related service available in Second Life is Jnana [2], uses voluntary human experts to advice novices on a particular domain. The expert’s knowledge is uploaded into an interactive question-answer system. When a novice needs to decide about a particular product or service, the system prompts him with questions based on the expert’s knowledge. A series of questions and answers ensues till the novice is able to make an informed decision based on the expert’s advice.While Jnana is very stable though, meaning that the user can usually find what he is looking for, this system has the following drawbacks. Firstly, since the expert knowledge is voluntary, it may not be available on all topics of interest to the novice or it may not be complete to the extent required by the novice. Secondly, questions are asked by the expert rather than the novice. So, the novice cannot control the conversation based on what he wants to know as opposed to what the expert wants to tell him. The NATAS engine, on the other hand, has knowledge about the domain it is being queried on as defined by the domain ontology - the more detailed the


99

ontology, larger is the question set that NATAS can answer. Further, expert comments or reviews may be included, if available. Since the conversation is initiated by the customer, it can be specific to what the customer wants to know, rather than what the system wants to tell the customer. Further, as the interface is in natural language, the customer can phrase the question in his own style rather than having to figure out if the question posed to him answers what he wants to know. 4.1

Integration Mechanisms

Virtual worlds provide tools for building objects or allow graphical models to be uploaded. Scripts or code may run on the models so that they may have a behavior associated with them. For example, a door may be coded so that it opens and closes, a car maybe coded so that it can be driven around. Depending on the underlying architecture of the virtual world, scripts may be written in standard languages like Python, Java or in specially created languages like the Linden scripting language used in Second Life. We use this functionality to link Second Life with NATAS in the context of a retail store. The tasks associated with a business application for a retail store include providing information about products like price, features, discounts and availability, completing transactions for purchase and shipping the product to a real-world destination. Scripts on objects in the virtual world also reside and run on the server. It is possible to script an object to connect to an external web service via hyper text transfer protocol (HTTP). NATAS is available as service enabled on a web server to which external applications can connect. Thus, NATAS can easily be integrated with an object, in this case a virtual assistant, in a virtual world. Note that Second Life is just one example world to which NATAS has been linked. In principle, NATAS can be linked to any virtual world using similar or other mechanisms. We have designed a retail store in Second Life with objects like Cameras, iPods, T-shirts to name a few that can be bought in-world. Further, we have added a robotic sales assistant in the store to answer queries posed by customers on the items in the store. Since the assistant is created using the building tools of Second Life, it is a graphical object within Second Life and does not require a human to be logged on. Thus, assistance is always available to customers who may login across different time zones. Also, the assistant can be programmed so that it moves around with the customer, if he/she so desires, or can stand in one place, answering any questions that the customer might have. The integration of Second Life and NATAS is shown diagrammatically in figure 1. When a user asks a question to the assistant using text chat, the query is extracted and sent as an HTTP request to a web server on which NATAS is running. NATAS connects to the appropriate retail store application, processes the query and formulates the answer. The answer is then sent back as an HTTP response to the virtual world where it is displayed as chat from the virtual assistant. Multiple users can query NATAS at the same time and it maintains context of each conversation.

100

G. Sharma, C. Anantaram, and H. Ghosh

Fig. 1. Broad architecture of Second Life integration with NATAS

(a)

(b)

Fig. 2. Broad architecture of Second Life integration with NATAS

Figure 2 (a) and (b) show some screenshots of a possible interaction with NATAS through the sales assistant (or chatbot). Since the querying is done via HTTP transfers, the response comes within a few seconds so that the conversation takes place in real-time. The interaction with a business system may also be used to create a customized interaction for the user. For example, the look and feel of the retail store may change depending on the profile of the customer. Certain products may be highlighted and others made to disappear completely depending on what the customer is interested in. This information may be extracted from the conversation the user has with the business system. We will discuss this aspect in the dynamic rendering section of this paper. 4.2

Carrying Out Real-World Transactions

The interactions through the virtual world can lead to carrying out concrete tasks and transactions on the business application system, such as “buy a camera”.


101

Such a task or transaction can actually lead to generation of an invoice for the customer and billing activities. An order form can be automatically filled and pushed to the customer (either directly onto a window on the virtual world, or via an offline mode such as an email) to confirm the order he or she has placed. Once the order is reconfirmed and the payment mechanism (such as credit card) is confirmed, the shipping of the product can occur. Thus interactions in the virtual world can lead to actual real-world transactions. 4.3

Dynamic Rendering

Dynamic rendering can help a virtual space to change on the fly depending on feedback from the user. There are different kinds of changes that can be triggered in a virtual space. For example, external changes such as the entire architecture of the building in the virtual space can be changed from say multi-storied to a single floor. Internally, the space can be made to look different- the colors of the walls, layout or presence of objects etc. can all be changed. The experience inside the space can also be changed - for example the same set of objects can be made to behave differently. Dynamic rendering has a number of advantages. From the perspective of the owner of the virtual space, the same piece of land can be used for multiple purposes. For example, one can create a retail store that turns into an insurance information centre, a bank or a space for holding virtual conferences. Thus, dynamic rendering can be used to switch between different domains. Even within a domain, dynamic rendering can be used to highlight items or information that the user may be interested in. From a virtual retail store perspective, for example, users visiting a retail store will be looking for different things - a younger person could be interested in a particular style of clothes or music, while an older person can be interested in another style. Dynamic rendering could help the same store cater to the needs of both customers by rendering it according to customers preference. This has a two-way benefit since the customer sees only what he is interested in, and this cuts down on the time required for him to decide what he wants to buy. The store owner, on the other hand, has more and quicker sales, as a customer does not waste time in identifying what he wants and is aided in quicker decision-making.

5

Role of Visual Semantics in Retail Stores

While natural language based interaction in retail stores provides a powerful shopping paradigm, there are many articles, whose properties cannot be easily articulated and are better illustrated with visual examples. Paintings and ethnic garments are a few examples of such media-rich commodities. Even for many articles of common use, it is often the visual appearance of the package that the buyer tends to remember rather than the detailed product attributes. For example, packages of grocery products, the design of DVD and book jackets help to uniquely identify the products. Thus, there is a need to deal with the

102


semantics that is hidden in the visual appearance of the products and packages, their color and texture, the distinctive product-marks. In this section, we describe two examples that exploit visual semantics. 5.1

Shopping by Example

With ubiquity of high resolution cameras with mobile phones, it is easy to capture the image of an empty carton of a grocery item or the jacket of a DVD or a book. This motivates example based shopping, where the shopper provides a visual example to request the intended product [3]. The overall operation of the system is depicted in figure 3. The product database of an on-line store includes a few image examples of the product packages from different perspectives. The shopper uses his mobile camera or a webcam to take a snap of the product package, which is submitted over MMS / the Internet. A search algorithm operates on the image database and retrieves the closest matching image. The desired product is so identified and the product details are shown to the buyer to make the final purchase decision. In a retail store in a virtual world environment, the avatar of the buyer shares a snap taken in the real world with a seller agent in the virtual store and requests the desired product. In this application, we identify the desired product by the visual appearance of the distinctive product-mark. The low level image features, such as color and texture are not suitable for this purpose. Difficulties also arise from imperfections in the user supplied images, because of imperfect lighting conditions, surface glare and improper alignment of the hand-held camera as well as wrinkles and damages of the used packages. PCA-SIFT [4] provides a robust way to compare the product-marks with keypoints derived from the images and can take care of many of the imperfections. The key-points are sharp and distinctive corners in the visual pattern characterizing the product-mark and can somewhat be compared with keywords in a text segment. Each product image contains an arbitrary number of key-points. Each keypoint is represented as a 128-dimensional vector. The similarity between two key-points is measured by the cosine of the angle between the vectors. The similarity between two images is computed as follows 1. Let K1 = k11 , k12 . . . k1m be the set of key-points in the query image Q and K2 = k21 , k22 . . . k2n be the set of key-points in a product P 2. Let k = 0 (no. of matching key-points in K1 and K2 ) 3. For each member k1i ∈ K1, do a. Let si = 1 (the largest possible value) b. For each member k2j ∈ K2, do i. sij = similarity(k1i , k2j ) ii. if sij < si , then si = sij c. If si > t (threshold), k = k + 1 k 4. Similarity(Q, P ) = |K1| In summary, a key-point in the query image Q is said to match a key-point in the image of a product P, if the similarity between them exceeds a threshold


103

Fig. 3. Shopping by example

(t). The similarity of the query Q and the product P is established in terms of number of matching key-points and is normalized by dividing the number with the cardinality of the key-point set in the query image. The list of products to be shown as the candidate solutions are computed as follows The products are ranked in the decreasing order of similarity. Let si be the similarity value of the i-th product in the ranked list. If sj > λ∗sj+1 (where λ is an arbitrary number), then j is treated as the cutoff point in the list, i.e. the buyer is shown the products till (and including) the j-th product from the top of the list. If s1 > λ ∗ s2, only the first result is shown and the product is said to have been identified uniquely. If j > k (when k is a pre-decided constant), we conclude that the system has failed to identify the product, either because the product has not been in the database, or because extreme aberrations in the query image. PCA-SIFT has the ability to distinguish key-points with great accuracy, and in most of the cases, the algorithm produces a unique and correct result. Figure 4 depicts some such query image examples. It may be noted that the images are distorted, have surface glare and out of focus. The system performs well despite these defects in the input images, which are expected in a real application scenario. Since there is a unique correct result for every query image, we use Mean Reciprocal Rank (MRR) as a performance measure of the system. With a database of more than 1000 products, the MRR of the system is found to be 97%. Shopping by Example has an interesting application in the context of virtual worlds. Suppose while navigating through a virtual environment, a user comes across a real-world or virtual item of interest, for example, a new CD at a friends house. The user may click an image of the item and store it on his hard-disk. Many virtual worlds allow the user to click photos “in-world” using their client software. Even otherwise, a user can use the print-screen facility to take an image of what is being displayed on the screen. This image can be submitted as a query to the SBE system to get more information about the item. Figure 5 shows the usage of the SBE system from Second Life.

104


Fig. 4. Example query images

Fig. 5. SBE in Second Life

6

Conclusion

A next-generation virtual world-based retail store is a distinct possibility in the near future. Such a store can provide the potential retail customer with a variety of mechanisms to interact and select the appropriate product suitable to his or her requirements. Natural language based interactions with a chatbot combined with visual image based search can lead to an easy shopping experience for the customer. With the store dynamically changing its layout and offerings, the customer can get a rich and enhanced shopping experience. The role of semantics in such interactions is important to be addressed, and it is also important to have a framework that delivers such an experience. We have described an innovative syst

References 1. Bhat, S., Anantaram, C., Jain, H.: Framework for text-based conversational userinterface for business applications. In: Zhang, Z., Siekmann, J.H. (eds.) KSEM 2007. LNCS (LNAI), vol. 4798, pp. 301–312. Springer, Heidelberg (2007) 2. http://www.jnana.com 3. Ashish, K., Hiranmay, G., Jagannathan, J.S.: SHOPPING BY EXAMPLE - A New Shopping Paradigm in Next Generation Retail Stores. VISAPP (February 2009)


105

4. Yan, K., Rahul, S.: PCA-SIFT: A More Distinctive Representation for Local Image Descriptors. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2 (2004) 5. 2003 Mall Shopping Patterns, Consumers Spent More Time in the Mall. ICSC Research Quarterly 11(2) (Summer 2004) 6. Answers Anywhere, Sybase Inc. An Application of Agent Technology to Natural Language User Interface 7. Popescu, A.M., Etzioni, O., Kautz, H.: Towards a Theory of Natural Language Interfaces to Databases. In: IUI 2003, Miami, Florida, USA (2003) 8. Yunyao, L., Huahai, Y., Jagadish, H.V.: Constructing a Generic Natural Language Interface for an XML Databases. In: Ioannidis, Y., Scholl, M.H., Schmidt, J.W., Matthes, F., Hatzopoulos, M., B¨ ohm, K., Kemper, A., Grust, T., B¨ ohm, C. (eds.) EDBT 2006. LNCS, vol. 3896, pp. 737–754. Springer, Heidelberg (2006) 9. Ferguson, G., Allen, J.F.: TRIPS: An integrated intelligent problem-solving assistant. In: Proceedings of AAAI 1998, pp. 567–573 (1998) 10. Allen, J.F., Ferguson, G., Stent, A.: An Architecture for More Realistic Conversational System. In: IUI 2001, Santa Fe, New Mexico, USA, January 14-17 (2001)

StellarSim: A Plug-In Architecture for Scientific Visualizations in Virtual Worlds Amy Henckel and Cristina V. Lopes University of California, Irvine, Irvine CA 92697, USA {ahenckel,lopes}@uci.edu

Abstract. More and more researchers in a variety of fields are turning to virtual worlds for 3D simulations and scientific modeling. The use of virtual worlds in this manner offers many benefits. However, the critical task of creating 3D objects for a simulation model is still a manual process, which can be time consuming. Our research concentrates on creating a process that allows for the automatic population of 3D objects in virtual worlds for researchers. This paper presents a plug-in architecture framework that allows the automatic creation of 3D objects and externalizes the behaviors of the objects. This plug-in architecture makes it possible to utilize the underlying framework of the virtual world platform for the display of arbitrary data, in a straightforward manner. A prototype application was created based off this framework, augmenting the 3D platform OpenSim. Keywords: virtual environments, content creation, 3D objects, simulation, modeling, astronomical modeling, OpenSim.

1

Introduction

The National Aeronautics and Space Administration (NASA), has announced its interest in modeling mission data from interplanetary probes [1] [2] and developing a Mars and moon virtual habit [1] in Second Life [3], a popular virtual world. This agency has also announced its interest in importing data from the International Space Station Mission and Mars Mission into Second Life [4] [5]. In the field of astronomy, there are additional research interests for the use of virtual worlds. For example, Piet Hut, Institute for Advanced Study, discusses the benefits of using virtual worlds for collaboration among astrophysicists. He started the group Meta-Institute for Computational Astrophysics (MICA) for this purpose [6]. He discusses using virtual spaces as collaboration tools, allowing users to see visual representations of other users (i.e., avatars), and allowing communication through voice or text with other avatars in real time. He believes this method of communication gives the user a sense of being in the same room as other people, which assists with sharing ideas [6] [7]. In addition to these examples, there exists a large number of research interests in the field of astronomy for the utilization of virtual worlds in simulation and modeling. Due to the tremendous amount of data for stellar bodies and F. Lehmann-Grube and J. Sablatnig (Eds.): FaVE 2009, LNICST 33, pp. 106–120, 2010. c Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering 2010

Plug-In Architecture for Virtual Worlds

107

their movements recorded from interplanetary probes and telescopes [8] [9], we examined the process of creating 3D representations of data (3D objects) used for simulations and modeling in virtual worlds. What we observed was an inflexible and time consuming process. For example, the 3D objects themselves must be manually created, and manually customized. Textures (i.e., images that can overlay a 3D object) are appointed to the object. To order to assign a behavior (i.e., controlled movement, change in appearance, etc.), a script defining the movement of an object has to be created and added to the individual 3D object. We then looked at existing simulation programs used by astrophysicists to discover if they experienced similar problems, to the ones we observed. We gathered information pertaining to the use of these programs though informal discussions and interviews from astrophysicists involved in the analysis and planning stages of the mission life cycle, as these stages make use of simulation programs. The results show there are various programs employed for modeling astronomical data. Each program either produces a simulation for one type of data, or requires extensive programming to visualize multiple types of data. This is due to the different behaviors of an object, various aspects of an observed object, and multiple sources of input for an object. The results also revealed only a few of the programs used allow for the addition or modification of a 3D object; these processes involved are manual and time consuming. Many of the astrophysicists we spoke to express the need to use more than one program for a task due to these restrictions. This paper presents StellarSim, a framework designed to address the weaknesses pointed out above. We show how attributes about a planet (size, texture, name, shape etc.) and modules defining the behavior of a planet can be easily imported into a virtual world. The end result produces simulations consisting of 3D representations of these kinds of data. These representations, or 3D objects, are automatically created and displayed in the virtual world. Our focus was to create a flexible framework for the importing of customized attributes and behaviors of an object into a rendering environment with minimal effort for the user. We refer to this as the automatic population of 3D objects. StellarSim allows for an easy representation of multiple types of data by externalizing the behaviors and attributes of these 3D objects, in particular when applied to astronomy. This paper focuses on StellarSim’s design. Future goals include receiving additional iterative feedback on the design from end-users, adding further requirements and performing a thorough user study in situ. This paper describes some of the motivations, findings, challenges, and future goals.

2

Related Work

Since this framework utilizes virtual worlds in creating a simulation program for astrophysicists, this section will first discuss current 3D simulation programs used in this field, then discuss current research with modeling in virtual worlds.

108

2.1

A. Henckel and C.V. Lopes

3D Programs in Use

The programs in use by this group of astrophysicists for 3D modeling and simulations are: Science Opportunity Analyzer (SOA) [10], Satellite Orbit Analysis Program (SOAP) [11], Java Mission-planning and Analysis for Remote Sensing (JMARS) [12], Solar System Simulator [13], Visualization ToolKit (VTK) [14], and Satellite Tool Kit (STK) [15]. As well, some users have built custom simulation programs for particular missions due to one or more missing requirements from available programs. SOAP and SOA are only available to NASA, JPL, Aerospace Corporation, and affiliates. SOAP is used to assist in projecting and analyzing satellite orbits, including positions and availability of sensors and communication links. It utilizes the Spacecraft, Planet, Instrument, Camera-matrix, and Events toolkit (SPICE) [16] [17], which is produced by NASA. SOA is used to find an ideal time for an observation on a mission. The user can select a specific point in space in which to ”view” the surroundings. JMARS full edition is only available on particular missions. JMARS displays images of 3D terrain and information from stellar instruments, such as maps and image footprints. Solar System Simulator is a website that displays images of projected trajectories of orbiting objects from a defined point of reference and date. STK and VTK are toolkits which are more flexible, and offer more options. However, they require extensive programming for modeling use. VTK is a rendering tool used to produce 3D images and plots, of any type of data. STK is used to calculate position, orientation, view maps and images, check visibility of a sensor, and project trajectories of satellites and probes. Each program has its own advantages and disadvantages. Of the programs listed, none allow for the automatic population of 3D objects from within. The task of creating 3D objects in these programs is a manual process, if it is allowed. Certain programs are restrictive on what an end-user can create. The programs listed, except for STK and VTK, focus on modeling only one type of data. For example, a program either focuses on the detailed terrain of a stellar body, or the projected orbit of a stellar body, but not both. STK and VTK are both toolkits and require extensive programming from the user for modeling. Because of this, most of the users questioned utilize more than one simulation program to accomplish a task. The majority of these users would prefer to utilize one program, mainly to avoid duplicating work. Flexibility is an issue as well. Most users expressed their requirements change from mission to mission. Because of this, the users may switch from one group of programs to another depending on the required tasks for planning a particular mission. One additional observation was the inability for real-time collaboration with these programs. All data files employed by these programs are stored on the user’s PC, and cannot be easily shared or accessed by other users. 2.2

Virtual Worlds

There are many virtual world platforms in existence today and there is much research on simulations and modeling in these virtual worlds. While it is beyond


109

the scope of this paper to discuss and compare the research in all virtual worlds, this paper will discuss the research taking place in the most popular virtual world platform used for simulation models, Second Life [3], and an open source virtual world with similar user functionality, OpenSim [18]. NASA is involved in several projects within Second Life, and looks to virtual worlds for assistance in future missions. Jessy Cowan-Sharp, who helped create NASA’s CoLab island in Second Life [2] [19] [1] sees virtual worlds as a flexible set of tools and useful for building scientifically accurate representations of data from planetary probes. She mentions that collaboration with the members of the virtual world community could add to their tests and sees the collaboration capability of a virtual world as beneficial to this field. Aside from the astrological simulations mentioned, Second Life is widely popular for creating simulation models for demonstrational, pedagogical, and analytical purposes. Examples include a simulation modeling a Personal Rapid Transit system [20], a demonstration showing how ants find food and leave a pheromone trail [21], a heart murmur simulation [22], a hallucination simulation [23], and a genetic model display [24]. The second virtual world platform discussed in this paper, OpenSim, is an open source project, which employs Second Life’s client software to connect to an OpenSim server. For the purpose of this feasibility study, OpenSim proved to offer a more viable solution for our needs than Second Life. Both Second Life and OpenSim were evaluated as a platform for this framework. Second Life had some limitations which prevented our framework from being feasible. OpenSim however produced a feasible and flexible solution. There are additional open source virtual worlds, such as Sun Microsystem’s Wonderland [25] and Darkstar [26], and Croquet [27]. Further studies would be needed to develop and test the operability of the StellarSim framework with such virtual worlds. With the continuous development of 3D virtual worlds, we believe more and more opportunities will arise for further development of simulation models.

3

Usage Scenario

StellarSim provides a method to input customized attributes and assign independent behaviors to 3D objects, which accommodates for greater control over customizing a simulation model on an ad hoc basis. Other modeling applications can be created based off of this framework. This section describes three example scenarios of how StellarSim can be employed. Scenario 1 - Projected Path: Emma is required to calculate the projected path of a shuttle and make adjustments to that path. She has to: (a) input data for the shuttle, (b) increase and decrease the speed or orbit of the shuttle to discover if the projected path will collide with other objects, (c) if the projected path will place the shuttle in the right place on a specified date, and (d) if adjustments are needed, modify the projected path accordingly.

110


Scenario 2 - Collaboration: After calculating the appropriate path of the shuttle, Emma now needs to: (a) share her calculations with a coworker and (b) both will have to make adjustments to the simulation model, as appropriate. Scenario 3 - Change Perspective: Jorge has received specifications on a new mission involving a probe to Jupiter. He will need to (a) input data involving the probe (b) monitor the projected path of the probe from Earth to Jupiter then (c) monitor the projected orbit of the probe around Jupiter up close to see if any other object, i.e., one of Jupiter’s moons, will interfere with the probe’s lens and its predetermined target. Details on the use of StellarSim for these three usage scenarios are described in the Evaluation section. Data Model. Current simulation programs are strongly coupled with the data they represent; behaviors are not dynamically assigned to 3D objects. In order to effectively create 3D models of multiple types of data, attributes (size, shape, texture, etc) and behaviors (controlling factor of an object’s movement) of the objects must be external to the main program. Figure 1 shows an illustration of distinct types of objects that can have representation in StellarSim and their structure within the virtual world platform.

Virtual World Platform Attributes Size, Shape, Texture

Attributes

Planet

Behaviors Movement

Size, Shape, Texture

Probe

Behaviors Movement

Attributes Size, Shape, Texture

Moon

Behaviors Movement

Fig. 1. Representations of stellar objects within StellarSim

The design of StellarSim allows for the externalization of the attributes and behaviors of the 3D objects through the use of configuration files and assemblies in Dynamic-link library (DLL) files. Once the attributes and behaviors are defined, the object appears to behave as expected to the end-user. Dynamics Modeling. Much research and progress have been made to better understand and model our universe. The progress is remarkable, and beyond the scope of this paper to fully address the outcome. There are many possible methods for calculating planetary positions in the virtual world environment. For this framework, algorithms simplified by Paul Schlyter [28], are used to calculate


111

the coordinates of the planets. To reference Paul Schlyter on the accuracy of his algorithms, ”The accuracy of the computed positions is a fraction of an arc minute for the sun and the inner planets, about one arc minute for the outer planets.” [28] This method was chosen for the initial design of the system for the reason that it supports the externalization of the behaviors of objects. DLLs utilizing these algorithms are assigned dynamically to the objects to calculate their positions.

4

Architecture and Implementation

The framework for StellarSim provides a plug-in architecture that applies an object’s attributes and behaviors from external files. The object’s attributes and behaviors are specified in configuration files and C# classes which are independently compiled into DLL files. StellarSim loads those independently developed components and executes them. This allows for the external data and behavior to be instantiated in the generic virtual world. First it allows for greater flexibility in modeling various types of objects. Second, it allows the systemic utilization of the underlying virtual world platform across a variety of applications. This architecture provides numerous benefits for an application design within virtual worlds. The benefits of using OpenSim for our framework include a direct access to the backend of the virtual world server for dynamic additions and modifications of 3D objects, use of the system timer for orbit simulations, and registration of events (discussed in more detail later in this section). The plug-in architecture was implemented using OpenSim Region modules. Region modules are collections of classes that implement the interface IRegionModule (DLLs themselves). There are many Region modules standard with OpenSim. A new Region module, OpenSim.Region.StellarSim (StellarSim module), was created for this application. The StellarSim module reads in attributes for new objects and loads the appropriate behavior modules designed to calculate the object’s position. It then calls on these behavior modules and uses an existing module in OpenSim, OpenSim.Region.Environment (Environment module), to create the 3D objects and update their positions in the virtual world. The StellarSim module also hosts web services used for a web form interface for our prototype application. It listens for http requests and executes appropriate functions for these requests. Figure 2 shows the architecture of the StellarSim framework. The interface IRegionModule, listed below, requires that the functions listed within it are included in all classes which implement this interface. During the initialization of the OpenSim server, the working directory of OpenSim (opensim/bin) and the scriptengines directory (opensim/ScriptEngines) are scanned for DLL files containing classes which implement IRegionModule. Once an appropriate DLL file is located, OpenSim loads this file and executes the Initialise and PostInitialise functions within it. After Region modules are loaded into the OpenSim server, they remain active until the server is shutdown. Region modules are flexible in nature, and can perform a variety of tasks, including creating

112


Files used to specify attributes and behaviors of 3D objects Behavior module (.dll)

Configuration file (.ini) Purpose: To specify the attributes of an object and its behavior class name.

Purpose: Contains behavior classes for objects specified in a configuration file

Format: Comma delimited

Language: C#

OpenSim Server consists of various interlinked modules) OpenSim.Region.Environment module

StellarSim Region module Purpose: Reads in and parses attributes for each object. Loads the specified behavior class for each object. Creates the html page for the web form interface and listens for commands. Contains functions to handle new positions, orbit changes, simulation instance changes. Calls on functions in the Environment module to create and modify the 3D object.

Purpose: Used to create and modify 3D objects in OpenSim. Language: C#

Language: C#

OpenSim

OpenSim Client used to view 3D objects and adjust the simulation) 3D object Web Form Interface Purpose: Allows the user to control aspects of the simulation. Language: html

3D object Purpose: To represent the attributes and behaviors specified.

Purpose: To represent the attributes and behaviors specified. Platform: OpenSim

Platform: OpenSim

Fig. 2. Diagram of the architecture of StellarSim


113

objects, modifying objects (such as updating the positions of objects), using the system clock timer and registering for events (i.e., listening to chat messages, user logins, http requests, texture transfers, etc.). Registration for events allows an action to occur within the registered module in response to an event. IRegionModule interface public interface IRegionModule { void Initialise(Scene scene, IConfig config); void PostInitialise(); void Close(); string Name { get; } bool IsSharedModule { get; } } The StellarSim module includes a new interface, IAstronomicalModule. IAstronomicalModule is designed to be implemented by external behavior modules for specifying the position of a 3D object. The StellarSim module reads in attributes from comma delimited text files with the extension of .ini (configuration files). These files reside under the StellarSim main directory (opensim/bin/StellarSim). For each object listed in a configuration file, a class name must be provided. This referenced class must implement the IAstronomicalModule interface and exist in a DLL file under the StellarSim lib directory (opensim/bin/StellarSim/lib). The StellarSim module loads the specified class and associates it with the object’s attributes from the configuration file. It then uses the Environment module to create a 3D object based on the provided information. Once the 3D objects are created, they can be viewed via logging into the virtual world. Listed below is the format of the configuration file, and the IAstronomicalModule interface. Examples using these files are shown in the Evaluation section. Format of the configuration file, *.ini ObjectName, ClassName, Size.x, Size.y, Size.z, Shape, Texture The IAstronomicalModule interface implemented by classes defining movement of a 3D object using System; using OpenMetaverse; namespace OpenSim.Region.StellarSim.Interfaces { public interface IAstronomicalModule { Vector3 PositionFromDate(DateTime date); } } By using C# interfaces in this manner, certain functionality is then guaranteed to exist in loaded modules. For example, the IAstronomicalModule interface requires that the function ’Vector3 PositionFromDate(DateTime date)’ exists in

114


an object’s behavior module. This ensures that the StellarSim module can call on a function PositionFromDate from a specified class, give it a date (in DateTime format), and receive a position vector (in Vector3 format). After the position vector is received, the StellarSim module scales the information appropriately to fit within the simulation region limits. It then calls on the Environment module to update the position of the 3D object. The orbit of a 3D object in the virtual world is controlled here by continuously updating that object’s position. The OpenSim system timer is used; the object’s position is recalculated and adjusted every second. Next, the StellarSim module registers for http request events. After a request is made through the web form interface, the StellarSim module will call on appropriate functions within itself to respond to the request. The end result is as such: the end-user can launch the web form interface of StellarSim and control the objects in the simulation. Figure 3 shows StellarSim’s web form interface through the virtual world client. (This interface can also be used through a web browser.) For example, to view the objects on a particular date, a user makes a request from the web form interface and the StellarSim module updates the objects’ positions based on information from the behavior modules. Using Region modules allowed for the dynamic additions of and modifications to the 3D objects, use of the system timer for the orbit simulation, and the registration of events. By using this approach, the creation and modification time of each object is relatively small. This allows for a smooth simulated orbit. The code for StellarSim is written in C# with 501 lines of code for the main module (OpenSim.Region.StellarSim), 7 lines of code for the interface class (IAstronomicalModule), and 735 lines of code for two example simulations (described next).

5

Evaluation: StellarSim

This section first shows two example simulations implemented with StellarSim. The first example simulation displays the planets in the solar system. The second example simulation displays Jupiter and its moons. These two simulations are created in the same region. When switching between simulations all 3D objects of the previous simulation are deleted then all 3D objects of the new simulation are created in the same space. Next, this section discusses previously defined usage scenarios and their application with these two example simulations. 5.1

Applications

Simulation 1 - Solar System: There are many 3D simulation programs that model the planets in our solar system, as this is a necessity for planning a mission in our solar system. The Solar System simulation was implemented by adding a configuration file and a DLL file. Shown below are sections of the configuration file and class library


115

files used to create the DLL file that will specify the attributes and behaviors, respectively, of the 3D objects in this simulation. SolarSystem.ini Mercury,...,http://maps.jpl.nasa.gov/pix/mer0muu2.jpg Venus,...,http://maps.jpl.nasa.gov/pix/ven0ajj2.jpg Earth,...,http://maps.jpl.nasa.gov/pix/ear0xuu2.jpg Mars,...,http://maps.jpl.nasa.gov/pix/mar0kuu2.jpg Jupiter,...,http://maps.jpl.nasa.gov/pix/jup0vss1.jpg Saturn,...,http://maps.jpl.nasa.gov/pix/sat0fds1.jpg Uranus,...,http://maps.jpl.nasa.gov/pix/ura0fss1.jpg Neptune,...,http://maps.jpl.nasa.gov/pix/nep0fds1.jpg Sun,...,http://solarviews.com/raw/sun/suncyl1.jpg Examples.SolarSystem:Behavior.cs using System; using OpenSim.Region.StellarSim.Interfaces; using OpenMetaverse; namespace Examples.SolarSystem{ public class Earth : IAstronomicalModule{ #region IAstronomicalModule Members Vector3 IAstronomicalModule.PositionFromDate(...){ Planet earth = new Planet(); int d = earth.convertTime(date); Vector3 newPos = earth.CalculateEarthPosition(d); return newPos; } #endregion } public class Sun : IAstronomicalModule{ ... } ... public class Neptune : IAstronomicalModule{ ... } } Examples.SolarSystem: Planet.cs shows sections of the class ”Planet”, which was used in Examples.SolarSystem:Behavior.cs, listed above. Combined, they return a position in Vector3 format for any object they define when given a Julian date. Examples.SolarSystem:Planet.cs using System; using OpenMetaverse; namespace Examples.SolarSystem{ public class Planet{ public Vector3 CalculateSunPosition(int d){ ...

116


Vector3 sunPos = new Vector3((float)sunx,...); return sunPos; } public Vector3 CalculateMercuryPosition(int d){ ... calculateXYZ(...); Vector3 planetPos = new Vector3((float)xeclip,...); return planetPos; } public Vector3 CalculateEarthPosition(int d){ ... } ... public Vector3 CalculateNeptunePosition(int d){ ... } public int convertTime(DateTime date){ ... } } } Remark on Scale. To accurately display a model of our solar system to scale, allowing the smallest planet Mercury the smallest representation possible in OpenSim, 73 regions of virtual land in diameter are required for a full orbit around the sun for the farthest planet, Neptune. For the sake of this example simulation (and to view more than one planet in a screen shot), the distances between the planets have been scaled down. Simulation 2 - Jupiter and moons: To switch from a general view to a detailed view, the web form interface is used. Figure 3 shows the web form interface and 3D objects in the Jupiter simulation. The moons shown are: Io, Europa, Callisto, and Ganymede. The Jupiter simulation was implemented in the same fashion as the Solar System simulation, with a configuration file and a DLL file. Shown below are sections of these files. Jupiter.ini Callisto,...,http://solarviews.com/raw/jup/callistocyl2.jpg Europa,...,http://solarviews.com/raw/jup/europacyl2.jpg Ganymede,...,http://solarviews.com/raw/jup/ganymedecyl2.jpg Io,...,http://solarviews.com/raw/jup/iocyl2.jpg Jupiter,...,http://solarviews.com/browse/jup/jupitercyl1.jpg Examples.Jupiter:Behavior.cs using System; using OpenSim.Region.StellarSim.Interfaces; using OpenMetaverse; namespace Examples.Jupiter{ public class Jupiter : IAstronomicalModule{ #region IAstronomicalModule Members Vector3 IAstronomicalModule.PositionFromDate(...){


117

Planet Jupiter = new Planet(); int d = jupiter.convertTime(date); Vector3 newPos = jupiter.CalculateJupiterPosition(d); return newPos; } #endregion } public class Callisto : IAstronomicalModule{ ... } ... }

Fig. 3. Image from the virtual world client showing Jupiter and its moons with StellarSim. StellarSim’s web form interface is shown on the right

5.2

Usage Scenarios with StellarSim

Scenario 1 - Projected Path: (a) To input new data for a shuttle within the Solar System simulation, Emma can create or modify a behavior module adding a class which implements the interface IAstronomicalModule. Next, information on the shuttle’s attributes and a reference for the new class is added to the configuration file under OpenSim/bin/StellarSim for the Solar System simulation (SolarSystem.ini). Then, by using the web form interface and selecting the instance ”Solar

118


System” Emma can now see the new shuttle along with the planets configured in the simulation. (b) To increase and/or decrease the speed of the shuttle, Emma will again use the web form interface and select ”Increase Orbit” or ”Decrease Orbit” accordingly. She can then view the shuttle with respect to other objects within the Solar System simulation to look for any potential collisions. (c) To align objects in the simulation corresponding to a particular date, Emma can use the web form interface and enter a date under ”Realign objects for a Date:”. (d) If any adjustments are needed, Emma can modify the behavior module for the shuttle then reselect the Solar System simulation and her new changes will take effect immediately. Scenario 2 - Collaboration: (a) Emma can share her simulation with anyone who has access to log into the OpenSim server hosting the simulation. (b) Both Emma and her coworker can modify the behavior module for the shuttle, reselect the Solar System simulation and see their changes immediately. Scenario 3 - Change Perspective: (a) Jorge can input new data about the probe in the same manner as Emma in scenario 1. (b) By viewing the Solar System simulation, and using functionality listed in scenario 1, Jorge can follow the projected path of the probe to Jupiter. (c) To switch to a more detailed view of Jupiter and its moons, Jorge can either add a new behavior module for the shuttle to depict its movement in orbit around Jupiter or use the behavior module from the Solar System simulation. Next he can modify the configuration file under OpenSim/bin/StellarSim for the Jupiter simulation (Jupiter.ini), adding a line for the shuttle’s attributes and referencing the desired behavior class name. By using the web form interface and selecting the instance ”Jupiter and its Moons” he can now see Jupiter in a more detailed view and monitor if one of Jupiter’s moons will interfere with the probe’s objective. 5.3

User Feedback

Our prototype application was shown informally to several astrophysicists from our user group and a couple of suggestions came up after. The first suggestion was to add the ability for the user to obtain a set of real rectangular coordinate points, for any point on the screen. Currently the rectangular coordinates shown through the virtual world client refer to a location within an area in the virtual world and not the rectangular coordinates that correspond to a location within the space being simulated. The second suggestion was to add SPICE [17] toolkit to the backend of the StellarSim framework. Currently its use is implemented with the programs SOA and SOAP. Implementing this within StellarSim is feasible and discussed in the next section.

6

Conclusions and Future Work

Virtual worlds are being used in research for simulations and modeling more and more. The advantages of these virtual worlds make them attractive for modeling and simulations.


119

In view of the increasing interest from the field of astronomy to utilize virtual worlds in simulations and modeling, and the large amounts of data typically involved with this field, we looked at the process of creating 3D objects in virtual worlds. We found this process to be arduous. We then looked to the existing simulation and modeling programs used by astrophysicists to see if a more automated process existed there. Not only did we find a similar problem among current simulation programs, but we also discovered these programs had further limitations including the lack of structure to enable collaboration with others. This particular problem is remedied through the use of a virtual world; other issues are addressed through the use of StellarSim. The framework of StellarSim was designed to be flexible in nature, utilizing the plug-in modular structure of OpenSim. It allows for the automated process of 3D object population and ad hoc modifications to the 3D objects. By externalizing the attributes and behaviors of 3D objects, this framework generates an application independent of the type of data being modeled which in turn makes the application usable for more than one type of data. The application, StellarSim, was designed for the use of astrophysicists during the analysis and planning stages. It is currently a prototype and online connected to UCIGrid. Future versions of StellarSim can implement additional functionality features. Features such as obtaining real rectangular coordinate points and adding the use of SPICE in the backend of the framework. Implementing the SPICE toolkit would allow for the use of SPICE available functionality within StellarSim which includes the use of SPICE-hosted ephemeredes (tables of values that provide positions of astronomical objects at a given time) in determining the movements of 3D objects. This functionality would allow for a greater accurancy in computed positions of planets, moons, probes, satellites, etc. Our framework presented here can be extended to other fields, and the prototype application for StellarSim can be modified to incorporate additional functionality. This approach utilizes an open source virtual platform to produce realtime 3D models of planetary objects. This framework provides instant shared access to a 3D simulation created in real-time and facilitating collaborative tools that enable scientists to review and discuss these simulations.

References 1. Boyle, A.: Virtual-space gurus build final frontier (March 2007), http://www.msnbc.msn.com/id/17841125/ 2. Holden, K.: NASA dreams of an interplanetary ‘Second Life’ for mars crew. Wired (January 2008) 3. SecondLife, http://www.secondlife.com 4. David, L.: NASA ames’ Second Life blends cyberspace with outer space (May 2007), http://www.space.com/adastra/070526_isdc_second_life.html 5. Taran. NASA in SecondLife: Plans for a synthetic world in 2007 (November 2006) 6. Hut, P.: Virtual laboratories and virtual worlds. In: Proceedings of the International Astronomical Union, 3(Symposium S246), pp. 447–456 (2007)

120


7. Hut, P.: Virtual laboratories. Progress of Theoretical Physics 164, 38 (2007) 8. Schechter, B.: Telescopes of the world, unite! a cosmic database emerges. The New York Times (May 2003) 9. Sloan digital sky survey, http://www.sdss.org/ 10. Streiftert, B.A., Polanskey, C.A., O’Reilly, T., Colwell, J.: Science opportunity analyzer – a multi-mission approach to science planning (March 2003) 11. Stodden, D.Y., Galasso, G.D.: Space system visualization and analysis using the satellite orbit analysis program (soap), vol. 1, pp. 369–387 (February 1995) 12. JMARS, http://jmars.asu.edu 13. Solar System Simulator, http://space.jpl.nasa.gov/ 14. The Visualization ToolKit (VTK), http://www.vtk.org 15. Satellite ToolKit (STK), http://www.stk.com 16. SPICE toolkit, http://naif.jpl.nasa.gov/naif/toolkit.html 17. Acton, C.H.: Ancillary data services of nasa’s navigation and ancillary information facility. Planetary and Space Science 44(1), 65–70 (1996); Planetary data system 18. OpenSim, http://opensimulator.org 19. CoLab Virtual Overview - NASA CoLab, http://colab.arc.nasa.gov/virtual 20. Lopes, C., Kan, L., Popov, A., Morla, R.: PRT simulation in an immersive virtual world. In: SIMUTools 2008, First International Conference on Simulation Tools and Techniques for Communications, Networks and Systems, Marseille, France (March 2008) 21. Ant simulation in Second Life, http://andrewcantino.com/sl/ants/ 22. CDB Barkely. Heart murmur sim, assessment of learning in sl- interview with a man in a surgical mask (September 2006), http://sl.nmc.org/2006/09/25/jeremy-kemp/ 23. Yellowlees, P.M., Cook, J.N.: Education about hallucinations using an internet virtual reality system: A qualitative survey. Acad. Psychiatry 30(6), 534–539 (2006) 24. Mesko, B.: Genetics in Second Life (April 2007), http://scienceroll.com/2007/04/11/genetics-in-second-life/ 25. Project Wonderland, https://lg3d-wonderland.dev.java.net/ 26. Project Darkstar, http://projectdarkstar.com/ 27. Croquet Consortium, http://opencroquet.org 28. Schlyter, P.: Computing planetary positions - a tutorial with worked examples, http://www.stjarnhimlen.se/comp/tutorial.html

Formalizing and Promoting Collaboration in 3D Virtual Environments – A Blueprint for the Creation of Group Interaction Patterns Andreas Schmeil1,∗ and Martin J. Eppler2 1

Faculty of Communication Sciences, University of Lugano (USI), Via Buffi 13, 6900 Lugano, Switzerland [email protected] 2 mcm – Institute for Media and Communications Management, University of St. Gallen, Blumenbergplatz 9, 9000 St. Gallen, Switzerland [email protected]

Abstract. Despite the fact that virtual worlds and other types of multi-user 3D collaboration spaces have long been subjects of research and of application experiences, it still remains unclear how to best benefit from meeting with colleagues and peers in a virtual environment with the aim of working together. Making use of the potential of virtual embodiment, i.e. being immersed in a space as a personal avatar, allows for innovative new forms of collaboration. In this paper, we present a framework that serves as a systematic formalization of collaboration elements in virtual environments. The framework is based on the semiotic distinctions among pragmatic, semantic and syntactic perspectives. It serves as a blueprint to guide users in designing, implementing, and executing virtual collaboration patterns tailored to their needs. We present two team and two community collaboration pattern examples as a result of the application of the framework: Virtual Meeting, Virtual Design Studio, Spatial Group Configuration, and Virtual Knowledge Fair. In conclusion, we also point out future research directions for this emerging domain. Keywords: group interaction, patterns, embodied collaboration, presence, virtual worlds, MUVE, CSCW, blueprint, framework.

1 Introduxction An ideal online, three-dimensional virtual environment would provide a space in which users can move freely, interact intuitively with all kinds of objects, recognize familiar people, and communicate in a natural manner with them – all in the most realistic look-and-feel setting, evoking a feeling of being part of the virtual world. In addition to that, it would allow displaying complex content or data in innovative and useful ways, neglecting the limitations imposed by physical reality. Such an environment holds the promise of moving remote collaboration and learning to another level of quality. But even if such platforms were available today (and they soon will be): without the right kind of dramaturgy, script or setup, users would not know how to best benefit from their infrastructure. ∗

Corresponding author.

F. Lehmann-Grube and J. Sablatnig (Eds.): FaVE 2009, LNICST 33, pp. 121–134, 2010. © Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering 2010

122

A. Schmeil and M.J. Eppler

We believe that today’s available online virtual environments are already capable of adding significant value to collaborative work and collaborative learning. However, companies, institutions as well as educators may not know how to utilize the spatial characteristics of these environments to the fullest. Moreover, many of the virtual environments that are currently (early 2009) being advertised as offering great productivity boosts for collaborative work emphasize on the collaborative editing of text documents, spreadsheets and presentation slides that are mounted on big walls – a method of working together that would work just as well (or better) without gathering in a three-dimensional virtual space. Our premise, consequently, is that the main two features of 3D virtual environments, namely being embodied in an immersive environment, and the environment being configurable at will, allow for new, innovative, and valuable forms of working and learning together. With our research we aim at improving collaboration in these virtual environments or virtual worlds following these steps: • • •

systemizing and formalizing the necessary elements for visual collaboration developing and identifying novel and existing collaboration patterns, and describing them in the developed formalism evaluating their effectiveness experimentally and comparing them (in terms of added value) to other collaboration arrangements

In this paper, we focus on steps one and two and present a framework for embodied collaboration in online 3D virtual environments, based on semiotics theory, as well as an overview on virtual collaboration patterns. Our framework represents a blueprint of how collaborative group interaction patterns in virtual environments can be described or generated. We also present four examples of the application of the framework, resulting in four online collaboration patterns. We believe this framework to form a first important step in the process of formalizing collaboration in virtual environments – a task that is crucial in order to put forward the application of 3D virtual environments for serious and productive uses. The remainder of this paper is structured as follows: First, we define online virtual environments and present their advantages for collaboration. In section 3, we then present a blueprint to formalize the design elements and necessary infrastructure of collaboration patterns in such environments. In section 4, we provide real usage examples of collaboration patterns based on virtual embodiment. In section 5 we highlight future research avenues for this domain. We conclude the article with a review of our main contribution and its limitations.

2 Online Multi-user Virtual Environments Virtual environments in general attempt to provide an environment where the user or spectator feels fully immersed and present. This presence is a psychological phenomenon that has been defined as the sense of being there in an environment. Immersion, on the other hand, describes the technology of the virtual environment and its user interface that aims to lead to the sense of presence. It can be achieved to varying degrees, stimulating a variable number of human senses. However, the expression of feeling immersed is often also used for online, desktop-based, virtual environments that are controlled only by keyboard and mouse and address only two sensory channels: the visual and auditory one.

Formalizing and Promoting Collaboration in 3D Virtual Environments

123

This kind of virtual environment, featuring multiple users to be in the same shared virtual space at the same time, has been named Online 3D Multi-User Virtual Environment, or MUVE for short. While formal definitions are generally rare in this area, a MUVE is agreed to be a special type of a Collaborative Virtual Environment (CVE). In the ongoing scientific discourse in the research community, a Virtual World, commonly understood as a special type of MUVE, has recently been defined as “a synchronous, persistent network of people, represented as avatars, facilitated by networked computers” [2]. Our research only regards MUVE and Virtual Worlds as opposed to locally installed multi-user VR systems, for the following two reasons: First, the major benefit of utilizing 3D virtual environments is widely believed to be the possibility to have instant team or group meetings without travel. Second, serious collaboration in and between companies is not likely to take place in Immersive Virtual Reality centers (due to availability, accessibility, costs, complexity, and constant need for technical staff). To date, there is an abundance of MUVE and Virtual Worlds available, for all age groups and for many different areas of interest. The Virtual Worlds consultancy K Zero keeps informative graphs up-to-date on their company website1. While systems like Second Life, OpenSim and Activeworlds enable users to design their worlds and to create static and interactive content themselves, others like Sun’s Wonderland and Qwaq Forums focus on productivity in conventional tasks like the editing of text documents, spreadsheets and presentation slides; only up-/download of documents and repositioning of furniture is possible in these latter worlds. Still others focus on providing training scenarios. New MUVE and Virtual Worlds are launched almost monthly, and it seems like each new one tries to fill another niche. Nevertheless, for most application domains, it is still unclear what value MUVE might add to the existing modes of communication and collaboration, just as it remains unclear which features and enhancements are needed to maximize the benefit of using virtual worlds [1]. In a previous paper, we have discussed the advantages (and potential risks) that collaborative virtual worlds bring for knowledge work and education – which are by definition also valid for MUVE [17]. In this paper, we try to define more specifically how these advantages can come about.

3 A Blueprint for the Creation of Collaboration Patterns As already stated as our premise, we believe that the fact of being embodied in a configurable three-dimensional virtual environment allows for innovative, valuable new forms of working and learning (and also playing) together. Embodiment terms the coalescence of recent trends that have emerged in the area of Human-Computer Interaction (HCI) and reflects both a physical presence in the environment and a social embedding in a web of practices and purposes [7]. It is in the same manner applicable to group interaction in MUVE, as users feel immersed in the virtual environment and present in the same setting with their colleagues or peers (co-presence). With configurable we mean the possibility of creating or uploading and editing or modifying interactive objects in the virtual environment. 1

http://www.kzero.co.uk [last access 11/02/2009]

124


While there has been research on the feasibility and usability of embodied conversational agents in Virtual Reality (VR) applications [15], and also on presence and copresence in VR [19], it is yet to be investigated how embodiment in online virtual environments affects group interaction and collaborative tasks. Manninen states that “the successful application of a social theory framework as a tool to analyze interaction indicates the importance of joining the research effort of various disciplines in order to achieve better results in the area of networked virtual environment interactions.” [12]. His work and results will be discussed in more detail in subsection 3.3. The approach we are presenting in this paper is also of interdisciplinary nature – in particular, we combine communication theory and insights from the field of HCI. The resulting framework presents a systematic view on the field of Multi-User Virtual Environments (MUVE) and their utilization for collaborative tasks. As such it represents a blueprint on which diverse collaboration tasks, such as planning, evaluation, decision making or debriefing can be designed and executed. It is based on the underlying distinctions of semiotics and employs concepts from the HCI research field. We present it in detail and discuss its use in 3.5. In the following, we first describe the various steps that we have taken in developing the framework. 3.1 Using Patterns for the Description of Virtual Embodied Collaboration We have realized the need for a solid formal framework that is capable of describing collaboration in MUVE in all its aspects while identifying group interaction patterns of collaborative work and learning in the virtual world Second Life [17]. The pattern approach is a useful and concise approach to classify and describe different forms of online collaboration. Manninen states that the utilization of real-world social patterns as basis for virtual environment interactions might result in usable and acceptable solutions [12]. An alternative approach to using patterns would be to describe collaborative situations as scenarios. A scenario is an “informal narrative description” [6]. However, comparing this with the definition of patterns, a “description of a solution to a specific type of problem” [9], reveals that the pattern concept has been contrived with more focus to solve a problem or to reach a goal. In addition to that, a look at the work of Smith and Willans, who implement the concept of scenarios for requirements analysis of virtual objects [21], makes it clear that the scenario-based approach is too finegrained and at a too low, functional level to describe whole collaborative tasks in flexible multi-user settings. Hence, we have decided to use the pattern approach. We adapt the collaboration pattern definition from [9] by adding the notions of tools and a shared meeting location, to give us the following definition: A collaboration pattern is a set of tools, techniques, behaviors, and activities for people who meet at a place to work on a common goal, together in a group or community. How exactly this definition fits with the resulting framework will be explained by means of an illustration in 3.5. 3.2 The Semiotic Triad as an Organizing Structure From a theoretical point of view, one can conceive of collaboration activities as interpretive actions and of collaboration spaces as sign systems in need of joint interpretation. Visual on-screen events in virtual spaces have to be interpreted by users of


125

MUVE as relevant, meaningful, context-dependent signs that contribute towards joint sense-making and purposeful co-ordination. As in any sign interpretation system or (visual) language, semiotic theory informs us that three different levels can be fruitfully distinguished, namely the syntactic, semantic and pragmatic ones [8]. This threefold distinction has already been applied effectively to various forms of information systems or social online media (e.g. [18]). These three distinct interpretive layers can be applied as follows to immersive virtual worlds: The syntactic dimension contains the main visible components of a collaboration pattern and its configuration possibilities. The syntactic dimension ensures the visibility and readability of a collaboration pattern. It provides the necessary elements as well mechanisms to use elements (digital artifacts and actions) in combination. The semantic dimension refers to the acquired meaning of elements and to the conventions used in a collaboration pattern. It outlines which operations or artifacts assume which kind of meaning within a collaboration pattern. While the syntactic dimension tells the user how to use a collaboration pattern (and with which elements or actions), the semantic dimension aligns the available visual vocabulary to the desired objectives or contexts. In this sense the semantic level is a liaison layer between the virtual world and the participants’ objectives. The pragmatic dimension reflects the social context of the participants, and their practices, goals and expectations. It is these intentions that need to be supported through the dramaturgy (semantic dimension) and the infrastructure (syntactic dimension). This dimension clarifies in which situations which type of dramaturgy and infrastructure use makes sense. 3.3 Action and Interaction in 3D Virtual Environments In our understanding, the support of action and interaction forms one major part of a virtual environment’s infrastructure. It determines how users can act and affects their behavior in both lonely jaunts and in group settings. Moreover, the way users can control their avatars and perform actions heavily influences the level of satisfaction of the user and thus in the end determines whether or not collaborative work or other planned tasks in the virtual environment succeed or fail, continue or are abandoned. We believe that a formalization of action and interaction in virtual environments on a high abstraction level is required. Manninen successfully applied a social theory framework to create a taxonomy of interaction, resulting in a classification of eight categories: Language-based Communication, Control & Coordination, Object-based Interactions, World Modifications, Autonomous Interactions, Gestures, Avatar Appearance, and Physical Contacts [12]. However, this classification is based on studies in multi-player online action and role-playing games, where different requirements regarding interaction must be assumed than for serious collaborative tasks. Also, the study might have focused too much on a language-centered perspective and neglected some of the genuinely visual aspects of virtual worlds. In the field of Human Computer Interaction there is a generally accepted distinction among navigation and manipulation techniques. Navigation techniques comprise moving the position and changing the view. Manipulation techniques designate all interaction methods that select and manipulate objects in a virtual space. In some cases, the side category System Control is used, consisting of all actions that serve to

126


change a mode and modify parameters, as well as other functions that alter the virtual experience itself. Bowman and colleagues refine this classification by adding a category Symbolic Input for the communication of symbolic information (text, numbers, and other symbols) to the system [5]. For our purpose of formalizing (inter)actions for collaboration, we build on this classification and make the following adjustments to align it with the requirements of the area of Online 3D MUVE: The importance of communicating text, numbers, symbols, and nowadays also speech to the system (and thus to other avatars or users, interactive objects, or the environment itself) has increased significantly. We call this first category Communicative Actions. A sub-division differentiates between verbal (i.e., chatting) and nonverbal communication (i.e., waving). Having both navigation techniques and methods for changing the view in one shared category, results from the fact that HCI and VR systems do not necessarily assume the existence of an avatar as a personalization device in the virtual environment; without this embodiment, navigating and changing the viewpoint can be considered as one and the same action. In our classification, changing one’s view would fall into the communicative actions category, as a non-verbal form of letting others know where the user’s current focus of attention is, or to communicate a point or object of interest to others in the virtual environment (the primary purpose of changing the view can be disregarded here, since it is only the actuating person who experiences the change). As a result, our second category, Navigation, comprises only walking, flying or swimming, and teleporting (in the nomenclature of Second Life). We rename the manipulation techniques category as Object-related Actions. Actions referring to the creation or insertion of virtual objects also belong to this category, along with selection and modification techniques. By insertion we mean the result of uploading or purchasing virtual objects, for instance. All system control actions are much less important in MUVE than they are in classic Virtual Reality systems. Due to the often customized or prototype forms of VR applications, system control is in many cases developed and tailored to only one application. In MUVE, by contrast, the viewer software (i.e. the client application to enter the virtual environment) is usually standardized and provides a predefined set of system control options. Hence, we dispense with a system control category. If one were to put these actions on a continuous spectrum, they could also be distinguished in terms of their virtual world effects or level of invasiveness or (space) intrusion. Chatting or changing one’s position, avatar appearance, or point of view is far less intruding than moving an object, triggering a rocket, or blocking a door. Further, it has to be noted that these distinctions and the resulting classification do not include virtual objects. These, in our view, require a separate classification that takes their manifold types and functions into account. In the following subsection, we discuss this important element of virtual environments. 3.4 A Typology of Objects in Virtual Environments In his successful book The Design of Everyday Things, Donald Norman postulates that people’s actions and human behavior in general profits from everyday objects being designed as to provide affordances, i.e., they should communicate how they should be used [13]. He argues that less knowledge in the head is required (to perform


127

well) when there is, what he calls, knowledge in the world. This insight can be fruitfully applied to virtual worlds by building on latent knowledge that users have and by providing cues that reuse appropriate representations [20]. This not only gives motivation for practitioners to utilize virtual environments for collaborative tasks, but implies that objects in virtual environments and their design are of great importance. Hence, we understand virtual objects as to form another major part of a virtual environment’s infrastructure. Affordances can (and should) be used to signal users how to interact with a particular object, or how objects with built-in behaviors may act without any direct influence from the user’s side. Fact is, however, that for a long time researchers active in virtual environments have focused largely on graphical representation and rendering issues. With the launch (and most of all with the hype) of Second Life, a new era of accessible online virtual environments has begun. Following the trend of enabling users to create content (also a vital element of Web 2.0), users of many MUVE can now create and edit objects, and customize the appearance of their avatars. With the possibility of scripting objects, they have become a powerful instrument in designing memorable user experiences in MUVE. In fact, interactive virtual objects represent technology in virtual environments; without active and interactive objects, any virtual environment would be nothing more than a virtual version of a world without technology. This comparison might illustrate the need for a formalization regarding virtual objects. In spite of their crucial functional importance, little research has been conducted on classifying virtual objects so far. More work has been done on the technical side; for instance, an approach of including detailed solutions for all possible interactions with an object into its definition has been proposed [11]. Another later presented framework takes up on this idea and adds inter-object interaction definitions [10]. Currently – to the authors’ knowledge – at least the two MUVEs Second Life and OpenSim support defining avatar positions for interaction within an object definition, as well as inter-object communication. A first informal classification of virtual objects was proposed by Smith and Willans while investigating the requirements of virtual objects in relation to interaction needs: the authors state that the task requirements of the user define the behavioral requirements of any object. Consequently, they distinguish between background objects, which are not critical to the scenario, contextual objects, being part of the scenario but not in the focus, and task objects, which are central to the scenario and the actions of the user [21]. While this distinction may be useful for determining the level of importance of virtual objects, i.e. in requirements analysis phase, it does not distinguish objects based on their functional characteristics. Hence, we present a classification of virtual objects according to their activeness and their reaction to user actions: Static Objects have one single state of existence; they do not follow any type of behavior and do not particularly respond to any of a user’s actions. We distinguish among static objects that are in a fixed position, i.e. not movable and not to take away, and objects that are portable. These latter static objects can be visibly worn, carried or just repositioned, and thus have a distinct value for visual collaboration. Automated Objects either execute animations repeatedly or by being triggered. Alternatively they follow a behavior (ranging from simple behaving schemes such as

128


e.g. following an avatar, through highly complex autonomous, intelligent behaviors). We further separate the most rudimentary of all object behavior forms into an extra sub-category – the behavior of merely constantly updating its state or contents. Interactive Objects represent generally the notion of a tool or instrument; either they produce an output as a response to a given input, or they execute actions on direct user commands (like e.g. a remote control), or they act as vehicles, meaning that the user directly controls their movement (with or without the user’s avatar on it), using his primary navigation controls. The border between automated and interactive objects may seem fuzzy at first, but it is clearly delineated by the differentiation whether a user triggers an object to act deliberately or indirectly. Considering alternative classification properties, for example the distinction of whether virtual objects are fixed in their position or not, whether they can be moved or deformed, or follow physical laws, e.g. move in the wind, is in our belief of secondary importance – especially for the use cases we try to support with our contribution (professional collaboration tasks). 3.5 A Blueprint for Embodied Virtual Collaboration Figure 1 illustrates the framework for virtual collaboration based on the distinctions described in the previous sections. It is intended as a blueprint for virtual, embodied collaboration in virtual environments. As such, it can be used as a basis to develop or describe collaboration patterns in MUVE. Its three-tier architecture reflects the syntactic, semantic, and pragmatic levels of a collaboration medium, as discussed in 3.2. In the following, we explain the parts of the framework, in a top-down order. Context and Goal. The context describes the application domain of a collaboration pattern, while the goal defines more specifically what kind of activity a pattern aims to support. A first category comprises patterns that aim for collaborative work in the traditional sense, i.e. having main goals such as to share information or knowledge, collaboratively design or create a draft, a product, or a plan, assess or evaluate data or options, or make decisions etc. Since these goals do not necessarily have to be associated with work in the narrow sense of the word, we label the first context category Collaborate (for a definition of collaboration see [16]). The category Learn frames the domain of education. We assigned six goals to it, selected according to Bloom’s Taxonomy [3]. Bloom distinguishes different levels of learning goals starting with simple memorizing or recalling information, to the more difficult tasks of comprehending something, being able to apply it, analyze it, being able to synthesize it or even evaluate new knowledge regarding its limitations or risks. In the domain of Play we do not strive for mutually exclusive and collectively exhaustive categories and simply allude to such usual game oriented goals as feeling challenged by competition, distracting oneself (losing oneself in a game), or socializing with others in a playful manner. A collaboration pattern can also be aiming at several goals. Dramaturgy. The term dramaturgy in this context designates the way in which the infrastructure in virtual world is used to reach a specific collaboration goal or in other words support a group task. While the goals and contexts specify the why of a


129

Fig. 1. A Blueprint for Embodied Virtual Collaboration

collaboration pattern, and the infrastructure (below) the how, the dramaturgy consists of the necessary participants and their roles and relations (the ‘who’), their interaction spaces and repertoire (the ‘where’), as well as the timing and sequencing of their interactions (the ‘when’). The dramaturgy also specifies the actions (the ‘what’) taken by the participants and the social norms and rules they should follow within a given collaboration pattern. The dramaturgy defines in which ways the infrastructure of a virtual world can be used by the participants to achieve a common goal. Infrastructure. The final, most basic level of the blueprint contains the previously discussed elements Actions and Objects. As explained in previous subsections, we think it is useful (for the design of patterns) to distinguish among communicative, navigational, and object-related actions and among static, automated, and interactive virtual objects. We refined a definition of a collaboration pattern in subsection 3.1, as being a set of tools, techniques, behaviors, and activities for people who meet at a place to work on a common goal, together in a group. Using the wording of the framework, this

130


would translate to a set of objects, actions, rules, and steps for participants with roles who meet at a location to collaborate on a common goal in a given context. A specific collaboration pattern is then an instance of the framework and can be defined using the parameters positioned within the framework. There are two distinct ways in which the above blueprint can be used for practical and research purposes: It can be used in a top-down manner from goal to infrastructure in order to specify how a given goal can be achieved using an online 3D virtual environment. Alternatively, the blueprint can be used bottom-up in order to explore how the existing virtual world infrastructure can enable innovative dramaturgies that help achieve a certain collaboration (or learning) goal. In the next section, we are going to illustrate how the elements of the framework can help in the description of collaboration patterns. Some of these patterns have been developed using the framework in a top-down manner, while others were created from a bottom-up perspective.

4 Examples of Collaboration Patterns Based on the Blueprint The theory of patterns, originally developed for architecture [14], but in practice more commonly used in software development, can be applied to the domains of collaboration, as outlined above. The documentation of collaboration patterns, however, needs to be adapted to the context of virtual environments. For this purpose, we have presented a collaboration framework in section 3 which we will now use to present a series of online collaboration patterns. We have collected a number of virtual collaboration patterns and formalized them using the blueprint of section 3. The resulting patterns range from Virtual Team Meeting, Virtual Town Hall Q&A, Virtual Design Studio, Online Scavenger Hunt, Virtual Role Playing, Project Timeline Trail, Project Debriefing Path, Virtual Workplace, Virtual Knowledge Fair, to Spatial Group Configuration (for these and other patterns, see [17]). In figures 2 and 3, we provide four examples of collaboration patterns based on our framework. The first two patterns support teams in their collaboration, while the patterns documented in figure 3 can be used by larger groups. As the figures illustrate, a collaboration pattern (i.e. an instance of the framework) is comprised of one or several alternatively applicable contexts, several possible goals for the pattern, a full dramaturgy description, and avatar actions and virtual objects that are required. Hereby, actions and objects are ordered by relevance for the particular pattern (e.g. talk and chat can be useful for most patterns, although are not crucial in every case, thus not documented there). These four examples illustrate that the framework presented can be used to analyze or document the core requirements for online, virtual embodied collaboration in the form of patterns (although a complete pattern description should also contain pointers to related patterns). The framework cannot, however, predict the actual value delivered by such collaboration patterns. We will address this important issue in section 5.


Fig. 2. Two Collaboration Patterns for Virtual Teams in the Structure of the Blueprint

131

132


Fig. 3. Two Collaboration Patterns for Virtual Communities, in the same Structure


133

5 Future Research Needs and Initiatives Having established a systematic map of the elements required to devise and implement virtual, immersive and embodied collaboration patterns, the question nevertheless remains which of these patterns are the most effective ones in terms of their benefit in supporting collaboration tasks in groups (and what drawbacks or risks they may contain). To this end, we are currently devising experimental settings in order to compare virtual collaboration patterns with other collaboration settings. Our first experiment will take place in an especially prepared project setting implemented in an OpenSim environment. It will consist of a series of typical project management tasks, such as introducing project team members to each other, team building, conducting a stakeholder analysis, or agreeing on a joint timeline of project milestones. In a first set of experiments we will use students as participants, in a second round managers. In addition to observing and recording the behavior and measuring the performance of the participants, we will also administer ex-post surveys on the participants’ satisfaction with the task and communication support provided by the collaboration pattern and the virtual environment. This should give us additional insights into how the elements of a virtual collaboration pattern work together. While these experiments will yield relatively reliable data, they nevertheless lack the real-life context in which collaboration usually takes place. Consequently, a further area of research consists of participatory observation (or alternatively online ethnographies) in real-life collaboration settings that take place in virtual worlds. This will allow researchers to better assess the real advantages and disadvantages of this new form of working together. Additionally, in another related ongoing research project we are investigating communication and the use of tools in real-life design studios [4]. This work might give further insights on the infrastructural requirements (i.e. actions and objects, in our blueprint nomenclature) for patterns for collaborative design.

6 Conclusion In this contribution, we have developed and presented a systematic framework that organizes the necessary elements for the design and implementation of collaboration patterns in virtual worlds. This framework is based on three levels, namely the pragmatic or contextual level, including the goals of an online interaction, the semantic or dramaturgic level that defines how elements and actions are used (and interpreted) in time to achieve the collaboration goal, and the syntactic or infrastructure level consisting of the actual objects and online actions that are combined to implement a collaboration dramaturgy. We have presented two team-based virtual collaboration patterns, and two community-based collaboration patterns to illustrate the use of the framework. In terms of limitations and future research needs, we have pointed out that our framework does not provide indications as to the value added of collaboration patterns. This is thus an area of future concern that we will examine through the use of controlled online experiments and in-situ participatory observation within organizations.

134


References 1. Bainbridge, W.S.: The Scientific Research Potential of Virtual Worlds. Science 317(5837), 472–476 (2007) 2. Bell, M.: Toward a Definition of “Virtual Worlds”. Journal of Virtual Worlds Research 1(1) (2008), http://www.jvwresearch.org/v1n1_bell.html 3. Bloom, B.S.: Taxonomy of Educational Objectives: The Classification of Educational Goals. McKay, New York (1956) 4. Botturi, L., Rapanta, C., Schmeil, A.: Communication Patterns in Design. In: Proc. of Communicating (By) Design Conference, Brussels (in press) 5. Bowman, D.A., Kruijff, E., Poupyrev, I., LaViola Jr., J.J.: 3D User interfaces: Theory and Practice. Addison Wesley, New York (2005) 6. Carroll, J.M.: Introduction to the special issue on Scenario-Based Systems Development. Interacting with Computers 13(1), 41–42 (2000) 7. Dourish, P.: Where the Action Is: The Foundations of Embodied Interaction. MIT Press, Cambridge (2001) 8. Eco, U.: A Theory of Semiotics. Indiana University Press, Indiana (1978) 9. Gottesdiener, E.: Decide How to Decide: A Collaboration Pattern. Software Development Magazine 9(1) (2001) 10. Jorissen, P., Lamotte, W.: A Framework Supporting General Object Interactions for Dynamic Virtual Worlds. Smart Graphics, 154–158 (2004) 11. Kallmann, M., Thalmann, D.: Modeling Objects for Interaction Tasks. In: Proc. of Eurographics Workshop on Animation and Simulation, pp. 73–86 (1998) 12. Manninen, T.: Interaction in Networked Virtual Environments as Communicative Action Social Theory and Multi-player Games. In: Proceedings of CRIWG2000 Workshop, Madeira, Portugal. IEEE Computer Society Press, Los Alamitos (2000) 13. Norman, D.: The Design of Everyday Things. Basic Books, New York (1988) 14. Price, J.: Christopher Alexander’s Pattern Language. IEEE Transactions on Professional Communication 42(2), 117–122 (1999) 15. Rickel, J., Johnson, W.L.: Task-Oriented Collaboration with Embodied Agents in Virtual Worlds. In: Cassell, J., Sullivan, J., Prevost, S. (eds.) Embodied Conversational Agents. MIT Press, Boston (2000) 16. Roschelle, J., Teasley, S.: The construction of shared knowledge in collaborative problem solving. In: O’Malley, C. (ed.) Computer-supported collaborative learning, pp. 69–197. Springer, Berlin (1995) 17. Schmeil, A., Eppler, M.J.: Knowledge Sharing and Collaborative Learning in Second Life: A Classification of Virtual 3D Group Interaction Scripts. Journal of Universal Computer Science (in print) 18. Schmid, B.F., Lindemann, M.A.: Elements of a Reference Model for Electronic Markets. In: Proceedings of the 31st Annual Hawaii International Conference on Systems Science (HICSS), vol. 4, pp. 193–201 (1998) 19. Schubert, T.W., Friedmann, F., Regenbrecht, H.T.: Embodied presence in virtual environments. In: Paton, R., Neilson, I. (eds.) Visual Representations and Interpretations, pp. 268– 278. Springer, Heidelberg (1999) 20. Smith, S.P., Harrison, M.D.: Editorial: User centered design and implementation of virtual environments. International Journal of Human-Computer Studies 55(2), 109–114 (2001) 21. Smith, S.P., Willans, J.S.: Virtual object specification for usable virtual environments. In: Annual Conference of the Australian Computer-Human Interaction Special Interest Group, ACM OzCHI 2006 (2006)

Conceptual Design Scheme for Virtual Characters Gino Brunetti1 and Rocco Servidio2 1

INI-GraphicsNet Stiftung, Rundeturmstrasse 10, 64283 Darmstadt, Germany [email protected] 2 Linguistics Department, University of Calabria, P. Bucci Cube 17/B, 87036 Arcavacata di Rende, Cosenza, Italy [email protected]

Abstract. The aim of this paper is to describe some theoretical considerations about virtual character design. In recent years, many prototypes of cognitive and behavioral architectures have been developed to simulate human behavior in artificial agents. Analyzing recent studies, we assume that there exists a variety of computational models and methods in order to increase the cognitive abilities of the virtual characters. In our opinion, it is necessary to perform a synthesis of these approaches in order to improve the existing models and avoiding the application of new approaches. Considering these aspects, in this paper we describe a taxonomy that explores the principal cognitive and computational parameters involved in the design, development and evaluation of a virtual character. Keywords: Virtual characters, Emotions, Gestures, Artificial behavior, Cognitive Modelling.

1 Introduction It is well known that nonverbal communication like emotions, gestures and body movements play an essential role in human communication. Consequently, we have seen an increase in interest in the design and realization of software and hardware systems able to simulate human abilities, e.g. for human-machine interaction such as multimodal interaction, interactive models, virtual reality and 3D interaction [1]. The high rate of evolution of virtual characters applications implies that it is necessary to manage more efficiently the design and development of complex and dynamic behavior. Much research [2] has shown that virtual characters’ expressions of empathic emotions enhance users’ satisfaction, engagement, perception of the virtual agents, and performance in task achievement [3, 4, 5]. In order to increase reliability, recent studies have proposed a new class of interpolation algorithm for generating facial expressions to manage emotion intensity [6]. MPEG-4 is a standard for facial animation [7, 8, 9] which researchers use to specify both archetypal facial expressions and facial expressions of intermediate emotions [10]. Experiments were conducted to study individual differences in users’ perceptions of blended emotions from virtual characters expressions [11, 12]. Layered F. Lehmann-Grube and J. Sablatnig (Eds.): FaVE 2009, LNICST 33, pp. 135–150, 2010. © Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering 2010

136

G. Brunetti and R. Servidio

models were defined for relating facial expressions of emotions on the one hand, and, on the other, moods and personality traits, using three different timescales [13]. All of the studies show the existence of different computational models of emotions. Often, the research describes the design and implementation of a complete new prototype, mixing discussion of technical innovations with new application areas or approaches and interaction techniques. In spite of the fact that most of these studies are generic, there is no well-defined and commonly accepted approach regarding how the virtual characters architecture should be designed. Moreover, experimental results show that researchers have designed excellent models of virtual character behavior. Virtual characters are not designed just for movies and games. They can be used for a variety of purposes such as training, education, psychological therapy, etc. For example, eLearning is one application field for virtual characters. They are used to present educational material, answer users’ questions and give feedback about learning progression. In general, the virtual characters applications are a topic of interest for many researchers. It is now necessary to identify specific guidelines in order to develop virtual characters able to exhibit more complex behavior. In this paper, we propose a taxonomy in order to identify the principal cognitive functions involved in the design and evaluation of virtual characters. In many cases, these models are not the result of flawed research, but the necessary negotiation made in the exploration of new approaches that integrate different research areas. So, these approaches allow the implementation of a good system, but the evaluation process is different in comparison to another approach. We want to define the state of the art involved in the design and implementation of virtual characters. We propose an attempt at a taxonomy which describes the principal research for the modelling, realization and evaluation of virtual characters. The paper provides an account of the following problems: 1) virtual character properties; 2) psychological aspects that influence the perception of the virtual character’s actions; 3) definition of a virtual character criteria set in order to design successful virtual characters. The paper consists of seven sections. In the next section, we offer a description of the relation between emotions and virtual characters. The artificial emotions recognition process is discussed in section 3. Gestural behavior is examined in section 4. Section 5 describes the taxonomy for virtual character research. Finally, in section 6 we then offer some conclusions and seek to trace some future directions in virtual characters design.

2 Emotions and Virtual Characters “Emotion researchers define an emotion as a short-lived, biologically based partner of perception, experience, physiology, and communication that occurs in response to specific physical and social challenges and opportunities” [14]. The aim of this definition is to distinguish the emotions from other phenomena. In general, emotions are evoked by flexible interpretations of stimuli and have specific intentional objects while moods have less specific cause and remain for longer periods of time. For example, emotion traits, such as hostility and shyness, respond emotionally to broad classes of stimuli. Mood represents the overall view of an individual internal state. Whereas the emotions are associated with a specific expression or cause, moods are

Conceptual Design Scheme for Virtual Characters

137

not identifiable in terms of cause. The difference between emotion and mood is that, emotions regulate actions, while moods modulate the cognition. Emotion researchers agree on the adaptive functions of emotions, but they propose different explanations for this aspect. The ability to communicate emotions is essential for a natural interaction between human and virtual characters. If a virtual character does not possess emotional expressions, it could be interpreted as indifferent towards the human. Therefore, it is important that a virtual character show its emotional state in order to improve the interaction with the user or with other virtual agents. Specifically, the researcher proposes different approaches and methodologies to design artificial emotions. The aim of this research is to implement, in virtual characters, artificial emotions able to generate affective behavior improving autonomy, adaptation, and social interaction in the virtual environment. Based on the functional role of emotions, [15] specifies 12 potential roles for emotions in artificial systems. A survey of relevant virtual character behavior is showed by [16]. The creation of virtual characters is an interdisciplinary research field. The disciplines involved include design and implementation of cognitive architecture [17], modelling of a nonverbal communication system [18], expressiveness of the virtual character to improve visual realism and to solicit a realistic response [19], and finally to design user-friendly Graphical User Interfaces (GUI) [20]. Other aspects concern behavioral analysis [21] and the realization of the virtual scenario where the virtual characters are posted in [22]. Designing expressive virtual characters raises several research questions [23]. From a computer science point of view, the characters should be able to display facial expressions of complex emotions in real-time based on different user inputs, whilst, from a psychological point of view, designers of virtual characters need to know the cognitive processes regarding user perception and which are involved both in the facial recognition and in the movement expression. In recent years, several virtual character cognitive architectures have been proposed. The aim of these architectures is to reproduce realistic human abilities, with the purpose of going beyond the display of individual basic emotions models, defined for the facial display through so-called blends of emotion or nonarchetypal expressions [12, 24, 25]. However, applied models and methods, even if derived from an interdisciplinary approach, show some limits. Often the design and implementation of virtual characters is based on specific application requirements or developed as a test to verify a research hypothesis. Such approaches do not always reflect the goals of this research area in terms of qualities of the results to achieve. However, the realizations of virtual characters that show human abilities is a highly complex task. De facto, the research methods used are much discussed. The major difficulty in this research field is the fact that believability of the virtual characters is essential for an effective interaction. Believability is the ability of the agent to be viewed as a living, although fictional, character. These studies can be divided into two separate but interconnected approaches, which use empirical results to design virtual characters. The first approach creates virtual characters without an internal mental state. In this case, the emotions are the results of mathematical and geometrical models that manage the visual movement of the virtual characters. Research results of this approach are used to build character’s animations. The analyses are based on the recognition of emotion by subjects [26, 27]. The second approach designs and develops virtual characters to be included within immersive virtual

138

G. Brunetti and R. Servidio

environments. The primary goal of this approach is to improve interaction and communication between agents and users. In this case, the computational model of the virtual characters includes a mental state of the personality in order to obtain a more realistic behavior. The purpose of these studies is to measure the communication between subject and virtual characters, such as interaction and collaboration. Compared to the first approach, which simulates all basic emotions, the latter simulates few emotional expressions, but the virtual characters are provided with body movements in order to increase the complexity of the realized actions.

3 Modelling Artificial Emotions and Its Recognition Research results from several fields such as cognitive psychology, social psychology, biomechanics studies on the movements and neuroscience, allow us to define a criteria set framework for the design of virtual characters able to realize realistic behavior [21, 22]. Empirical evidence shows that behavioural expressivity is connected to nonverbal communication, which is generally taken to be indicative of the true psychological state of a virtual character especially when the cues are negative [23, 27]. In the communication process, a smile or another face movement can have different meanings. The reason for implementing artificial emotions in virtual characters is twofold: one is to generate realistic virtual characters e.g. to support Human-Computer Interaction (HCI) applications, the other is to investigate their recognition processes. For example, research results show that human subjects are able to recognize the artificial emotions realized using the Facial Action Coding System (FACS) developed by [24], which measure facial expressions by Action Units (AU). Each AU allows us to measure how few changes of the face involve more facial muscles. [24] have calculated 44 AUs that realize facial expression changes and 14 AUs that describe grossly the changes in the direction of the look and in the orientation of the head. When AUs occur in combination, they may be additive, in which the combination does not change the appearance of the constituent AU, or non-additive, in which the appearance of the constituents does change. Ekman has observed more than 7,000 combinations from which he derived specific combinations of FACS Action Units representing prototypic expressions of emotion like joy, sadness, anger, disgust, fear, and surprise. Currently, FACS is recognized as a reference system enabling the codification all kinds of facials expressions. Inspired by FACS, the MPEG-4 standard is particularly important for facial animation. The Facial Definition Parameters set (FDPs) and the Facial Animation Parameter set (FAP) were designed to allow the definition of facial shape and texture, as well as the animation of faces reproducing expressions, emotions and speech pronunciation. FDPs are used to customize a given face model for a particular face. The FDP set contains a 3D mesh (with texture coordinates if texture is used), 3D feature points, and optional texture and other characteristics such as hair, glasses, age and gender. The FAPs, on the other hand, are based on the study of minimal facial actions and are closely related to muscle actions [28]. They represent a complete set of basic facial actions, such as squeeze or raise eyebrows, open or close eyelids, and therefore allow the representation of most natural facial expressions. All FAPs involving

Conceptual Design Scheme for Virtual Characters

139

translational movement are expressed in terms of the Facial Animation Parameter Units (FAPU). FAPUs aim at allowing interpretation of FAPs on any facial model in a consistent way, producing reasonable results in terms of expression and speech pronunciation. “For example, the MPEG-4-based facial animation engine for animating 3D facial models works in real time and is capable of displaying a variety of facial expressions, including speech pronunciation with the help of 66 low-level Facial Animation Parameters” [28, p. 91]. By contrast, the development of automated systems able to comprehend human emotions is more complicated. The reasons are manifold, and some of these can be summarised as follows: 1. The capabilities for modelling characters are limited. Experimental results show the difficulties in modelling the psychological state of a virtual character and to map it to the expression of the corresponding emotion. Other results indicate that subjects perceive characters purely on the basis of their visual appearance or enhanced capabilities. 2. Body expression and emotion perception have a high cognitive value. Face and body both contribute in conveying the emotional state of the individual. In our natural environment, face and body are part of an integrated whole. This correlation is problematic during the modelling of virtual character behavior. Experimental results suggest that if the parametric model of a body posture is not associated with the emotion expression, participants are not able to interpret the behavior of virtual characters. 3. Integration of facial expression and emotional body language is not present or very poor. Electrophysiological correlates indicate that this integration of affective information already takes place at the very earliest stage of face processing. Recognition of the emotion conveyed by the face is systematically influenced by the emotion expressed by the body. When observers have to make judgments about a facial expression, their perception is biased toward the emotional expression conveyed by the body. [29] have shown that “our behavioral and electrophysiological results suggest that when observers view a face in a natural body context, a rapid (

Facets of Virtual Environments: First International Conference, FaVE 2009, Berlin, Germany, July 27-29, 2009, Revised Selected Papers

Facets of Virtual Environments: First International Conference, FaVE 2009, Berlin, Germany, July 27-29, 2009, Revised Selected Papers (Lecture Notes of ... and Telecommunications Engineering)

Arts and Technology: First International Conference, ArtsIT 2009, Yi-Lan, Taiwan, September 24-25, 2009 - Revised Selected Papers

Mobile Networks and Management: First International Conference, MONAMI 2009, Athens, Greece, October 13-14, 2009. Revised Selected Papers

Business Process Management Workshops: BPM 2009 International Workshops, Ulm, Germany, September 7, 2009, Revised Papers

Computational Intelligence: Revised and Selected Papers of the International Joint Conference IJCCI 2009 held in Funchal-Madeira, Portugal, October 2009

Electronic Healthcare: Second International ICST Conference, eHealth 2009, Istanbul, Turkey, September 23-15 [i.e. 25], 2009 : Revised Selected Papers

Information Security and Cryptology: 5th International Conference, Inscrypt 2009, Beijing, China, December 12-15, 2009. Revised Selected Papers

Networks for Grid Applications: Third International ICST Conference, GridNets 2009, Athens, Greece, September 8-9, 2009, Revised Selected Papers

Future Internet - FIS 2009: Second Future Internet Symposium, FIS 2009, Berlin, Germany, September 1-3, 2009, Revised Selected Papers (Lecture Notes ... Networks and Telecommunications)

Proceedings of International Science Education Conference 2009

Professional Photographer July 2009

Chess Life - July 2009

Architectural Record - July 2009

Digital Business: First International ICST Conference, DigiBiz 2009, London, UK, June 17-19, 2009, Revised Selected Papers (Lecture Notes of the Institute ... and Telecommunications Engineering)

Digital Business: First International ICST Conference, DigiBiz 2009, London, UK, June 17-19, 2009, Revised Selected Papers (Lecture Notes of the Institute ... and Telecommunications Engineering)

Embodied Artificial Intelligence: International Seminar, Dagstuhl Castle, Germany, July 7-11, 2003, Revised Selected Papers

Active Networks: First International Working Conference, IWAN'99, Berlin, Germany, June 30 - July 2, 1999, Proceedings

Formal Methods for Industrial Critical Systems: 12th International Workshop, FMICS 2007, Berlin, Germany, July 1-2, 2007, Revised Selected Papers

Embodied artificial intelligence: international seminar, Dagstuhl Castle, Germany, July 7-11, 2003; revised selected papers

Sensor Systems and Software: First International ICST Conference, S-CUBE 2009, Pisa, Italy, September 7-9, 2009, Revised Selected Papers (Lecture Notes ... and Telecommunications Engineering)

Information Security and Digital Forensics: First International Conference, ISDF 2009, London, United Kingdom, September 7-9, 2009, Revised Selected Papers ... and Telecommunications Engineering)

Mobile Lightweight Wireless Systems: First International ICST Conference, MOBILIGHT 2009, Athens, Greece, May 18-20, 2009, Revised Selected Papers (Lecture ... and Telecommunications Engineering)

Mobile Networks and Management: First International Conference, MONAMI 2009, Athens, Greece, October 13-14, 2009. Revised Selected Papers (Lecture Notes ... and Telecommunications Engineering)

Functional and Constraint Logic Programming, 18th International Workshop, WFLP 2009, Brasilia, Brazil, June 28, 2009, Revised Selected Papers

Autonomic Communication: First International IFIP Workshop, WAC 2004, Berlin, Germany, October 18-19, 2004, Revised Selected Papers

Environments for Multi-Agent Systems: First International Workshop, E4MAS, 2004, New York, NY, July 19, 2004, Revised Selected Papers

Security in Pervasive Computing: First International Conference, Boppard, Germany, March 12-14, 2003, Revised Papers

E-Voting and Identity: First International Conference, VOTE-ID 2007, Bochum, Germany, October 4-5, 2007, Revised Selected Papers

Frommer's Germany 2009 (Frommer's Complete)

Software language engineering first international conference; revised selected papers SLE <1. 2008. Toulouse>

Facets of Virtual Environments: First International Conference, FaVE 2009, Berlin, Germany, July 27-29, 2009, Revised Selected Papers