Clemens Holzmann Spatial Awareness of Autonomous Embedded Systems
VIEWEG+TEUBNER RESEARCH
Clemens Holzmann
Spatial Awareness of Autonomous Embedded Systems
VIEWEG+TEUBNER RESEARCH
Bibliographic information published by the Deutsche Nationalbibliothek The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie; detailed bibliographic data are available in the Internet at http://dnb.d-nb.de.
Dissertation Universität Linz, 2008
1st Edition 2009 All rights reserved © Vieweg+Teubner | GWV Fachverlage GmbH, Wiesbaden 2009 Editorial Office: Christel A. Roß | Anita Wilke Vieweg+Teubner is part of the specialist publishing group Springer Science+Business Media. www.viewegteubner.de No part of this publication may be reproduced, stored in a retrieval system or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the copyright holder. Registered and/or industrial names, trade names, trade descriptions etc. cited in this publication are part of the law for trade-mark protection and may not be used free in any form or by any means even if this is not specifically marked. Cover design: KünkelLopka Medienentwicklung, Heidelberg Printing company: STRAUSS GMBH, Mörlenbach Printed on acid-free paper Printed in Germany ISBN 978-3-8348-0798-4
To my mother and father, who have always believed in me and supported me all the way. Thank you.
Acknowledgements I would like to especially thank the following people who contributed to this thesis in various ways and without whose support it would not have been possible to finish it in such a short time and with the present extent and quality: • My supervisor Univ.-Prof. Mag. Dr. Alois Ferscha, for giving me the opportunity to work at his department and supporting me in every way with this dissertation. He offered me an interesting topic to work on, provided generous financial support with regard to the professional equipment I could use in my research as well as for being able to publish and present results at international conferences, he was always available for discussions, helped with useful suggestions to keep the thesis on track and in time, and allowed me to realize and develop my ideas in an industrial research project with Siemens AG Germany. Last but not least, he gave me all the time I needed when actually writing the thesis. • Dipl.-Ing. Manfred Hechinger, who implemented major parts of the software framework presented in Chapter 6 as well as the vibro-tactile space awareness application presented in Section 7.2. Also the framework architecture and many of the related concepts were developed in close cooperation with him. • Univ.-Prof. Mag. Dr. Gabriele Kotsis, for reading this thesis as the second supervisor and being a great help particularly in its initial stage by providing her expertise with regard to technical issues and writing a dissertation in general. • The project partners of Siemens AG Germany, and in particular Dipl.-Inform. Dr. Andreas Zeidler, for their critical and useful feedback on parts of my work as well as for sharing their knowledge on real-world requirements. • The co-authors and reviewers of my publications, who provided helpful suggestions for their improvement and thus also contributed to a higher quality of the respective sections in this thesis. I would like to especially thank the reviewers of my submission to the Ubicomp 2005 doctoral colloquium,
VIII
Acknowledgements
who suggested to extend my sole focus on the spatial direction of artifacts at that time to different types of spatial relationships and anchor the work in real-world application domains. • Dominik Hochreiter and Dipl.-Ing. (FH) Bernadette Emsenhuber for their work on realizing the hardware of the vibro-tactile belt (cf. Section 7.2.1), and in this regard Dipl.-Ing. Andreas Riener for bringing in his expertise on haptic perception. • My partner Mag. Anja Razenböck, for supporting me emotionally and showing understanding for the many evenings and weekends I was working on this thesis. • All the colleagues I have come to know since my employment at the Johannes Kepler University Linz, who contributed to an inspirational and pleasant working environment and made the time at the university enjoyable and fruitful.
Dipl.-Ing. Mag. Clemens Holzmann
Abstract The invisible integration of technology into everyday objects like home appliances, cars and mobile phones, which is the declared vision and fundament of research in the field of pervasive computing, leads to huge quantities of smart objects which are situated in the surrounding physical space. Equipped with embedded systems technology, they become increasingly heterogeneous and interconnected, raising the challenge of a semantically meaningful interplay with each other. In this thesis, such physical objects with embedded computing and communication technology are referred to as digital artifacts. One approach to meet this challenge is to design and implement systems that are able to operate autonomously in the background, namely with as little human intervention as possible, and interact with humans in a more unobtrusive way. In order to achieve autonomy, two aspects are particularly important: (i) context-awareness, which refers to the ability of a digital artifact to acquire environmental information in order to become aware of its situation and adapt to changing situations at runtime, and (ii) context sharing, which relates to an artifact’s ability of exchanging context information with other artifacts in communication range. An essential part of the context of spatially distributed objects is their position, direction and spatial extension with respect to an external reference system or with respect to other objects. The focus of this thesis is to make digital artifacts aware of such context information in order to enable their autonomous adaptation to spatial changes in the environment, whereas our main interest is on qualitatively abstracted spatial relations and their use at the application level. Concepts for recognizing, representing and reasoning about qualitative spatial relations among autonomous artifacts and their changes over time are presented, as well as an according architecture which has been implemented in a flexible software framework that builds upon qualitative relationship abstractions and the rule-based inference of conclusions from them. Special attention is paid to the adaptivity to different application domains, which influence the semantics of spatial abstractions and thus the behavior of digital artifacts. Evaluation results show the feasibility of the proposed concepts for developing spatially aware applications which involve one or more spontaneously interacting digital artifacts, and in particular that qualitatively abstracted relations can constitute an adequate basis for it.
Contents 1
Introduction 1.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2
Spatial Awareness 2.1 Context and Awareness . . . . . . . . 2.2 Autonomous Embedded Systems . . . 2.2.1 Exchange of Self-Descriptions 2.3 Mechanisms of Self-Organization . . 2.3.1 Self-Organization in Space . . 2.4 Spatial Context in Time . . . . . . . . 2.5 An Architecture for Spatial Awareness 2.6 Related Work . . . . . . . . . . . . . 2.6.1 Projects . . . . . . . . . . . . 2.6.2 Comparison . . . . . . . . . .
3
1 4 5 7
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
9 9 12 15 16 19 20 23 26 27 41
Representation of Space 3.1 Quantitative Representation . . . . . . . . . . . . . . . . . . 3.1.1 Spatial Abstraction with Zones-of-Influence . . . . . 3.1.2 Reference Systems for Position and Direction . . . . 3.2 Qualitative Representation . . . . . . . . . . . . . . . . . . 3.2.1 Qualitative Abstractions of Space . . . . . . . . . . 3.2.2 Recognition and Representation of Spatial Relations 3.2.3 Static and Dynamic Spatial Relations . . . . . . . . 3.2.4 Frames of Reference . . . . . . . . . . . . . . . . . 3.2.5 Spatiotemporal Relations . . . . . . . . . . . . . . . 3.3 Structure of Self-Descriptions . . . . . . . . . . . . . . . . 3.3.1 Quantitative Spatial Properties . . . . . . . . . . . . 3.3.2 Qualitative Spatial Relations . . . . . . . . . . . . . 3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
45 46 46 49 51 51 53 56 60 61 66 67 70 72
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
XII
4
5
6
7
Contents
Distributed Spatial Reasoning 4.1 Overview of Qualitative Approaches . . . . . . . . . . . . . . 4.1.1 Properties and Closures of Binary Relations . . . . . . 4.1.2 Compositional Reasoning . . . . . . . . . . . . . . . 4.2 Reasoning about Positional and Directional Relations . . . . . 4.2.1 Related Approaches . . . . . . . . . . . . . . . . . . 4.2.2 Composition of Positional Relations . . . . . . . . . . 4.3 Spatial Relationship Inference and Distribution . . . . . . . . 4.3.1 General Concept by Exploiting Relation Properties . . 4.3.2 Distribution Algorithm Using Compositional Inference 4.4 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . 4.5 Findings and Discussion . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . .
75 . 75 . 77 . 79 . 80 . 81 . 82 . 87 . 87 . 92 . 96 . 103
Rule-Based Spatial Awareness 5.1 Achieving Spatially Aware Behavior . . . . . . . . . . . . 5.1.1 Reasoning with Rules . . . . . . . . . . . . . . . 5.1.2 Using a Rule Engine . . . . . . . . . . . . . . . . 5.2 Rule-Based Qualitative Spatial Reasoning . . . . . . . . . 5.2.1 Inferring Relations and Application-Level Actions 5.3 Proof of Concept . . . . . . . . . . . . . . . . . . . . . . 5.4 Summary and Open Issues . . . . . . . . . . . . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
105 106 106 108 109 112 115 119
Zones-of-Influence Framework 6.1 Architecture . . . . . . . . . . . . 6.1.1 Design Considerations . . 6.1.2 Architecture Overview . . 6.1.3 Runtime Platform . . . . . 6.2 Components . . . . . . . . . . . . 6.2.1 Digital Artifact Service . . 6.2.2 Zones-of-Influence Service 6.2.3 Relations Service . . . . . 6.3 Runtime Behavior . . . . . . . . . 6.4 Discussion . . . . . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
121 122 122 123 126 127 127 130 135 140 143
Framework Evaluation 7.1 Development of Spatially Aware Applications 7.2 Application Scenarios . . . . . . . . . . . . . 7.2.1 Vibro-Tactile Space Awareness . . . . 7.2.2 Focus/Nimbus Awareness . . . . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
147 147 149 151 158
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
Contents
7.3 8
XIII
7.2.3 Spatiotemporal Awareness . . . . . . . . . . . . . . . . . 163 Comparison and Discussion . . . . . . . . . . . . . . . . . . . . 169
Conclusion 177 8.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 8.2 Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
Bibliography
191
List of Figures 2.1 2.2 2.3 2.4 2.5
Components of a digital artifact. . . . . . . . . . Exchange and comparison of self-descriptions. . Emerging “glider” pattern in “Game of Life”. . . Static spatial properties and relations of artifacts. General architecture for spatial awareness. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
Zone-of-Influence for a car’s braking distance [FHR+ 08]. . . . . . Types of Zones-of-Influence in 2-D. . . . . . . . . . . . . . . . . WGS84 spherical (ϕ,λ ,h) and Cartesian (x,y,z) coordinates. . . . . Recognition of spatial relations by comparing self-descriptions. . . Orientation relations of nearby extended objects, after [Her94]. . . Definition of static qualitative spatial relations in 2-D. . . . . . . . Cone-/projection-based orientation and qualitative distance [CFH97, RM04]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Definition of dynamic qualitative spatial relations in 2-D. . . . . . 3.9 Epsilon-neighborhood of a static qualitative orientation relation. . 3.10 Intrinsic and extrinsic orientation relations. . . . . . . . . . . . . 3.11 Qualitative temporal interval relations, cf. [All83]. . . . . . . . . . 3.12 Qualitative spatiotemporal relations. . . . . . . . . . . . . . . . . 3.1 3.2 3.3 3.4 3.5 3.6 3.7
4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9
14 16 18 21 24 47 48 50 54 55 56 57 58 59 60 63 65
An example for compositional reasoning. . . . . . . . . . . . . . 80 Composition of positional relations with same (top) and similar directions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 Distribution of the transitive relation left. . . . . . . . . . . . . . . 90 Distribution of the symmetric and reflexive relation near. . . . . . 91 Recognition, composition and intersection of qualitative positional relations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Compositional inference and distribution started at artifact b. . . . 98 Simulated topologies full binary tree, line, mesh and ring. . . . . . 101 Traffic induced by different algorithms depending on the topology. 102 Relative accuracy of different algorithms compared with flooding. 103
XVI
List of Figures
5.1 5.2 5.3 5.4
Rule-based reasoning [PNF+ 08]. . . . . . . . . . . . . . . . . Acquisition of position and direction from two IS-900 trackers. Trajectories of quantitative spatial relations. . . . . . . . . . . Recognized and inferred qualitative relations. . . . . . . . . .
. . . .
108 116 117 119
6.1 6.2 6.3 6.4 6.5 6.6 6.7 6.8
Overview of the Zones-of-Influence Framework architecture. . . . Layered architecture of the OSGi framework, after [All05, All07]. Interfaces of the Digital Artifact Service. . . . . . . . . . . . . . . Interfaces of the Zones-of-Influence Service. . . . . . . . . . . . . Interfaces and classes for representing Zones-of-Influence. . . . . Interfaces and classes of the Relations Service. . . . . . . . . . . Time for relationship recognition (top) and repository update. . . . Repository update time (top) and involved number of relations. . .
124 127 129 131 133 137 142 143
7.1 7.2 7.3 7.4 7.5 7.6 7.7 7.8 7.9 7.10
Vibro-tactile belt with eight vibrator elements. . . . . . . . . . Controller of the belt. . . . . . . . . . . . . . . . . . . . . . . Person wearing the belt. . . . . . . . . . . . . . . . . . . . . . Visualization of the 2-D scene and the belt’s vibrator elements. Exemplary application scenario for focus/nimbus awareness. . Visualization and relations of the focus/nimbus application. . . Qualitative spatial relations at a point in time. . . . . . . . . . Time series of qualitative spatial relations. . . . . . . . . . . . Visualization and relations of the scenario shown in Figure 7.7. Visualization and relations of the scenario shown in Figure 7.8.
157 157 157 158 160 162 164 165 168 170
. . . .
. . . . . . . . . .
. . . . . . . . . .
List of Tables 2.1 2.2
Static characteristics of an artifact’s spatial situation. . . . . . . . Comparison of related work. . . . . . . . . . . . . . . . . . . . .
21 43
4.1 4.2
78
4.3 4.4 4.5 4.6
Extrinsic orientation and distance relations with their properties. . Comparison of approaches for reasoning about static spatial relations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Composition table for static orientation relations. . . . . . . . . . Composition table for static distance relations. . . . . . . . . . . . Composition table for combined orientation and distance. . . . . . Simulation results for the topology of Figure 4.6. . . . . . . . . .
5.1
Combinations of qualitative spatial and temporal relations. . . . . 113
6.1
Time in [s] for adding new artifacts/updating their self-descriptions. 141
7.1 7.2
Relevancy of spatial abstractions for application scenarios. . . . . 150 Framework aspects covered by implemented spatially aware applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
81 85 85 86 99
Listings 3.1 3.2 3.3 3.4
Structure of an XML-based self-description. . . . . . . . . . Representation of a static and fragmented Zone-of-Influence. Representation of a dynamic Zone-of-Influence. . . . . . . . Representation of a qualitative relation. . . . . . . . . . . .
. . . .
. . . .
5.1 5.2 5.3
Maintenance rule for merging equal relations. . . . . . . . . . . . 111 Inference rule for the relation passing-by-right. . . . . . . . . . . 114 Inference rule for the relation passing-by-right-in-vicinity. . . . . 115
6.1
Specification of sensor data within a self-description. . . . . . . . 134
7.1 7.2 7.3 7.4 7.5
Zone-of-Influence for the vibro-tactile belt’s awareness shape. Rule for the vibro-tactile space awareness application. . . . . . Rule for the focus/nimbus awareness application. . . . . . . . Rules for the scenario shown in Figure 7.7. . . . . . . . . . . Rules for the scenario shown in Figure 7.8. . . . . . . . . . .
. . . . .
. . . .
. . . . .
67 69 70 72
153 155 161 167 168
List of Abbreviations API CSCW CSV DOF ECA ECEF EPC FoV GIS GML GPS GSM GUI HTTP IEEE IDE IR JEPD KML MEMS NED OSI OSGi PDA PML P2P RCC SQL TOTA URL Wi-Fi
Application Programming Interface Computer Supported Cooperative Work Comma Separated Value Degrees of Freedom Event-Condition-Action Earth-Centered Earth-Fixed Electronic Product Code Field of View Geographic Information Systems Geography Markup Language Global Positioning System Global System for Mobile Communications Graphical User Interface Hypertext Transfer Protocol Institute of Electrical and Electronics Engineers Integrated Development Environment Infrared Jointly Exhaustive and Pairwise Disjoint Keyhole Markup Language Micro-Electro-Mechanical Systems North-East-Down Open Systems Interconnection Open Services Gateway initiative Personal Digital Assistant Physical Markup Language Peer-to-Peer Region Connection Calculus Structured Query Language Tuples on the Air Uniform Resource Locator Wireless Fidelity
XXII
WGS84 XML ZoI
List of Abbreviations
World Geodetic System 1984 Extensible Markup Language Zone-of-Influence
1 Introduction Time and space are modes by which we think and not conditions in which we live. Albert Einstein, 1879-1955
At the time of this writing, we are facing a world with a huge, and ever increasing number of real-world objects with embedded computing and communication capabilities. This development is mainly driven by technological advances within the past few decades, which have made it possible to shrink sensors and actuators as well as processing and wireless communication technologies to a size that enables their integration into virtually everything. Things of everyday use in industrial settings, in manufacturing, in offices or in homes, like tools, appliances, machinery, furniture or even clothing have become the constituent “devices”, through which applications interact with the user, and there is strong evidence that this trend will continue. It can be observed that the “computer” is no longer a single desktop device, but is rather associated with services originating in the “digital world” and perceived through the “physical world”. The computer is more and more hidden in the fabric of everyday life, invisibly networked, and accessible everywhere; thus, we could say that computers become invisible, while their interfaces become omnipresent. A field of research tackling related challenges is ubiquitous computing, which was coined by Mark Weiser in a Scientific American article in 1991 [Wei91]. The vision of ubiquitous computing is the fusion of computing and communication technologies with the environment in such a way that the technology disappears. There are several terms such as calm computing, invisible computing, hidden computing, ambient intelligence, autonomous computing or pervasive computing, which all refer to the same vision but with a different focus at a time; in the context of this thesis we use the term pervasive computing to stress the pervasion of real-world objects with computing technology. Two key objectives of ubiquitous computing are identified in [Wei91]: ubiquity, namely the availability of computation and communication anytime and everywhere, and invisibility, which refers to the disappearance of computing technology. Invisibility can be achieved by embedding computational elements into real-
2
Introduction
world objects, but also by their autonomous operation [ECPS02] which makes them – due to the reduced human involvement – more or less disappear from the users’ consciousness. Pervasive computing is an inherently multidisciplinary topic, as many research fields including embedded systems, distributed systems, mobile computing, human/computer interaction and artificial intelligence [Sat01] address specific aspects for making the vision of “computing beyond the desktop” become reality. A typical service may consist of a large number of sub-services that themselves can be deployed on various devices at different – possibly changing – physical locations; the range of devices contributing to the service and their available interaction modalities may differ widely, as well. Obviously, the configuration and management of such dynamic “service landscapes” should be automated to a huge extent and not be of concern for the human user. This calls for some means of self-management and -organization, as their number, heterogeneity and complexity will sooner or later exceed the limits of human capability [HG03]. Moreover, interaction within such service environments will have to be more implicit (i.e. at the periphery of human attention) rather than explicit (i.e. at the focus of attention), so that it becomes very important that the individual devices manage themselves autonomously, with as little human intervention as possible. Clearly, the traditional approach of instructive systems [Wan04] with their deterministic and context-free nature appears less appropriate as an architecture for providing services by the interaction of embedded and networked devices; instead, a more autonomous system architecture [Hor01, KC03] is required. According to [FHdSR+ 07], two aspects of system properties contributing to their ability to operate autonomously in the background can be identified: self-management and self-organization. First, self-management relates to the individual devices. It stands for the ability of a single device to acquire information that can help to understand its situation or context, and to automatically adapt to changing contexts at runtime in a semantically meaningful way. There has been much research on this topic [CK00], which was mainly driven by technological advances in miniaturization of sensors and actuators for acquiring information about an environment and influencing it. One of the first definitions of context with regard to pervasive computing systems is given in [SAW94], where context is considered as the location of use, the collection of nearby people, hosts, and accessible devices, as well as the changes of such things over time. A more formal definition is given in [Dey00], defining context as any information than can be used to characterize the situation of an entity and denoting a system context-aware if it uses context to provide relevant information and/or services to the user. The second system property is self-organization, which re-
Introduction
3
lates to spontaneous configurations of such devices, and it stands for their ability to spontaneously (e.g. upon service requests or detection of certain spatial situations) join into ad-hoc “service ensembles” to e.g. negotiate and achieve ensemble goals – as for example the provision of a certain service to the user – through coordinated actions. Both self-management and self-organization have received much research attention in computer science over the past years [MMTZ06, SFH+ 03]. In particular, self-organization principles as inspired by nature attracted the attention of computer scientists [KE01, ZGMT04]. In the respective literature, self-organization is defined as a process in which pattern at the global level of a system emerge solely from numerous interactions among lower-level components of the system [CFS+ 01], where “pattern” refers to structure and organization in both space and time. Self-organization hence is way beyond centralized coordination, and complex collective behavior results from contextual local interactions between components [SFH+ 03]. Local interactions in turn are based on individual goals and the perception of the respective environment. The essence of self-organization is that system structure – and thus collective behavior – often appears without explicit trigger or pressure from outside the system, but is immanent to the system and results from interactions within it. System structure can evolve in time and space, may maintain a stable form or exhibit transient phenomena, or may grow or shrink in size, number or feature. An example for that is John Conway’s cellular automaton called “Game of Life”; it consists of a two-dimensional collection of cells which die, survive or become to live by simple local rules, whereas complex patterns emerge depending on an initial configuration of the automaton [Gar70]. The focus in this work is on autonomously operating real-world objects which are equipped with sensors, actuators, as well as with computing and wireless communication technology to support ad-hoc networking. These devices can have various different kinds of appearance (like shape, size, mobility, etc.) and embedded digital technology (e.g. mobile phones, smart appliances, smart rooms, etc.), and we refer to them as digital artifacts. A digital artifact has to have the ability to sense its context, and it must possess reasonable means to process and reason about the perceived context as well as the possibility to share its perceptions with others in range. Obviously, this is a fundamental requirement for such autonomous embedded systems, as sharing information about their context is a key property for collaboration and autonomous adaptation in order to reach ensemble goals based on local information gathering. As digital artifacts are by nature situated in physical space, their spatial properties (e.g. position and physical shape) – and in particular spatial relationships between them (e.g. distance and orientation) – are valuable context information
4
Introduction
and constitute a distinguishing feature of our work. Actually, most of the known phenomena of self-organization and -adaptation in nature are phenomena of selforganization in space [MZ05], and [ZM04] identifies the concept of space and the awareness of distributed components of their surrounding to play an important role for mechanisms of self-organization, as for example the coordination of activities by exploiting their spatial structure. In this regard, spatial abstractions are considered to be important means for implementing services which are distributed in physical space [Leo98, ZM04]. Self-organization is based on (direct or indirect) contextual local interactions between the components of a system; for this reason, both the inference of highlevel contextual information from spatial relationships, as well as standardized means for exchanging spatial information between the components are issues of research [HE03, MZ03]. This thesis is primarily concerned with techniques for making digital artifacts spatially aware (i.e. aware of spatial context information), which facilitates the semantically meaningful interaction among them.
1.1 Problem Statement The focus of this thesis is on mechanisms to provide digital artifacts, which are spatially distributed real-world objects with embedded digital technology, with an awareness about their spatial contexts. In this regard, an essential part of the information describing the spatial context of an artifact is its position, direction and spatial extension. In order to achieve such spatial awareness, it must have the ability to acquire its spatial context with sensors, possess means to reason about the perceived context and share this perception with others in communication range. Sharing spatial context information is a key property of autonomous artifacts, which allows them to adapt to changing spatial situations at runtime and thus contributes to their semantically meaningful, contextual interaction in space. As artifacts may have to coordinate their actions for providing certain services, we consider especially spatial relations among them as well as relationship changes to be of particular relevance. When developing context-aware applications, a tight coupling between the application and sensors is problematic due to the fact that it forces the programmer to deal with sensor details. Hence, low-level context information provided by sensors must be abstracted to be used by context-aware applications [Dey00]; for example, an application using location information may only be interested in highlevel information like rooms and buildings instead of geographical coordinates [FHO04a]. Generally speaking, abstracting context information is about separat-
1.2 Contribution
5
ing details which are not relevant for a certain application. Such symbolic abstractions of locations have been addressed by several researchers [Leo98, Sch95], with the aim to provide location information in a sensor-independent and more natural way that is closer to human concepts of space. Abstracted relations between the locations of devices are used in [HKG+ 05, KKG05], which are represented with meaningful names such as left or near. In this work, we go beyond utilizing just location information; in addition, the direction and spatial extension of mobile artifacts with respect to a global reference system and with respect to each other are taken into account. Our primary focus is on the investigation and development of concepts for recognizing and representing spatial relations between autonomous digital artifacts by using qualitative abstractions, as well as the inference of high-level context information out of it. The representation of and reasoning about qualitative spatial relations, which are concerned with the abstraction of continuous properties of the physical world and inferring knowledge from the respective qualitative representations, has already attracted much interest by researchers [CH01]. Compared with quantitative approaches, qualitative ones have clear advantages whenever the spatial cognition of humans is involved [Mus00] or systems with limited computing resources are concerned [Fre92b], among others. Summing up, the focus of this thesis can be stated as follows: What are concepts and architectures for making autonomous embedded systems aware of qualitatively abstracted spatial relations over time and using them in spatially aware applications? The thesis, however, does not address spatial representation and reasoning in general, but rather concentrates on methods for providing qualitative spatial relationship information to autonomous digital artifacts at application level. Central is the development of an architecture for enabling spatial awareness of autonomous artifacts without any kind of centralized instance, as well as its application to real-world scenarios. This includes the issues of recognizing, representing and reasoning about qualitative spatial relations both at certain points in time and by considering their changes over time.
1.2 Contribution Within the scope of this thesis, application-independent concepts that allow autonomously operating digital artifacts to become aware of and use spatial context
6
Introduction
information have been investigated and developed. In order to implement spatial awareness of artifacts among each other, quantitative and qualitative representations of spatial aspects are used. For the former, a concept referred to as Zones-of-Influence has been developed, which builds upon initial work published in [FHR+ 08]. Zones-of-Influence encode spatial properties of artifacts – in particular their absolute and relative position, direction and spatial extension – with numerical values that may be provided by sensors. They are explicitly defined twoor three-dimensional shapes of a certain size which are positioned and directed in physical space, thus representing geographic regions that are of relevance for the artifacts and their users or applications. Each digital artifact is associated with one or more of such zones at a time, together representing the relevant spatial knowledge which is distributed across them in physical space. In order to enable artifacts to autonomously recognize spatial relations to others around, they share their Zones-of-Influence through an exchange of generic, structured self-descriptions. This allows for determining spatial relations between their Zones-of-Influence, including distance, orientation or topological relations between the zones’ spatial extensions. In this regard, a data format for representing both spatial properties and determined relations within self-descriptions has been developed. A central point is the abstraction of spatial relations in a qualitative way, depending on the application domain which influences their semantics. As the relations an artifact is aware of can also be included in its self-description and exchanged with others, digital artifacts are able to reason about both self-determined and received relationships. A literature survey has been conducted in order to find suitable qualitative abstractions of relations between Zones-of-Influence on the one hand, and to identify approaches for inferring new relations on the other hand. We have developed a spatial calculus for compositional reasoning about positional relations as well as an algorithm which makes use of it for inferring and distributing spatial relations among autonomous digital artifacts over multiple hops, without the need for exchanging quantitative spatial properties between those that are out of communication range. In this regard, effects on the induced network traffic as well as the achieved accuracy have been evaluated by simulation means. A reference architecture for the spatial awareness of digital artifacts has been developed, demonstrating how the above concepts for recognizing, representing and reasoning about spatial relations among spontaneously interacting artifacts can be realized. A middleware framework has been prototypically implemented, with the aim to support the development of spatially aware applications for autonomous digital artifacts. For reasoning about spatial relations over time, in order to infer new relations or trigger appropriate application-level actions, this so-called Zonesof-Influence Framework makes use of rules. All relations are associated with time
1.3 Thesis Outline
7
intervals in which they exist, and new high-level relations are inferred by combining such intervals, for which reason inferred relations can in turn be combined with others. The qualitative relationship abstractions are encapsulated in components that can easily be exchanged at runtime, enabling artifacts to adapt the availability of spatial relations and their semantics to the current application demands. The proposed framework with its underlying concepts provides powerful means for facilitating the development of applications which take into account the contextual relations among autonomous digital artifacts. An evaluation with multiple application scenarios showing the relevance of spatial relations for developing spatially aware applications as well as the technical feasibility and quality of the Zones-ofInfluence Framework has been conducted.
1.3 Thesis Outline The thesis is structured as follows. Chapter 2 gives an introduction in contextawareness of autonomous embedded systems, with which physical objects are augmented and to which we refer to as digital artifacts then, motivates its significance for mechanisms of self-organization, discusses the role of spatial context with regard to their autonomy and semantically meaningful interaction among each other, and presents a general architecture for providing spatial awareness to artifacts. At the end of this chapter, a comprehensive survey and comparison of related work are given. Chapter 3 elaborates on quantitative and qualitative representations of spatial aspects, which are the basis for representing spatial properties of artifacts with so-called Zones-of-Influence as well as for recognizing and using spatial relations between them. In this regard, the exchange of such spatial contexts among artifacts with structured self-descriptions is discussed. Chapter 4 builds upon the qualitative abstractions of spatial relations and is concerned with the inference of knowledge from them. In this chapter, fundamentals of qualitative reasoning approaches are presented, and an algorithm for inferring and distributing qualitative spatial relationships among digital artifacts is proposed and evaluated. In Chapter 5, a rule-based approach for reasoning about qualitative spatial relations over time is presented, which is used for inferring new relations and triggering application-level actions upon observing certain patterns on the stored relations. The above concepts of exploiting spatial abstractions have been implemented in the flexible and modular Zones-of-Influence Framework presented in Chapter 6, which enables digital artifacts to maintain and use a spatial model of their environment. An overview of the architecture is given in this chapter, its components for supporting spatial awareness are described and implementation details as well as
8
Introduction
open issues are discussed. Chapter 7 eventually evaluates the implemented middleware framework by means of multiple real-world applications scenarios which cover the concepts presented in the preceding chapters. In Chapter 8, the thesis is summarized, the main contributions are discussed and an overview of open issues for possible future work is given.
2 Spatial Awareness The real power of the concept comes not from any one of these devices; it emerges from the interaction of all of them. Mark Weiser, 1952-1999
This chapter introduces spatial awareness of autonomous embedded systems, which is the main subject of this work. In Section 2.1, we first give an overview on context-awareness, which denotes a system’s property to acquire and use information about its environment, as well as on the abstraction of context for developing context-aware applications. Afterwards, Section 2.2 presents the components of autonomous embedded systems which are able to sense their context, process and reason about it and share this knowledge with others in communication range. These abilities contribute to their autonomous operation in the background and thus to a reduction of human involvement for their management and configuration. Mechanisms of self-management of individual devices and self-organization of spontaneous configurations of multiple devices are discussed in Section 2.3, which rely on the perception of context information and its local exchange among devices in range. In this regard, our focus is on the relevance of spatial information, as we are dealing with real-world objects which are distributed in physical space. The subsequent Section 2.4 thus deals with spatial context information, whereas a classification is given and our focus on spatial relations as well as their changes over time is emphasized. Afterwards, an architecture for spatial awareness is presented in Section 2.5, and Section 2.6 finally provides a comprehensive survey and comparison of related work addressing the issue of how to make autonomous embedded systems aware about abstracted spatial relations over time.
2.1 Context and Awareness Subject of the field of pervasive computing is the integration of technology into everyday objects. With computing power invisibly integrated in a huge number of networked objects it becomes necessary that they operate more autonomously,
10
2 Spatial Awareness
as the embedding of digital technology leads to constrained human-computer interfaces on the one hand, and the ubiquity of such technology-enriched physical objects makes it impossible for humans to individually manage their operation on the other hand. A contributor to the autonomy of embedded systems is contextawareness [SAW94, Sch95], which refers to the ability of a system to acquire environmental information in order to understand its situation or context – e.g. its location or spatial relations to other systems – and adapt to changing contexts at runtime. Context-awareness is considered important especially for applications where the environment changes frequently [Dey00, SAW94], which is commonly the case in mobile and pervasive computing and requires an adaptation to the current context of use. A survey on the foundations of context-aware computing and first research projects is given in [CK00]. Several definitions of context and awareness can be found in literature. In the field of computer supported cooperative work (CSCW), [DB92] considers an awareness of the others’ activities necessary for successful collaboration, and defines awareness as an understanding of the activities of others, which provides a context for your own activity. Awareness is usually supported with mechanisms for sharing information between collaborators in CSCW systems, as for example by means of document annotations or electronic conferencing systems. In the field of pervasive computing, the more specific term context-awareness refers to a system’s awareness of the context in which it is used. It has been introduced in [SAW94], where contextawareness is defined as a system’s ability to adapt according to the location of use, the collection of nearby people, hosts, and accessible devices, as well as to changes of such things over time. Three important aspects of context pointed out in [SAW94] are where you are, who you are with and what resources are nearby. A later definition of context is given in [BBC97], where it is also defined by means of examples, including the user’s location, the time of the day and the temperature. From these definitions, it can be observed that location and spatial proximity are considered important context information; in fact, [CK00] states that few contexts other than location have been used in actual applications. Our literature survey conducted in Section 2.6 leads to a similar result, namely that location and proximity are the most often used contexts in spatially aware applications (i.e. applications that are aware of spatial context information, cf. Section 2.5). Time is also considered important for context-aware applications [CK00, Dey00], and it allows for reasoning about
2.1 Context and Awareness
11
both contexts and their changes over time. However, there are many other types of context information, which [SBG99] classifies in (i) human factors (i.e. information on the user, his social environment and his tasks) and (ii) information on the physical environment (i.e. location, infrastructure and physical conditions). A more general definition of context can be found in [Dey00]: Context is any information than can be used to characterize the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and application themselves. According to this – probably most often used – definition, it depends on the particular application scenario what kinds of information are context in that they have an influence on the application. We adopt this definition for our work, and consider context as any information than can be used to characterize the situation of a person or an object which is considered relevant for an application. In order to be able to use context information – i.e. information about a system’s physical environment – in an application, it has to be acquired with sensors first. Research on context-awareness is thus driven by technological advances to a certain extent, and many projects had a focus on technologies for sensing various physical contexts like location [WHFG92], direction [SBG99], light, noise, movement, skin conductance and temperature [BGS01, GSB02, HMS+ 01] or relative orientation and distance [HKG+ 05]. With emerging micro-electro-mechanical systems (MEMS) technology, smaller, more powerful and cheaper sensors are becoming available [GSB02] (e.g. MEMS-based accelerometers and gyroscopes for sensing linear and angular acceleration, respectively), which facilitates their integration into everyday objects at low cost. If the application has to have an impact on the physical world, actuators are required for changing the environment’s state. An actuator is any output to the physical world (cf. [Dey00]), as for example a mechanical switch, a light-emitting diode or a computer display. According to [Fer03], sensors and actuators can be considered as the input/output interface between the physical and the virtual world. For using sensed context information at the application level however, it must be abstracted to decouple sensor details that are not relevant for the application and therewith allow application programmers to use context at a higher level of abstraction [Dey01, Sch95], namely without having to consider low-level aspects of certain sensor technologies. This is a necessary step as sensors often do not provide context information in the form required by an application [SDA99]. With regard to the spatial focus of this thesis, sensor data has been abstracted with symbolic locations [Leo98] and qualitative spatial relations [KKG05], representing the
12
2 Spatial Awareness
respective spatial context at a high level and more naturally by using semantically meaningful names. As will be seen in Section 2.6, abstractions of sensors are actually employed in most projects which make use of context information. The acquisition, abstraction and use of context information are essential components of context-aware architectures. A more detailed view on a possible architecture of a context-aware system is given in [Fer03, FHO04a], where the following layers are in-between those for sensing and actuating: (i) context transformation (i.e. aggregation and interpretation of the low-level contexts into high-level information which is meaningful for the application), (ii) context representation (i.e. data structures representing the abstract context information of the previous layer), (iii) context prediction (i.e. inference of future from past contexts) and (iv) context triggering (i.e. rules for triggering actuators upon observing certain context conditions). Similar layered architectures are common in context-aware systems (e.g. in [Dey00, GSB02, HKG+ 05, LM98, SGKK04]), as is the use of rules for defining context-dependent application behavior (e.g. [BCFM03, GSB02, SAW94, SGKK04, TTH+ 04]). In our framework outlined in Section 2.5 and refined later in Chapter 6, sensed context goes through similar layers of abstraction until reaching the application, and a rule-based approach has been chosen for defining application-level behavior.
2.2 Autonomous Embedded Systems As stated in the previous section, context-awareness is a crucial contributor to the autonomy of embedded systems, which are considered as computer systems that are parts of larger systems and realize dedicated functions. In this section, the components of autonomous embedded systems are discussed in more detail and with respect to existing research, as well as their mechanisms for sharing contextinformation with other systems in order to allow for spontaneous contextual interaction with them. The following explanations are in large parts limited to aspects which are relevant regarding the conceptual and architectural contents in the remainder of this thesis. Since Mark Weiser’s seminal paper [Wei91], there has been a considerable amount of research on the embedding of computing technology in the physical environment. An example are wireless sensor networks [ASSC02], which typically consist of a huge number of sensor nodes deployed in an environment in order to sense certain phenomena. The nodes perform local processing on the sensed data and cooperate with other nodes in routing it back to a sink by means of multi-hop wireless communication. Another related field of research is tangible
2.2 Autonomous Embedded Systems
13
user interfaces [Fis04, IU97], which deals with user interfaces that augment the real physical world by coupling digital information to everyday physical objects and environments [IU97]. We also conducted research in this area by studying the use of physical objects with embedded direction sensing for the purpose of device control (cf. [FHR06b, FRH06, FRHR05, HRLF06]). The starting point of our work are physical artifacts augmented with autonomous embedded systems, which – in contrast to the two above examples – comprise sensing, actuating, computing and wireless communication capabilities, and allow for the autonomous provision of contextualized services by sharing information among each other without relying on an external infrastructure. The general aim of augmenting artifacts that are not computers themselves with computing technology is stated in [BGS01]: Computer-augmentation of artefacts will not be geared toward making them more computer-alike but at preserving their individual purposes and uses while enabling added value through digital information processing. The embedding of computing capabilities in everyday artifacts, and in particular their context-awareness and inter-networking capabilities, were subject of many projects surveyed in Section 2.6, as for example [BGS01, DBK+ 04, FHdSR+ 07, HMS+ 01, SGKK04]. We adopt the term digital artifact (often simply referred to as artifact in the following) for computer-augmented physical artifacts, which has been defined in [BGS01] as follows: A digital artefact is an everyday artefact augmented with computing and communication, enabling it to establish and exchange information about itself with other digital artefacts and/or computer applications. The concepts and solutions for spatial awareness developed in this thesis are based on digital artifacts. They consist of the following components also shown in Figure 2.1, which are either part of the augmented artifact’s original components or have been added by embedding computing and communication technology: • Sensors: acquisition of context information from the environment, e.g. using location, motion or optical sensors (cf. [SBG99]) • Actuators: output to the physical world, e.g. by creating mechanical motion or with light-emitting diodes
14
2 Spatial Awareness
Figure 2.1: Components of a digital artifact.
• Communication facilities: ad-hoc exchange of data with other artifacts and IT-systems in range, e.g. using IEEE 802.11 or IEEE 802.15 wireless communication • Runtime system: representation and reasoning about context information, which is either acquired with local sensors or received from other artifacts, and triggering respective application actions based on the current context • External services: provision of contextualized functionality to the artifact’s surroundings, as for example to users or other artifacts A digital artifact is typically a mobile device with a small form factor, constrained computing capabilities and limited energy resources due to battery operation, it is context-aware because of its sensors and operates autonomously (i.e. without centralized control), and it acts according to certain defined goals – like for example the provision of a certain service to the user – which may be achieved in cooperation with other artifacts. As this thesis is primarily concerned with software concepts for spatial awareness at the application level, hardware aspects regarding the embedding of miniaturized computing technology in everyday objects are beyond its scope. For evaluation purposes, notebook computers with attached sensors and actuators have been used. A possible platform for digital artifacts are Peer-its [FHdSR+ 07], which comprise the components listed above and are capa-
2.2 Autonomous Embedded Systems
15
ble of discovering and spontaneously interacting with other such artifacts (cf. Section 2.6.1 for more details).
2.2.1 Exchange of Self-Descriptions Clearly, information sharing is a key property for the collaboration among artifacts and their autonomous adaptation to changing conditions at runtime. Within the scope of this thesis, our focus is on sharing context information among digital artifacts, which we consider fundamental for a semantically meaningful, contextual interaction among them and thus beneficial to their autonomous operation in the background. According to [BGS01], context-awareness and ad-hoc information sharing are essential properties of digital artifacts, and they are enabler for emerging functionality (cf. Section 2.3). Ad-hoc context sharing has been the basis for interaction in several of the projects presented in Section 2.6.1; for example, [BGS01] employs message-based communication for broadcasting context information of artifacts via infrared transceivers mounted on the ceiling, [HMS+ 01] uses wireless communication for sharing contexts like e.g. movement patterns among artifacts in communication range, and XML-based documents containing information about the connections and spatial directions of physically connected artifacts are exchanged via a wired network in [KOY+ 05]. We employ peer-to-peer concepts for the interaction between digital artifacts in a spontaneous ad-hoc networking fashion, which appear to be more suitable than client-server architectures due to the decentralized nature of networked digital artifacts. In order to share context information, so-called self-descriptions are used for representing the contexts of artifacts and exchanged among them upon coming into communication range; each artifact then performs a pairwise comparison of its own with the received self-descriptions and triggers certain actions (cf. Figure 2.2). Hence, self-descriptions build up a standardized interface for the sharing of context information, and their ad-hoc exchange provides mutual awareness about e.g. physical properties and contexts to artifacts, which facilitates their interaction with regard to an open-world assumption (i.e. the interacting artifacts do not know each other in advance). This approach builds on previous work presented in [FHdSR+ 07, FHM+ 04, FHR+ 06a], where the authors suggest to use XML-based documents representing information about an artifact and exchanging them with other artifacts in range; upon sufficiently high similarity between two such documents at a time, the respective artifacts start to interact. Similarly, self-descriptions in this thesis are structured documents based on XML which contain information that may be relevant for the interaction among artifacts, as for example an artifact’s physical properties
16
2 Spatial Awareness
Figure 2.2: Exchange and comparison of self-descriptions.
like its size or weight, its capabilities in terms of provided services or spatial context information like position and direction. It enables artifacts to autonomously become aware of the contexts of others in a decentralized way, but has the disadvantages that just contexts of artifacts in communication range can be accessed and each artifact has to store and maintain all the information it may require. The scope of the present thesis is on the exchange of spatial information, whereas a detailed explanation of our representation of spatial contexts in self-descriptions, which we kept as simple as possible for the purpose of realizing and evaluating the proposed concepts, is given in Section 3.3. However, issues of how self-descriptions are exchanged from a technological viewpoint, under which circumstances (e.g. taking into account energy constraints) and among which artifacts (e.g. with respect to certain communication policies) are not considered.
2.3 Mechanisms of Self-Organization In the introduction, we argued that the growing number of computationally enhanced and increasingly interconnected, heterogeneous and complex everyday artifacts – be it home appliances, cars or mobile phones – calls for mechanisms of self-management and -organization which allow them to operate autonomously in the background, with no or little human interaction. The reason for that is that systems have become too complex and heterogeneous for humans to be able to use them without difficulties [HG03, KC03, MMTZ06]. According to [Fer07b, FHdSR+ 07], self-management stands for the ability of individual artifacts to operate autonomously by adapting to changing contexts at runtime, while self-organization stands for the spontaneous contextual interaction of multiple devices in order to achieve collective behavior. One approach to self-management is IBM’s autonomic computing vision [KC03], in which systems are able to self-configure, -heal, -optimize and -protect and thus function more or less independently from
2.3 Mechanisms of Self-Organization
17
human supervision, just based on high-level administrative goals and constraints. Self-organization goes further by not only autonomously controlling a system’s behavior, but creating its own structure which is not explicitly defined by a programmer. Self-organization has received much research attention in computer science over the past years, as can be seen from several recent survey papers [Mil07, MMTZ06, SFH+ 03]. In particular, mechanisms of self-organization which are inspired by nature (e.g. ants food foraging or nest building [MMTZ06]) have attracted the attention of computer scientists [KE01, MMTZ06, ZGMT04]. In the context of pattern formation in biological systems, [CFS+ 01] defines self-organization as follows: Self-organization is a process in which pattern at the global level of a system emerges solely from numerous interactions among the lowerlevel components of the system. Moreover, the rules specifying interactions among the system’s components are executed using only local information, without reference to the global pattern. According to [CFS+ 01], patterns – i.e. organized arrangements of objects in space or time – are emergent properties, which means that they are produced without external input but rather result from interactions among system components that are based on local information only. A prominent example is the cellular automaton “Game of Life” [Gar70]; it consists of a two-dimensional orthogonal grid of cells which are either white (“dead” cells) or black (“alive” cells). The cells interact with their eight neighbors at a time only, and may change – simultaneously, at discrete points in time – by using three simple rules: (i) each white cell with exactly three black neighbors becomes black, (ii) each black cell with more than three or fewer than two black neighbors becomes white and (iii) each black cell with two or three black neighbors remains black. Interestingly, although just simple rules about a cell’s local neighborhood are used, complex patterns such as the so-called “glider” shown in Figure 2.3 may emerge depending on the automaton’s initial configuration. A related approach is amorphous computing [AAC+ 00], which differs from cellular automata in that the “agents” are not regularly placed on a grid and that they do not operate synchronously [JBL+ 06]. A more general definition is given in [WH04], where the authors define selforganization as a dynamical and adaptive process where systems acquire and maintain structure themselves, without external control. In this regard, “structure” can be a spatial, temporal or functional structure, and “without external control” refers to the absence of any control instructions from outside the system, which does not exclude data inputs however. We adopt this definition for our work, and consider systems as self-organizing in some respect if they spontaneously, and de-
18
2 Spatial Awareness
Figure 2.3: Emerging “glider” pattern in “Game of Life”.
pending on their determined context, interact with other systems in order to achieve collective behavior. Following the considerations in [BGS01, MZ05], we identify two enabling properties for the self-organization of autonomous embedded systems: (i) contextawareness (e.g. using local sensors for acquiring physical contexts, cf. Section 2.1) and (ii) ad-hoc context sharing (e.g. via an exchange of self-descriptions, cf. Section 2.2). These two properties in combination allow for developing applications whose functionality not only depends on the individual contexts of autonomous embedded systems, but on the co-existence of other such systems and their contextual relations to one another. In this regard, the present thesis covers self-organization insofar as it provides enabling concepts and solutions regarding the awareness of digital artifacts about spatial properties and relations among each other, as well as the exchange of such spatial contexts with other artifacts in range. In the pervasive computing literature, a number of projects can be found which exhibit some forms of self-organization. So as to clarify what we mean by selforganization in the context of this thesis, a few examples – taken from projects which are explained in more detail in Section 2.6 – will be given in the following. In [KOY+ 05] for example, so-called u-Textures (i.e. digital artifacts equipped with displays and direction sensors) can be assembled together, whereas (i) each of them autonomously adapts its behavior according to the current shape of the overall structure and its spatial relation to it, and (ii) the assembled u-Textures as a whole cooperatively run a corresponding application. Another example are Mediacups [BGS01, HB00] (i.e. coffee cups with embedded computing, sensing and communication capabilities) that have been used for a smart doorplate application, in which an interactive display (the doorplate) adapts its content based on both the spatial co-location of Mediacups and their temperature, which are context information that is shared among those digital artifacts and thus enables the emergence of functionality by spontaneous interaction. The TOTA (“Tuples on the Air”) middleware [MZ04, MZ05] is an approach for building distributed data structures through multi-hop propagation of tuples containing both content and propagation rules, by means of which self-organizing applications e.g. for distributed motion coordina-
2.3 Mechanisms of Self-Organization
19
tion can be realized. Further projects which show mechanisms of self-organization can be found in [BIK+ 04, KKP99, LSBP02, ZGMT04], among others.
2.3.1 Self-Organization in Space We consider spatial context information valuable for mechanisms of self-organization among digital artifacts, as they are by nature situated in physical space and thus spatial properties as well as spatial relations between them, which are particularly important due to the fact that interaction is based on local information, are crucial for characterizing an artifact’s situation. From the examples above, it becomes apparent that self-organization is often related to spatial aspects like the position of artifacts or their distance- and orientation-relations to each other, and [ZM04] actually states that a vast majority of known phenomena of self-organization and self-adaptation in nature [...] are actually phenomena of self-organization in space [...]. In [Zam04], it is argued that spatial concepts play an important role in distributed computing scenarios like sensor networks, mobile ad-hoc networks and peer-to-peer (P2P) networks, and that such distributed systems require some forms of self-organization which can be based on a spatial computing model. According to [MZ05], spatial self-organization can be found at different levels in distributed systems: at (i) the physical level which enables spontaneous interaction between system components (e.g. radio broadcasts), (ii) the structure level at which spatial structures emerge (e.g. beacon-based self-localization), (iii) the navigation level which exploits available spatial structures (e.g. for geographical routing) and at (iv) the application level where the underlying navigation mechanisms are used (e.g. pattern formation). As the basis of a spatial computing model, [Zam04] proposes the exploitation of spatial abstractions, meaning that the activities of application components are abstracted as taking place in some sort of abstract metric space, and for which self-organization derives from the autonomous capability of components of sensing, acting in, and navigating that space. Thus, instead of addressing entities by their names, their positions in space are used for interaction purposes. In a similar spirit, we propose the use of spatial abstractions for the implementation of spatially aware applications in this thesis, but with a focus on physical objects and their relations among each other. We therefore distinguish the following two types of spatial abstractions, which are in detail discussed in Chapter 3: (i) quantitative abstractions by means of one or more spatial zones representing an artifact’s position, direction and spatial extension, and (ii) qualitative abstractions of relations
20
2 Spatial Awareness
between such zones representing relationships between artifacts that have a certain meaning for the application.
2.4 Spatial Context in Time The focus of this thesis is on the awareness of digital artifacts about their spatial contexts, which is important information characterizing the situation of artifacts that are by nature distributed throughout physical space. Similar to [CFH97], we classify spatial context information both in terms of the artifacts’ inherent characteristics and with respect to other artifacts, and distinguish two corresponding types of spatial context information: (i) the spatial properties of artifacts, which are independent of other artifacts, and (ii) spatial relations between artifacts – or rather relations between their spatial properties. Spatial properties and relations are valuable context information for a variety of applications, and they constitute a distinguishing feature of this work. Awareness about spatial contexts is a precondition for the autonomous adaptation of digital artifacts to changing spatial environments, and spatial relations among them – as well as relationship changes over time – are particularly relevant with regard to their spontaneous contextual local interaction [HHS+ 02, HKG+ 05, Sch02]. In fact, location information is the most prevalent type of context in context-aware applications (cf. [CK00, SBG99]), and our related work survey of Section 2.6 also shows that location and proximity are the most commonly used spatial contexts. In Table 2.4, which was inspired by [CFH97, CH01], three types of spatial properties of an artifact are distinguished: (i) its position (synonymously used with the term location in this thesis), namely where it is located in physical space, (ii) the direction in which it is aligned and (iii) the artifact’s spatial extension comprising its shape and size. It should be noted that the terms orientation and direction are not used consistently in literature, and sometimes inverse to our notion. Correspondingly, three types of spatial relations between the spatial properties of two artifacts at a time are distinguished: (i) positional relations, which include both orientation and distance relations between the positions of two artifacts, (ii) directional relations, which are relations between two artifacts’ axes of direction, and (iii) topological relations, namely relations concerning the spatial arrangement of extended objects which is defined by their extensions in addition to position and direction. A simple visualization of static properties and relations with a rectangularand a triangular-shaped artifact can be seen in Figure 2.4. The actually used relations depend on application demands, but also on the sensing and processing capabilities of the involved digital artifacts; for example, the consideration of spa-
2.4 Spatial Context in Time
21
Table 2.1: Static characteristics of an artifact’s spatial situation.
Position Direction Extension
Spatial properties
Spatial relations
geographic position intrinsic direction axis shape and size
orientation and distance relations relations between direction axes arrangement of extended objects
Figure 2.4: Static spatial properties and relations of artifacts.
tial extensions – especially when freeform shapes are involved – requires much computational power, which may exceed the capabilities of embedded systems with limited resources. Within the scope of this thesis, spatial context can be given either absolute, namely with respect to a fixed external frame of reference such as the earth reference frame, or relative, namely with respect to the reference frame of an artifact. This distinction between absolute and relative space ranges back to a historical correspondence [LCA98] between Leibniz and Clarke, who wrote Newton’s letters, in the eighteenth century. For Newton, space was an infinite and immovable three-dimensional container with its origin at the center of the universe and which exists whether or not anything is in it [Lev96]. From their correspondence, it is obvious that an absolute space was not a meaningful concept for Leibniz; instead, he considered space as something merely relative, as time is, namely as an order of coexistences, as time is an order of successions [LCA98]. Hence, relative space according to Leibnitz’ view is basically a means for describing relations among objects. We will come back to the distinction between absolute and relative space in Section 3.2.4, where reference frames for qualitative spatial relations are discussed.
22
2 Spatial Awareness
Another distinction of spatial contexts concerns its representation, where we distinguish two types: quantitative and qualitative representation. We use the former for representing spatial properties of artifacts by means of numerical values, and according to certain reference systems in order to ensure a common understanding between artifacts, which is a precondition for the recognition of spatial relations out of them. An example are WGS84 coordinates used by GPS, which represent absolute positions with numerical longitude, latitude and altitude coordinates (cf. Section 3.1). For the representation of spatial relations, qualitative abstractions are used, which have advantages compared to quantitative ones whenever the spatial cognition of humans is involved [Mus00] for example. Qualitative spatial relations are based on discrete symbols such as left or near, representing continuous properties such as orientation or distance in an abstract way which is meaningful for a certain application [CH01]. The six types of spatial context information shown in Table 2.4 are referred to as static spatial context in that they describe an artifact’s spatial situation at a particular point in time. According to [AEG94], the position, direction and extension of an artifact – i.e. its spatial properties – can be changed through translation, rotation and scaling respectively, determining how its spatial relations to other artifacts are changed. The latter are thus referred to as dynamic spatial context, as they describe how its static situation is changing at a certain point in time. We use a discrete time for the sensor observations, which meets the requirements of computing systems with discrete clock cycles [Mus00]. In addition, the spatial context of an artifact need not correspond to a certain point in time, but may also represent an artifact’s situation over a time period by comprising a time series of spatial contexts. Summing up, we distinguish four categories of spatial context information within the scope of this thesis: • Absolute (i.e. with respect to a fixed external frame of reference) vs. relative (i.e. with respect to the reference frame of an artifact) • Quantitative (i.e. using numerical values) vs. qualitative (i.e. using discrete symbols which represent intervals of quantitative values) • Static (i.e. at a certain point in time) vs. dynamic (i.e. the degree of change at a point in time) • Point in time (i.e. a snapshot at a certain time) vs. time series (i.e. a trajectory of spatial contexts over a sequence of points in time) With respect to the above classification, the focus of this thesis is on (i) quantitative spatial properties of artifacts given with respect to an absolute frame of
2.5 An Architecture for Spatial Awareness
23
reference, and on (ii) qualitative spatial relations, which are defined either absolute or relative to another artifact; in Chapter 3, these two types of spatial context will be discussed in detail. While in the case of spatial properties neither qualitative representations (e.g. symbolic locations [Leo98, Sch95]) nor dynamic contexts (e.g. velocity or acceleration with respect to the earth reference frame) are considered in the following, spatial relations are – based on the respective quantitative representations – abstracted in a qualitative way, and both static and dynamic relations between artifacts are taken into account. In contrast to spatial properties, where our scope is on spatial context given at points in time, we also consider time series of spatial relations as will be seen in Section 3.2.5.
2.5 An Architecture for Spatial Awareness The preceding sections elaborated on context-aware digital artifacts, their autonomous operation and self-organization as well as our focus on spatial context information in this regard. Based on these considerations, we have developed a multilayered architecture for providing spatial awareness to digital artifacts, which is an awareness about spatial contexts as discussed in the preceding section. We thus define spatial awareness as follows: A system is spatially aware if it is able to determine and use its spatial context, which comprises an awareness about the system’s spatial properties as well as about spatial relations to other systems. According to the focus of this thesis stated in Section 1.1, our scope is on spatial relationships between digital artifacts. Figure 2.5 shows the simplified architecture for spatial awareness, which has first been published in [HF07] and builds up the basis for the contributions in this thesis. Its comprises layers for (i) the acquisition of spatial context information, (ii) the recognition, representation and reasoning about qualitative spatial relations, and (iii) the provision of services for the development of spatially aware applications. It has a similar structure like the contextaware architecture proposed in [Fer03], except for the explicit distinction between the direct (with sensors) and indirect (via communication) way of acquiring spatial contexts as well as the focus on spatial relationships. We have implemented a middleware framework based on this general architecture, which will be presented in Chapter 6. In the following, the layers of the architecture are discussed in more detail. A precondition for recognizing spatial relations is the mutual awareness of artifacts about their spatial contexts, which requires two architectural components at
24
2 Spatial Awareness
Figure 2.5: General architecture for spatial awareness.
the lowest layer of the architecture: a sensing and a communication module. The sensing module collects sensor readings from the artifact’s spatial sensors (e.g. a GPS receiver and a digital compass for acquiring position and direction with respect to the earth reference frame), and optionally transforms them into a certain reference system such that they are comparable with the spatial contexts of other artifacts. Our focus is on sensors which provide quantitative data (i.e. numerical data, in contrast to e.g. symbolic locations [LM98]) with respect to an external reference system (e.g. the earth reference frame or the working area of a certain indoor tracking system), thus representing an artifact’s spatial properties as defined in Section 2.4. More details about reference systems and the representation of spatial properties with so-called Zones-of-Influence will be given in Chapter 3. Although out of scope, our architecture would also allow to incorporate sensors which directly provide spatial relations, like for example the ultrasonic sensor presented in [HKG+ 05] or by interpreting signal strength measurements as spatial proximity [FHO04a]. The second component contributing to mutual spatial context-awareness is the communication module, which is concerned with exchanging spatial information via self-descriptions with artifacts in communication range. Each artifact collects its own spatial context and gets aware of the others’ contexts through an exchange
2.5 An Architecture for Spatial Awareness
25
of self-descriptions as discussed in Section 2.2.1. It should be noted that an artifact’s spatial context may contain both its spatial properties and spatial relations it is aware of, which is – among other things – a precondition for our approach of inferring and distributing spatial relations presented in Section 4.3. An artifact a for example, which is aware that another artifact b is near, could include this qualitative spatial relation in its self-description. Exchanging spatial relations with self-descriptions thus enables them to become aware of relations they cannot recognize directly, as for example due to missing sensor data or because of the fact that the respective artifacts are out of range. For the above example, an artifact c, which is in a far relation to a and receives its self-description containing the distance relation between a and b, could then conclude that b is not near artifact c (given that the metric distance of far is greater than double the range in which the relation near holds). The core of the architecture is the relationship recognition layer, which is concerned with recognizing and abstracting binary spatial relations with respect to the recognizing artifact (i.e. just relations between the recognizing and one other artifact are considered at a time). Relations are recognized by a pairwise comparison of an artifact’s own with a received self-description, or – more precisely – of the quantitative spatial properties contained therein. The recognition step is performed separately for each received self-description; in this regard, the recognition of relations as well as their qualitative abstraction (i.e. by means of symbolic names like near and far) depends on the availability of sensor data and on the current application demands. Recognized relations are eventually stored in the artifact’s local repository, and provided for the purpose of spatial reasoning or directly for the use by spatial services. A comprehensive discussion on qualitative relations and their use for achieving spatial awareness of digital artifacts will be given in Section 3.2. The spatial reasoning layer is responsible for reasoning about (i.e. inferring conclusions from) the history of relations stored in the repository. In general, known qualitative relations are combined and respective actions are executed if they fulfill certain predefined conditions. We decided for a rule-based approach in order to infer new (high-level) relations or trigger application-level actions upon observing certain patterns on the stored relations. Such a pattern could be the simultaneous existence of a distance relation near and an orientation relation front to the same artifact, meaning for example that it is positioned within the recognizing artifact’s physical interaction range. Detailed information about reasoning mechanisms for qualitative spatial relations can be found in Chapter 4, our approach for rule-based reasoning will be presented in Chapter 5. At the top of the architecture, spatial services are provided to spatially aware applications. Examples are a service for
26
2 Spatial Awareness
querying the relations repository or a communication service for exchanging messages with artifacts to which a certain spatial relation exists.
2.6 Related Work This subsection finally discusses work which is related to the spatial awareness of autonomous embedded systems. According to our observations, the use of spatial relations – and particularly their qualitative abstractions – is a sparsely investigated topic in the field of pervasive computing. Although there has been much work on location systems providing symbolic (i.e. non-numerical) location information (e.g. [LM98, RAMC+ 04, WHFG92]), there are just a few projects dealing with spatial relations at the application level. In this respect, the work closest to ours is the Relate project [HKG+ 05, KKG05], which provides mobile devices with an awareness about both quantitative and qualitative spatial relations to other devices around; however, it differs in how relations are acquired and abstracted, and no rule-based approach for reasoning about them is utilized. The Cooperative Artefacts project [SGKK04, SKGK04] uses rule-based reasoning for inferring knowledge and triggering actions, but does not consider spatial relations other than proximity. Another closely related project is Mediacup [BGS01] with its spatially aware RAUM communication model [HB00] for the interaction based on topological relations. However, neither domain-specific qualitative abstractions of spatial relations nor rule-based reasoning about them by considering qualitative temporal relations seem to have been addressed in other work, and they constitute two of the novelties of this thesis. Another issue of research tackled in our work is the sharing of context information among artifacts which has been used in other projects also (e.g. [BGS01, FHR+ 06a, HMS+ 01, KOY+ 05]), but not – as proposed in this thesis – with structured documents containing both spatial properties and relations an artifact is aware of. Mutual knowledge about the artifacts’ contexts is a precondition for selforganizing behavior, as demonstrated for example in the smart doorplate application in [HB00] or the self-organization of u-Texture panels depending on their spatial configuration in [KOY+ 05]. Within this thesis, we have developed an algorithm for inferring and distributing qualitative spatial relations between artifacts over multiple hops (cf. Section 4.3), which is self-organizing in that they autonomously establish an awareness about spatial relations among each other in an ad-hoc manner. There are related approaches for localization in ad-hoc networks [BSHK06, CHH02], but ours seems to be the only one which relies on purely qualitative spatial relations.
2.6 Related Work
27
The following subsections contain a detailed discussion of related work. In Section 2.6.1, projects which contribute to making autonomous embedded systems spatially aware are surveyed and discussed with respect to their relation to this thesis. As there are many publications on related topics available already, just those which are closely related to the focus of our work are considered. This includes projects dealing with context-aware frameworks and middleware in general, systems and technologies for making spatial information available to applications, and projects which address the acquisition and use of spatial relations between objects. A comparison of selected projects with regard to the contributions of this thesis is given in Section 2.6.2. Related work that is specific to the representation of space – in particular the qualitative representation of spatial relations – as well as reasoning about qualitative spatial relations can be found in the Chapters 3 and 4, respectively.
2.6.1 Projects University of Toronto (US) One of the first projects about spatial awareness beyond pure positional information is presented in [Fit93]. The Chameleon prototype has been developed, a portable spatially aware palmtop computer for visualizing and manipulating electronic information which is associated with physical objects in the real world; it can thus be seen as a window to the three-dimensional information space which is situated in a physical context. The palmtop computer is aware of its position and direction in physical space by the help of a 6-D input device attached to its display, which allows for example to display additional information to a map depending on the palmtop’s position and orientation relative to the map. A graspable user interface for manipulating electronic objects on a horizontal display by means of one or more physical artifacts, which are referred to as bricks, is presented in [FIB95]. As in [Fit93], position and orientation of the bricks are again sensed with a 6-D input device mounted on the table. Context Toolkit The Context Toolkit (http://www.cc.gatech.edu/fce/contexttoolkit), which has been presented in [Dey00, SDA99], facilitates the development of context-aware applications by building upon the widget concept from GUI toolkits. It provides abstractions of low-level context information acquired from sensors according to the application requirements by means of several components: (i) “widgets” which encapsulate sensors and make context available to applications and components, (ii)
28
2 Spatial Awareness
“aggregators” or “servers” which are special widgets that collect context information about certain entities like users from multiple widgets, and (iii) “interpreters” which can be used by widgets and applications to abstract context information into higher-level contexts. The Context Toolkit is based on a distributed architecture which uses XML and HTTP for communication between the components. It is related to the framework presented in this thesis in that it provides context abstractions to applications. In contrast, this thesis specifically addresses the abstraction of spatial relations and the development of spatially aware applications thereof; in this regard, our relation recognizers (cf. Section 6.2.3) are comparable to context widgets which also provide abstract context information and allow applications to register in order to be notified about context changes. Pin&Play and Pushpin Computing In the Pin&Play project (http://eis.comp.lancs.ac.uk/pin&play), which was a joint project of the Lancaster University (UK) and the Victoria Institute (SE), a new ad-hoc networking approach for everyday objects that can be spatially arranged on surfaces has been developed [LSG02]. The surfaces are augmented with layered conductive sheets, in order to serve both as a power supply for the objects as well as a communication medium which is based on a bus-topology. The objects are miniaturized computing platforms that can simply be attached to the surface with pin connectors. The Pin&Play approach thus corresponds to how people are used to attach objects to surfaces and freely arrange them in a spatially meaningful way. A similar infrastructure has been used in the Pushpin Computing project [LSBP02], which however uses the layered sheets for power supply only; Pushpins take advantage of capacitive coupling or infrared to communicate with other Pushpins in spatial proximity. The Pushpin platform is designed to explore principles of self-organization in distributed peer-to-peer sensor networks, by incorporating real-world sensor data instead of simulation. The programming model is inspired by the Paintable Computing work of [But02], and it uses process fragments containing executable code and state information which are executed on Pushpins and may be exchanged among them. Smart Dust The Smart Dust project [KKP99, WLLP01] at the University of California explored the integration of autonomous sensing, processing and communication capabilities in small devices of an intended cubic-millimeter scale, together with a power supply consisting of batteries and solar cells. These so-called Smart Dust
2.6 Related Work
29
motes use the optical medium for communicating with other motes or base stations, either passively using reflectors or actively with a laser diode and steerable mirrors, and they are based on micro-electro-mechanical systems (MEMS) technology for sensing and optical communication. [KKP99] points out the usefulness of self-organizing mote ensembles, which enable more sophisticated behavior by coordinating the individual motes’ actions, by means of an application scenario. Similar to Pushpin Computing [LSBP02], the project aimed at providing a platform for investigating distributed sensor networks in the real world. The TinyOS operating system (http://www.tinyos.net), which is already supported by numerous platforms and sensor boards, emerged from the Smart Dust project and provides an interesting future runtime system for our framework. Cooltown The goal of the Cooltown project [KBM+ 02] at the HP laboratories in Palo Alto was to associate people, places and things with a digital presence on the web, in order to support nomadic users, which are moving from place to place, with a ubiquitous, location-aware system. Location-awareness is provided by locationdependent services, which are based on URLs for addressing information, wireless beacons for broadcasting them in a limited area and client devices for discovering and using them. This approach allows to associate physical objects with links to information and services, and thus bridges the physical with the virtual world. Several applications have been implemented, as for example the Cooltown museum in which infrared transceivers broadcast URLs to additional information about nearby exhibitions, which is displayed on the PDAs of visitors passing by. In the present thesis, we do not focus on location as a context for service provision, but rather exploit spatial relationships between objects for influencing the application-level behavior. TEA The European Technology for Enabling Awareness (TEA) project (http://www. teco.edu/tea) investigated the development of a self-contained module which provides context-awareness as add-on component to mobile devices such as mobile phones and PDAs [GSB02]. The focus of this project was on the use of multiple low-cost sensors for resource-constrained devices, with the aim to acquire context information of their environment and situation which cannot be derived from location, and allow mobile devices to improve their services by adapting to the context of use. TEA uses a layered architecture which separates the context-layer from the
30
2 Spatial Awareness
sensor-layer, and thus provides means for modelling contexts as abstractions of real-world situations and independent of certain sensor technologies. The developed context-awareness module has been evaluated as add-on device for mobile phones, where sensors such as microphones, light-sensors and accelerometers are used to identify situations like in-hand, on-table or in-pocket, and the corresponding user-defined setting are automatically activated. With regard to this thesis, an interesting aspect mentioned in [GSB02] is the sharing of contexts between two mobile phones, by which the caller may be enabled to decide if it is a good time to call for example.
Mediacup The Mediacup project [BGS01] was like TEA concerned with the integration of multiple sensors for context acquisition, but it addressed the issue of how noncomputational everyday physical objects can be augmented with hardware and software for sensing, processing and communicating context information, without limiting their original appearance and functionality [GSB02]. According to the present thesis, the authors of [BGS01] refer to such objects as “digital artifacts”. The Mediacup has been developed, which is an ordinary coffee cup with invisibly integrated sensors for recognizing abstract contexts such as if the cup is stationary or if somebody is drinking out of it. The main research interest was on the emergence of new functionality from a spontaneous exchange of context information among networked artifacts, which are thus equipped with infrared diodes for communicating with transceivers mounted on the ceiling. The spatially aware RAUM communication model [HB00] has been used, which makes interaction based on spatial relations in the physical world instead of identities possible. It allows an artifact to define a geometric space – either stationary or relative to its own position – as a scope for communicating its context, which is only processed by other artifacts that are physically inside that space. This communication concept is demonstrated with a smart doorplate in [HB00], which defines the respective room as context communication space and displays the message that a meeting takes place whenever some hot Mediacups are co-located. A valuable extension of the RAUM model would be to consider not just topological “inside” relations, but to exploit other spatial relations such as distance, orientation and direction between artifacts for the definition of communication scopes, as for example to send messages to artifacts in front or accept messages from similarly oriented artifacts in spatial proximity only.
2.6 Related Work
31
eSeal Like the Mediacup, eSeal [DBK+ 04] is an example of a small digital artifact which autonomously, and without infrastructure support, acquires context information from embedded sensors and abstracts it to domain-specific context. The eSeal system is an electronic seal that can be attached to physical goods in order to monitor their integrity and notify about possible violations as well as the time and location of their occurrence. Different types of integrity are distinguished, including conditional integrity (i.e. the maintenance of given internal properties like temperature and light) and relational integrity (i.e. the maintenance of certain relations such as spatial proximity to other goods). Our work contributes to the latter by providing means for recognizing application-specific spatial relations and triggering certain actions when they are no longer fulfilled.
Smart-Its The project Smart-Its (http://www.smart-its.org) has been inspired by the TEA and Mediacup project, but changed the research focus from individual context-aware devices to ad-hoc networking and context sharing [GSB02]. Smart-Its are smallscale, embedded devices with sensing, processing and wireless communication capabilities, which are not intended as add-on components for mobile devices, but as flexible autonomous devices for their own. They consist of two modules, one for sensors and perception and a second one for application-specific processing and communication. With Smart-Its Friends [HMS+ 01], a demonstrator for evaluating the value of context sharing has been built. The underlying idea is to make connections between artifacts based on proximity-relations between their contexts; artifacts are considered to be in context-proximity if they experience similar situations. In this application, Smart-Its devices broadcast their movement pattern acquired from accelerometer data, and receiving devices compare it with their own movement and establish a connection at the application level if they are similar (i.e. if they have been manipulated the same way). Spatial proximity is thus not acquired by a direct relation of positions, but rather by a comparison of movements. Smart-Its Friends is similar to this thesis in that context information is exchanged between artifacts and used for processing qualitative relationships; however, just similarities between contexts – and thus only a small subset of the possible relations – are considered.
32
2 Spatial Awareness
Aura Aim of the Aura project (http://www.cs.cmu.edu/aura) at the Carnegie Mellon University (US) was the creation of distraction-free pervasive computing environments which adapt to the users’ contexts and needs [GSSS02, SG02]. Central are the support of user mobility as well as a transparency of dynamic resource variability by automatically adapting the environment or ongoing tasks to the user’s context. Location information – which is acquired from IEEE 802.11 access points – is a key context in this regard, as for example in the “Portable Help Desk” [GSSS02] application which provides users with an awareness about nearby people and resources. A hybrid location model [JS02] has been developed within the Aura project, which uses a combination of hierarchical and geometric location representations by allowing each space in this hierarchy to define geometric points or areas. A space service based on this model has been implemented, answering SQL queries about distance, containment and intersection relationships, among others.
Easy Living The project Easy Living at Microsoft Research aimed at the development of a prototypical architecture and technologies for intelligent environments in which stationary and mobile devices cooperate to provide user access to information and services [BMK+ 00]. The geometry of the world plays a central role in this project, as knowledge about physical relationships between people, devices, places and things increases the shared understanding between user and system and thus contributes to a more natural and implicit interaction (e.g. the selection of a display which is in the user’s field of view) [BKMS00]. A geometric model has been developed which represents the measured quantitative position and direction of one entity’s reference frame with respect to that of another entity, and optionally represents their physical extensions with polygons. In this model, reference frames of the entities are vertices and geometric measurements between them are edges in an undirected graph, which allows to process relationships between entities – including the intersection of regions (e.g. to select all devices within a particular radius around the user) – which are connected with a path in the graph. The project focus concerning the perception area is on stereo computer vision for distinguishing and localizing people [KHM+ 00]. Devices are networked without a central server using asynchronous message passing. Although the geometric model explicitly represents spatial relationships between extended entities, it is maintained on a central server and not distributed among the entities as in our work. Moreover, no
2.6 Related Work
33
qualitative representation of and reasoning about qualitative spatial relations are addressed. ParcTab The ParcTab system (http://sandbox.xerox.com/parctab) was developed at Xerox PARC for investigating mobile and context-aware computing in office settings, and it pioneered the use of location information for a variety of context-aware applications. ParcTabs are small hand-held devices with a touch-sensitive screen, which are able to communicate with each other and with desktop computers using an infrared (IR)-based cellular network [WSA+ 95]. A room provided with an IR transceiver becomes a cell in this network, which also allows for sensing locations at the granularity of rooms. Infrared transmissions are used for determining location the same way as in the Active Badge system [WHFG92], which is one of the first indoor location sensing systems. It provides symbolic location information about people who wear badges that are equipped with a microprocessor and an IR transmitter. The badges periodically emit unique user-identifiers, which are collected by IR receivers placed in the environment and sent to a central location server. The proximity of ParcTabs as a context for interaction was explored in several application scenarios [SAW94], e.g. in the form of “proximate selection” which is a user interface technique for emphasizing objects that are located nearby. The ParcTab system and applications are related to this thesis in that location and proximity information are used for building context-aware applications. Place Lab Place Lab (http://www.placelab.org) is an open-source software solution developed at Intel Research, which works both indoors and outdoors and allows clients like notebooks, PDAs or mobile phones to estimate their own positions by listening for the identifiers of stationary radio beacons like IEEE 802.11 access points or GSM cell towers and looking up the beacons’ positions in a database [LCC+ 05, SLB+ 03]. According to [LCC+ 05], clients have access to the positions of more than 2 million beacons worldwide (primarily IEEE 802.11 access points) and localize themselves with an accuracy of about 20 (802.11) to 100 (GSM) meters within an urban area. Sentient Computing The idea behind the Sentient Computing project is to give applications the ability to adapt their behavior based on a model of the environment, which allows them
34
2 Spatial Awareness
to act as if they shared the user’s perception [ACH+ 01]. This world model is implemented with a set of software objects containing up-to-date information about the states and locations of real-world objects. To achieve a level of accuracy close to human’s perception, a building-wide ultrasonic location system has been developed; a prototype of this system has been presented in [HHS+ 02]. Each user wears a mobile device (i.e. referred to as bat), which emits ultrasonic pulses to a sensor network mounted on the ceiling and can be localized with an accuracy of up to 3cm. The system is also able to estimate a user’s direction, assuming that the bat is mounted on the body’s front side. Similar to the Zones-of-Influence approach presented in Section 3.1.1, spatial relations between objects are processed in terms of geometric containment and overlaps relationships between 2-D spaces associated with those objects. They have been used for “follow-me” applications, for example to display a user’s desktop on the nearest screen by defining two spatial zones, one from where the machine can be used and another one from where the screen can be seen [HHS+ 02]. In contrast to this thesis, neither relations beyond topology nor spatial reasoning have been considered. NearMe The NearMe Wireless Proximtiy Server [KH04], which has been developed at Microsoft Research, allows to discover people and things in physical proximity using Wi-Fi. Clients register with the server, scan for Wi-Fi access points and report observed access points and their signal strengths to the server, who compares the data with that of other clients. The server can then send back a list of nearby users and things that have been registered. NearMe provides no absolute location but only proximity information with a resolution in the tens of meters; an a-priori calibration of the environment is not required. Implemented applications include the discovery of resources in proximity and sending e-mails to people nearby. NearMe inspired recent work [VPKdL08] for calibration-free localization, which uses relative distances between pairs of mobile phones that are acquired from the measurement of signals from observed GSM cell towers. Imperial College of Science (UK) In [Leo98, LM98], a location service is proposed which collects data from multiple sensors, abstracts the sensor readings to a uniform representation and performs fusion by exploiting overlaps between locations – which are represented in a semi-symbolic (i.e. containing both symbolic and geometric representations) hierarchical model – in order to improve accuracy. Spatial containment, overlaps
2.6 Related Work
35
and equivalence relations can be acquired from the location model. The location service thus provides an abstraction from location sensor details and allows for developing location-aware applications independently of certain sensor technologies, which corresponds to the application-level provision of abstract spatial relations in this thesis.
Location Stack The Location Stack [HBB02] is an abstract layered software engineering model for the use of location in ubiquitous computing applications, which is inspired by the seven-layer OSI model for networking. Similar to [Leo98], it is designed to support multiple sensor technologies and multiple applications with generic abstraction layers in-between. These include a fusion layer for merging readings from multiple location technologies, and an arrangements layer for reasoning about relationships between objects. The Location Stack has been applied to the Labscape project [AGH+ 02] and the Easy Living system [BMK+ 00] in order to illustrate the value of separating concerns with a layered architecture.
GISS At the Johannes Kepler University Linz (AT) we have developed a Group Interaction Support System (GISS) [FHO04a, FHO04b] for the support of interaction within groups in mobile distributed computing environments, for example by visualizing the locations of group-members in an instant messenger or by automatically forming a temporary group with people staying at the same location. The implemented system uses the context-framework presented in [BCFM03], which is based on a peer-to-peer communication architecture and uses Event-ConditionAction (ECA) rules for defining the behavior of the system. A central part of GISS is its location sensing subsystem, which acquires location and proximity information from multiple different sensors, abstracts the spatial sensor readings to standardized symbolic values, performs fusion algorithms on them to increase precision and accuracy, and provides the spatial information in a sensor-independent way at the application level. In contrast to this thesis, the abstraction of proximity relations depends on the used sensors and cannot be adapted to application demands, just proximity relations are considered and no reasoning mechanisms are included in GISS.
36
2 Spatial Awareness
Peer-It In recent work also conducted at the Johannes Kepler University Linz (AT), a software/hardware platform called Peer-it [FHdSR+ 07, FHR+ 08] has been developed which comprises sensors, actuators and wireless communication facilities on the hardware level as well as a component-based framework at the software level. Peer-its are intended to be used as stick-on computer solutions for physical objects such as machines or appliances, thus enabling their autonomous operation and spontaneous interaction with other peers. Interaction among peers is based on their discovery once coming into proximity, an exchange of XML-based “profiles” representing their properties and interests as well as a similarity analysis of such profiles [FHR+ 06a]. It could thus serve as an implementation of the Digital Artifact Service of our Zone-of-Influence Framework described in Section 6.2, which is responsible for discovering artifacts and exchanging self-descriptions – which correspond to the “profiles” in [FHR+ 06a] – with them. The Peer-it system emerged from previous work on spontaneous interaction [FHM+ 04], in which autonomous artifacts exchange and compare their profiles once they come into spatial proximity. Also the concept of Zones-of-Influence, which constitutes the basis for recognizing relations in the present thesis (cf. Section 3.1.1), has originally been introduced in [FHR+ 08] for limiting interaction to peers whose Zones-of-Influence overlap; however, just topological relations and no reasoning mechanisms for inferring conclusions from spatial relations are considered. u-Texture The u-Texture project [KOY+ 05] at the Keio University (JP) addresses the issue of creating smart environments in an intuitive way by physically assembling smart building blocks with embedded computing, sensing and networking capabilities into shapes like tables or shelves. A u-Texture prototype including a touch screen, serial interfaces at the four sides to detect a connection with other u-Textures, a 3D direction sensor as well as both wired and wireless network interfaces has been developed. For assembling u-Textures horizontally or vertically to one another, a connection structure called u-Joint is used in this prototype. Assembled u-Textures have self-organizing capabilities by autonomously adapting their behavior to the current shape and cooperatively running a corresponding application. By exchanging information on their connections and spatial directions with XML-based documents similar to our self-descriptions, each one of the u-Textures recognizes the assembled shape as well as its location and direction in it. One of the realized applications is the “Collaboration Table” which allows for exchanging drawings
2.6 Related Work
37
among connected u-Textures by means of simple drag-and-drop operations, similar to the “Gateways” concept [GSG07] of the Relate project. Relate The Relate project (http://eis.comp.lancs.ac.uk/relate) was funded by the European Commission in the Sixth Framework Programme (FP6) and coordinated by the Lancaster University (UK). Its aim was to investigate relative positioning of mobile objects without a surrounding infrastructure, including research issues such as the measurement of spatial relationships and software platforms for relative positioning systems. A hardware/software system is presented in [HKG+ 05], which is based on an ultrasonic peer-to-peer sensing hardware for distance and orientation measurements. It allows mobile devices to independently establish and maintain a model of fine-grained quantitative and qualitative spatial relationships to other devices in two dimensions. The Relate system architecture consists of several layers, including a processing pipeline for computing quantitative relations, performing abstractions to qualitative relations and inferring information over time. A spatial model maintains a list of directed labelled graphs where each one represents the spatial arrangement of devices at a certain point in time. As an extension of the Relate system, a set of “spatialized widgets” – i.e. visual interaction elements whose appearance and behavior adapts depending on the spatial situation – for incorporating spatial relationships in the user interface is proposed in [KKG05]. [BSHK06] uses the Relate system for constraint-based distance estimation, as well as a graph in which the nodes represent mobile devices and the edges are labelled with distance constraints (i.e. intervals for modelling uncertainty). With transitive inference steps (cf. Section 4.1), more specific distance constraints between devices are inferred. Further work within the Relate project deals with securing wireless connections by spatial relationships [May06] and visualizing “Gateways” to discovered services in a GUI according to their spatial orientation [GSG07]. The architecture of the Relate system is similar to ours insofar as it allows mobile devices to recognize and abstract spatial relations, infer new relations from existing ones and provide applications with access to the spatial model. Within this thesis however, spatial relationships are not directly recognized with sensors like the one presented in [HKG+ 05], but on the basis of explicitly defined spatial context information exchanged among artifacts. A main focus of our work is on the qualitative abstraction of relations, which has not been considered for spatial aspects other than position in the Relate project, and qualitative reasoning using first-order-logic is part of the Relate architecture presented in [HKG+ 05] but not addressed in more detail.
38
2 Spatial Awareness
DOLPHIN DOLPHIN [MFH+ 04] is a distributed 3-D indoor positioning system that uses devices for bidirectional ultrasonic ranging similar to the ones used for the Relate system presented above. The aim of the project is to reduce the high initial configuration costs of the system, which are inherent to other ultrasonic positioning systems [HHS+ 02, PMBT01] in that they require a huge number of precisely positioned references. DOLPHIN tries to overcome this problem by using iterative multilateration, which is based on the idea that sensor nodes estimate their position using a small number of a-priori configured reference nodes, and hence become references by themselves. In contrast to existing ultrasonic positioning systems, just the location of objects and not their direction in space is detected. Cooperative Artifacts In [SGKK04, SKGK04], the instrumentation of everyday physical objects with embedded computing, wireless communication, sensors and actuators is described, which allows them to autonomously perceive and cooperatively assess their situation without the need for an instrumented environment; such instrumented objects are referred to as Cooperative Artifacts. A platform-independent architecture is depicted, which consists of (i) a sensor component for acquiring context information of the real world, (ii) a perception component for associating sensor data with meaning, (iii) a knowledge base containing domain knowledge and observational or inferred knowledge about its situation, (iv) an inference component for reasoning about its own knowledge and those provided by other artifacts, and (v) an actuator layer for effecting the real world. Rule-based reasoning is used for both inferring further based on existing knowledge and for determining the behavior depending on the environmental context, and artifacts cooperate by sharing their knowledge using some kind of query/reply protocol. The concept has been demonstrated with chemical containers [SGKK04], which are augmented with Cooperative Artifacts and are thus able to detect hazardous situations, as for example that incompatible chemicals are stored too close together. The ultrasonic range sensors developed in the Relate project presented above detect distance relations between containers, and infrared light sensors are used for recognizing if they are stored inside or outside an approved area. The authors of [SGKK04] consider their approach novel with regard to embedding generic reasoning capabilities into otherwise non-computational artifacts. The Cooperative Artifacts work overlaps with ours regarding the cooperation between autonomous artifacts by sharing knowledge, the inference of new knowledge and triggering of actions with rule-based
2.6 Related Work
39
reasoning as well as the use of spatial distance relations in [SGKK04] for detecting hazardous situations. The focus of this thesis however is on the spatial awareness of artifacts beyond location and distance estimations, including the abstraction of spatial relations and qualitative reasoning about them over time, which is not covered in [SGKK04]. Ubiquitous Chip The Ubiquitous Chip [TTH+ 04] is a flexible and autonomous prototype hardware/ software platform which can be connected with devices such as sensors, actuators and other ubiquitous chips, and allows for describing its behavior by means of Event-Condition-Action (ECA) rules. Inputs from sensors and packets received via the serial port are evaluated with respect to the rules’ conditions, and the actions of a rule – as for example to control a connected actuator – are executed when all its conditions are fulfilled. Although this rule-based approach is similar to ours in that the system functionality can be programmed easier and changed dynamically at runtime, the rule conditions are formulated on low-level sensor data and not on high-level contexts as in our work. MiddleWhere A distributed middleware called “MiddleWhere” for location-awareness is presented in [RAMC+ 04], which is based on a hierarchically organized hybrid location model using both geometric and symbolic representations, and enables probabilistic fusion of location data from different sensor technologies. A spatial database is used for representing and reasoning about location information and defining triggers that generate events when a certain location-based condition is fulfilled. Among other things, the middleware is able to infer spatial relationships, where three types are distinguished: (i) topological RCC-8 relations [Ege89] and additional relations for representing connectedness between regions, (ii) containment, Euclidean distance and usage (i.e. if a person is within the usage area of an object, similar to [ACH+ 01]) relations between an object and a region, as well as (iii) proximity, co-location and distance relations between two objects. TOTA “Tuples on the Air” (TOTA) [MZ03, MZ04] is a middleware for developing distributed context-aware and self-organizing applications. It is based on spatially distributed tuples, which are injected in the network from artifacts running the TOTA
40
2 Spatial Awareness
middleware, autonomously propagate hop-by-hop according to tuple-specific patterns and can be queried by artifacts within their one-hop neighborhood. The propagation pattern of a tuple is defined by its spatial scope (i.e. distance and spatial direction of propagation), the presence or absence of other tuples and how its content has to be changed during propagation. TOTA tuples can thus be used for creating and exploiting distributed spatial structures of the network, including application examples such as routing in mobile ad-hoc networks, motion coordination and gathering of context information [MZ05, MZL03]. As an example for the latter, an artifact may broadcast query-tuples with the information it is looking for; these tuples create gradients to their source by incrementing a counter at every hop, and receiving artifacts which possess the requested information can inject answer-tuples following the path in the direction where the query-tuples’ gradients are decreasing. With respect to our work, the TOTA middleware could be used for implementing our spatial relationship distribution algorithm of Section 4.3. Gator Tech Smart House The Gator Tech Smart House [HMEZ+ 05] is a laboratory for investigating pervasive computing systems with the aim to create assistive environments specifically for the elderly and disabled people. A multi-layered OSGi-based middleware and numerous applications based on it have been realized, as for example a smart floor for tracking the locations of people as well as ongoing work on a smart projector which uses an ultrasonic tracking system for detecting where people are facing in order to decide on which wall to display the contents. In the spirit of our approach for developing applications based on qualitative spatial relations, the Gator Tech Smart House allows domain-experts to model application behavior by associating actions with abstract high-level contexts. Rutgers University (US) A spatial programming model similar to the TOTA approach [MZ04] is presented in [BIK+ 04], which allows for implementing distributed applications in a networktransparent way. It is based on “smart messages” containing data and code, which migrate to nodes of interest in the network using geographical routing algorithms and are executed there. The core idea is to identify spatial regions which are relevant for a certain application (either static or dynamically defined by a spatial proximity or orientation relation with respect to a certain node), and access network resources through spatial references by specifying their expected spatial regions and properties.
2.6 Related Work
41
Spatial Cognition: Reasoning, Action, Interaction Spatial Cognition: Reasoning, Action, Interaction is an interdisciplinary research center at the Universities of Bremen and Freiburg, which covers various aspects of knowledge about spatial environments (http://www.sfbtr8.uni-bremen.de). It comprises tens of research projects and spans a huge number of topics, ranging from the investigation of human spatial cognition to mobile robot navigation. One of the projects related to this thesis is “Q-Shape”, which deals with reasoning about paths, shapes and configurations. Several qualitative spatial calculi [MDF05, MRW00] have been developed in this project, which are defined by a set of relations between objects from a certain domain and operations on these relations (cf. Section 4). Of particular interest is the SparQ toolbox [DFWW06, WFDW07] which makes existing calculi available for integration in own applications, and it can be run in server mode and accessed via a TCP/IP connection. It provides services for the qualification of quantitative geometric descriptions, performing computations with relations based on the operations defined in a certain spatial calculus and constraint-based reasoning for checking consistency or determining if particular relations between certain objects hold, among other things. Another interesting project is “OntoSpace” which deals with the linguistic aspects of spatial terms [Ten05] and investigates human-robot interaction through dialogue [TM03], among others.
2.6.2 Comparison Table 2.2 compares selected projects of Section 2.6.1 by means of a set of aspects which concern software frameworks and architectures using spatial context information, and in particular spatial relations, at the application level. First, the abstraction of sensor technology – namely a separation from low-level sensor details which are not relevant at the application level – is an important property of context-aware systems (cf. Section 2.1), and local context exchange refers to their ability to share perceived context information in a peer-to-peer manner with other systems in communication range. In this thesis, we distinguish three types of spatial properties describing an artifact’s state, which are its position, direction and physical extension, as well as four types of spatial relations between two artifacts at a time, which are positional (distance and orientation), directional and topological relations. They can be recognized from a single or multiple sensor technologies, and represented quantitatively and/or with qualitative symbols. In this regard, adaptivity describes the ability of a framework to dynamically adapt the consideration of certain qualitative relations as well as their semantics to the
42
2 Spatial Awareness
application demands. The maintenance of a relations history means that relations are stored together with points in time or with time intervals in which they have been recognized or exist, which is a precondition for reasoning about relations over time. Qualitative reasoning is a central issue in this thesis, where our focus is on rule-based reasoning about qualitative spatial relations and about temporal relations between the spatial ones (i.e. spatiotemporal relations). Several observations can be made from Table 2.2. First, in most projects sensor data is abstracted for their use at the application level, which can thus be considered state-of-the-art and also plays an important role in our work with respect to relationship abstractions. Second, most approaches for spatial awareness exploit the position of artifacts, which is actually the most often used context in existing context-aware applications [CK00] and is usually the basis for calculating positional orientation and distance relations. However, proximity can be determined by other means also, as for example in the Smart-Its Friends application [HMS+ 01] where two devices are considered to be in proximity if their movement-patterns are similar. Third, although context sharing is used in several projects for cooperation between autonomously operating artifacts, none of them uses structured XMLbased self-descriptions for the representation of spatial properties and relations. Fourth, directional relations – as for example that two objects are aligned in the same direction – are hardly used in the field of pervasive computing, which may be due to sensing difficulties. Fifth, qualitative representations of spatial relations are utilized by almost all projects compared in Table 2.2; in most cases however, just proximity or containment relations are used. The contributions of this thesis, which distinguish it from the related work discussed above, include (i) the adaptive recognition and qualitative abstraction of relations from the explicitly defined spatial properties position, direction and physical extension, (ii) a standardized representation for the exchange of spatial contexts between autonomous entities and (iii) the proposed approach for rule-based reasoning about qualitative spatial and temporal relations. This together facilitates the development of spatially aware applications in that it allows for simply specifying qualitative abstractions of spatial relations which are relevant for the application, and defining actions to be executed once a certain pattern of spatial relations over time has been recognized.
2.6 Related Work
43
a determined
xa x
x
x x x
x x
x x
x
x x x
x
x x
x x
x
x x
x
x
x x
x
x x x
xa x
x x
x
x
x
x
xe
x
xe
x
x x x
x
x x
by the sensor range which defines a geographical area ad-hoc exchange of structured profiles with discovered peers in range c via a wired backbone infrastructure d supported by the framework, but not readily available at the time of this writing e implicitly by initiating a connection upon recognizing the respective qualitative relation f given by tuple gradients of neighboring nodes g depends on the plugged-in relation recognizers b wireless
x xd
x
x x x
x xf
x x x x
x
x x
x x x
x
x
x x
x
ZoI Framework
x x
x xc
TOTA
x x x
x xb
MiddleWhere
Sentient Computing
Relate
Easy Living
x
u-Texture
x
x x
Smart-Its Friends
x
x
Mediacup
x x
Peer-it
Software/architecture sensor abstraction local context exchange Spatial properties position direction extension Spatial relations positional (distance) positional (orientation) directional topological Relations recognition multiple sensors quantitative represent. qualitative represent. runtime adaptivity relations history Qualitative reasoning rule-based inference spatial relations spatiotemp. relations
ParcTab
Cooperative Artifacts
Table 2.2: Comparison of related work.
x
x xg xg x x x x x
3 Representation of Space Simple things should be simple. Complex things should be possible. Alan Kay, 1940-
In order to deal with space computationally, it has to be represented in a standardized way such that it can be processed by a computer. In this regard, we distinguish between the quantitative and qualitative representation of space, which are discussed in detail in the Sections 3.1 and 3.2, respectively. While the former deals with concrete facts about the spatial properties of an artifact using numerical values (as provided by sensors for example), the latter is concerned with symbolic and human-readable abstractions of spatial relations (i.e. symbols are used for the representation rather than numeric values) which are acquired by a pairwise comparison of quantitative spatial properties. For the representation of an artifact’s spatial properties we propose to use so-called Zones-of-Influence, which are explicitly defined geographical regions that are relevant for the application. Our focus is on qualitative spatial relationships as well as their changes over time and the application-dependent semantics. We consider qualitative relationship abstractions to be particularly valuable for implementing services that are distributed among multiple artifacts in physical space. Quantitative spatial properties and qualitative spatial relations – which together constitute an artifact’s spatial context – are exchanged between artifacts in range by means of XML-based self-descriptions, which enables them to reason about both self-determined and received relations; an example structure of self-descriptions, which we used for evaluation purposes in this thesis, is presented in Section 3.3. Section 3.4 eventually sums up findings concerning the quantitative and qualitative representation of spatial properties and relations with regard to their representation and exchange by autonomous embedded systems.
46
3 Representation of Space
3.1 Quantitative Representation 3.1.1 Spatial Abstraction with Zones-of-Influence For providing spatial awareness to digital artifacts, we propose a concept referred to as Zones-of-Influence (ZoI). It builds on initial work published in [FHR+ 08], where ZoIs are three-dimensional spatial regions associated with digital artifacts; in this work, they serve as an explicit proximity model in that they are used for limiting the interaction between artifacts to those whose ZoI-geometries overlap. In contrast to proximity sensors like Bluetooth signal-strength measurements of nearby artifacts as used in [FHM+ 04, FHO04a] for example, a Zone-of-Influence provides added value by allowing to explicitly define the “proximity range” by its shape, the size of the shape as well as its position and direction in space relative to that of the artifact for which it is defined. A similar concept has also been used in [BMK+ 00, HB00, HHS+ 02], where geometric containment relations between regions of interest that are associated with physical objects (e.g. a person and the area in front of a screen) are used for location-aware applications (e.g. to display the person’s desktop on that screen). In [FHR+ 08], ZoIs are specified with XMLbased documents similar to the self-descriptions used in this thesis and exchanged among artifacts, which enables them to decide on the initiation of an interaction with other artifacts in communication range based on a comparison of their own with received Zones-of-Influence. Applications for this Zones-of-Influence concept are manifold, as it becomes useful whenever digital artifacts, namely technology-enriched objects which are situated in physical space, have to interact with one another and their spatial context has an influence in this regard. For example, ZoIs can be used for modelling dangerous zones of a machine (e.g. its area of operation in which goods are processed) or the physical extension of a parking place, in order to detect if workers are in a dangerous situation (i.e. within the machine’s operational area) or if the parking place is occupied (i.e. a car is within it), respectively [FHR+ 08]. Another automotive example taken from [FHR+ 08] is the representation of the car’s braking distance by means of a dynamically changing ZoI, which is a simple bounding box around the car whose length in the car’s front corresponds to its velocity and thus to the respective braking distance; once an overlap between the ZoIs is recognized, the involved drivers are warned that the space between them is too small (cf. Figure 3.1). The work conducted within this thesis builds on the above Zones-of-Influence concept as the basic building block for determining spatial relations between digital artifacts. In contrast to [FHR+ 08] however, Zones-of-Influence are not utilized
3.1 Quantitative Representation
47
Figure 3.1: Zone-of-Influence for a car’s braking distance [FHR+ 08].
for the mere purpose of initializing the interaction among artifacts whose ZoIs overlap; instead, different types of static and dynamic spatial relations are recognized and provided for the use by arbitrary spatially aware applications. We define a Zone-of-Influence as an explicitly defined spatial region, which is given by its extension as well as its position and direction in space, and represents spatial properties of an artifact that are relevant for its application-level behavior with respect to the spatial properties of other artifacts. Hence, Zones-of-Influence are spatial two- or three-dimensional shapes of a certain size which are positioned and directed in physical space, and they encode spatial properties of artifacts that are relevant for and have an influence at those applications in which the artifacts are involved. In the following, we use the term shape synonymously for the spatial extension of a ZoI, namely including also its size. Special types of Zones-of-Influence are physical ZoIs representing an artifact’s natural physical position, direction and extension, as well point-shaped ZoIs, which are point abstractions for zones whose spatial extension is not relevant for the application or cannot be considered due to limited computational power. As stated in Section 2.4, our focus is on the quantitative (i.e. numerical) representation of spatial properties with respect to an absolute (i.e. fixed, external) frame of reference. In particular, WGS84 coordinates for representing absolute position and Euler angles for absolute direction can be used, which are explained in Section 3.1.2. Position and direction of a Zone-of-Influence are related to a single point, to which we refer to as the ZoI’s anchor point; they are visualized as black dots in Figure 3.1. Moreover, just static spatial properties are used for representing a Zone-of-Influence, meaning that dynamic contexts which characterize the degree of change at a point in time (e.g. acceleration) are not considered in this thesis –
48
3 Representation of Space
Figure 3.2: Types of Zones-of-Influence in 2-D.
although in principle supported by the Zones-of-Influence Framework presented in Chapter 6. Each digital artifact is associated with a single or multiple Zones-of-Influence at a time, together representing the spatial application-relevant knowledge which is distributed across them in physical space. A ZoI can be dynamic, which means that its spatial properties depend on sensor values (e.g. if its position and direction depends on that of a digital artifact), or static for ZoIs whose position, direction and shape do not change over time. Moreover, a single Zone-of-Influence can be fragmented, namely split in multiple – relatively positioned and directed – shapes, which is meaningful for representing ZoIs with complex or disconnected spatial extensions. The following list sums up these four types of ZoIs, which have been distinguished in [FHR+ 08] already and are clarified by means of two-dimensional examples in Figure 3.2: • Static: does not change over time (i.e. no dependence on sensor values) • Dynamic: one or more spatial properties change over time • Multiple: more than one ZoI is assigned to the same artifact • Fragmented: a single ZoI is split in multiple shapes Due to the exchange of Zones-of-Influence by means of self-descriptions (cf. Section 3.3), they can be discovered by artifacts in communication range, which are thus able to determine spatial relationships between their own and received Zones-of-Influence – including positional, directional and topological relations. Therewith, application-level behavior of artifacts can not only be determined by
3.1 Quantitative Representation
49
proximity relations to other artifacts as in most projects compared in Section 2.6, but rather by arbitrary spatial relations among them that are acquired by a pairwise comparison of Zones-of-Influence, which constitutes the core of this thesis.
3.1.2 Reference Systems for Position and Direction The spatial properties of a Zone-of-Influence are either static or dynamic in that they depend on sensor values which change over time. Two spatial properties of a digital artifact, which are acquired from respective sensors, are considered in this work: position and direction. The shape of a ZoI is assumed to be statically defined and independent of any sensor values for now. We consider positionand direction-sensors that provide quantitative readings given for two- or threedimensional space, and with respect to an absolute reference system. Knowledge about the used reference system is required for a common understanding of artifacts about exchanged – static or dynamic – Zones-of-Influence, which in turn is a precondition for recognizing spatial relations by their pairwise comparison. In the case that digital artifacts have to deal with different reference systems at the same time, e.g. if two location systems – one for outdoor and another one for indoor – are used together, there must be means for transforming to a common reference system (cf. [JS02]). In the following, reference systems for position and direction are surveyed in short. For the property position, numerous sensing techniques and technologies with different reference systems have been developed (cf. [HB01]). An indoor location system may provide three-dimensional Cartesian coordinates relative to a certain point in the building for example, and a GPS receiver provides spherical polar coordinates in a global reference system; a comprehensive overview of coordinate systems for georeferencing can be found in [Dan99]. The most common global system, which is also used by GPS, is the World Geodetic System 1984 (WGS84) [IoD00]. In this system, points are typically represented as spherical polar coordinates in latitude ϕ (i.e. the angle between any point on the surface of the earth and the plane of the equator), longitude λ (i.e. the angle between the prime meridian and a plane through a point on the earth’s surface, where both planes are perpendicular to the equator) and altitude h (i.e. the height of a point’s location above a reference ellipsoid which approximates the earth’s surface). Also, they can be represented as Cartesian coordinates (x,y,z) using Earth-Centered (i.e. the origin is the mass center of the earth) and Earth-Fixed (i.e. coordinates of a point on the earth’s surface do not change with its rotation) coordinates. In such a Cartesian ECEF system, positional relations between two points on the surface of the earth require special treatment; for example, the shortest Euclidean distance be-
50
3 Representation of Space
Figure 3.3: WGS84 spherical (ϕ,λ ,h) and Cartesian (x,y,z) coordinates.
tween them differs from the actual distance on the surface. A visualization of the WGS84 spherical and Cartesian coordinates can be seen in Figure 3.3. There are also different reference systems for the spatial property direction. As mentioned in Section 2.4, we do not use the term orientation in order not to confuse the spatial property direction with spatial orientation relations. A common direction system is the North-East-Down (NED) system defined by the x-axis pointing to the north, the y-axis pointing to the east and the z-axis heading toward the center of the earth. One way of representing directions in three-dimensional Euclidean space is the use of Euler angles, which represent spatial directions with a sequence of rotations around the three axes by the following values: yaw (i.e. a rotation around the z-axis), pitch (i.e. a rotation around the y-axis) and roll (i.e. a rotation around the x-axis). An important point is that the order of rotations matters, and different sequences lead to different resulting directions. For evaluation purposes, the IS-900 6 degrees of freedom (DOF) motion tracking system from Intersense Inc. [Int08] has been used within this thesis (cf. Chapters 5-7). It has a local reference system with the z-axis pointing to the earth’s center and the x- and y-axis pointing in two orthogonal horizontal directions according to the boundaries of the room in which the tracking system is installed. Direction is provided with Euler angles, where the sequence starts with a rotation about yaw (around the z-axis), followed by pitch (around the y-axis) and roll (around the x-axis). In this regard, we distinguish between two-dimensional (2-D)
3.2 Qualitative Representation
51
representations, as well as three-dimensional (3-D) ones where the z-axis – and thus both pitch and roll – are disregarded.
3.2 Qualitative Representation Zones-of-Influence as discussed in the previous section provide means for representing spatial properties of digital artifacts, and they are shared among them using XML-based self-descriptions. As stated before already, our main interest is on spatial relations acquired from a comparison of Zones-of-Influence as well as their qualitative abstraction for the application-level use; however, until now we have not explained what qualitative exactly means and why our focus is on qualitative spatial relations. These points will be clarified in Section 3.2.1, and more details about how spatial relations can be represented in a qualitative way as well as the importance of reference frames in this regard will be given in the subsequent sections. Section 3.2.5 afterwards introduces a combined consideration of qualitative spatial and temporal relations, which allows for qualitative reasoning about spatial relations over time.
3.2.1 Qualitative Abstractions of Space In order to process spatial relations computationally, the mathematics of Euclidean space probably comes to mind first, which is actually the most precise system for spatial representation [HN02]. For example, if we consider the two artifacts shown in Figure 2.4, it could be determined that artifact B is located +93,2° with regard to the direction axis of A, and in a distance of e.g. 3, 52m. Although these quantitative orientation and distance relations are correct and precise, the qualitative information that B is right and near could be sufficiently precise and more appropriate for a particular application – especially if the spatial cognition of humans is involved [CH01, Mus00]. It can be seen from these examples that qualitative abstraction is about naming quantitative values, which means to represent continuous properties of the world by discrete systems of symbols [CH01]. According to [CH01, HN02], the actual quantization, namely the mapping of continuous to a finite number of discrete values, must be relevant with regard to their intended use. This means that only as many distinctions as necessary are made and thus details not relevant for a certain application are hidden, which is a key motivation for using spatial abstractions. With regard to our work, both the types of spatial relations (e.g. static topological and distance relations as classified in Section 2.4) and the semantics of their qual-
52
3 Representation of Space
itative abstraction – including the represented dimensionality of physical space, the granularity of abstraction, the naming of the relations and the used reference system (cf. [FR93]) – depend on the respective application domain, which has to be taken into account for the design of the middleware framework presented in Chapter 6. As stated above, qualitative abstractions are often the preferred choice when the spatial cognition of humans is involved, as for example in human computer interaction. Reasons for that are that humans are used to think in coarse abstractions and draw conclusions from them [For97, Fre92b, Mus00], and that qualitative approaches are similar to natural language terms and thus considered cognitively more adequate [RM04, Ten05]. With regard to this thesis, it should be easier for a programmer to develop spatially aware applications if he is able to use high-level qualitative spatial relations such as right and near, whose semantics are adapted to the respective application domain, rather than having to deal with quantitative distances and angles for example. A related disadvantage however is that applicationspecific qualitative abstractions of quantitative data constitute an additional layer of indirection which needs to be constructed manually (cf. [Leo98]). Further advantages of qualitative representations are that they allow for dealing with coarse and imprecise information [CFH97, HN02] and that reasoning about qualitative knowledge can be computationally less expensive than processing complex quantitative data, which is due to the separation from unnecessary details that are not relevant for a certain application or in a particular context [CFH97, FR93, GGH06]. These two aspects are important with regard to resource-constrained embedded systems and whenever precise sensor information is not available or simply not required by the application. A considerable amount of research is dedicated to qualitative spatial representation and reasoning, which is mainly concerned with (i) abstracting continuous spatial relations of the physical world and (ii) inferring knowledge from the respective qualitative representations. Application areas include robot navigation [BF04, MF00, MSER99], geographic information systems [AEG94, Sha96], highlevel computer vision [Aie02, CH01] and recently also context-aware applications in the field of pervasive computing [BSHK06, KKG05]; our work is related to the latter. Although the present thesis covers many aspects of qualitative spatial representation and reasoning, that is as far as they are relevant to facilitate the development of spatially aware applications for autonomous embedded systems, just a few of them are discussed in detail; comprehensive surveys with further and advanced information are given in [CH01, FR93, Her94, HN02]. Cognitive and linguistic aspects are not addressed in this thesis, detailed discussions about that can be found in [CBGG97, Her94, Ten05, TM03].
3.2 Qualitative Representation
53
3.2.2 Recognition and Representation of Spatial Relations In the following, we will describe how different types of relations can be recognized by comparing Zones-of-Influence and represented in a qualitative way. This builds up the basis for qualitative spatial reasoning, which is covered in the subsequent Chapters 4 and 5. The recognition of spatial relationships is based on a pairwise comparison of Zones-of-Influence, namely their quantitative spatial properties position, direction and shape, which are exchanged between digital artifacts using XML-based self-descriptions; the exchange of self-descriptions has been discussed in Section 2.2.1 already. We consider just sensors with absolute quantitative spatial information, others which directly provide spatial relations are out of scope. In general, we are convinced that it is much easier for an artifact to determine its own absolute spatial properties and share them with other artifacts for the purpose of relationship recognition, for which a multitude of location systems [HB01] and wireless communication technologies are available, rather than directly acquiring relations to surrounding artifacts without using their spatial properties. Although the latter would allow for an autonomous solution without the need for an external infrastructure, only few technological solutions like the ultrasonic peer-to-peer sensor presented in [HKG+ 05] or the infrared beacons of Polulu Robotics and Electronics [Pol08] are available by now, and they suffer from multiple severe constraints including the required line-of-sight, the limited sensor range and the comparatively low accuracy of the sensor readings. As soon as an artifact receives a self-description, it compares the Zones-ofInfluence of the transmitting artifact which are contained therein with its own ZoIs, and calculates binary relations from them. The latter artifact is commonly referred to as primary object in literature [CH01], the former one – i.e. the artifact which recognizes the spatial relation – is referred to as reference object. As we relate Zones-of-Influence and an artifact may be associated with more than one ZoI at a time (cf. Section 3.1.1), we refer to them as primary ZoI and reference ZoI, respectively. The scope of our work is on binary relations only (i.e. relations between two ZoIs), unary relations (e.g. if and how fast a ZoI is moving) and ternary relations (e.g. that a ZoI is located between two other ZoIs) are not considered. A binary relation is often denoted as R(x, y) and it is read as x is in relation R to y, where x and y denote the primary and reference ZoI; more details on binary relations and their properties will be given in Section 4.1. From an architectural viewpoint, it depends both on the application demands and on the availability of the required spatial information in the compared self-descriptions which relations are being recognized.
54
3 Representation of Space
Figure 3.4: Recognition of spatial relations by comparing self-descriptions.
The relationship recognition is shown in Figure 3.4 by means of a simple example, where the quantitative distance between the physical Zones-of-Influence (i.e. their natural physical position, direction an extension) of two artifacts A and B is determined. In this example, each one of the artifacts is aware of its 2-D Cartesian coordinates from an indoor tracking system, that is (xA , yA ) = (2m, 3m) and (xB , yB ) = (5m, 7m), encodes this information in the description of a ZoI which is exchanged with the respective other artifact using a structured self-description, and eventually calculates the quantitative Euclidean distance by comparing the ZoI-description of its local with that contained in the received self-description (i.e. d = 5m). Detailed information about the structure of self-descriptions will be given in Section 3.3. The recognition of quantitative relations is often a first step before performing a qualitative abstraction, and in this respect especially considered in the Relations Service of the Zones-of-Influence Framework (cf. Section 6.2.3). For relations between physical ZoIs, we often refer to as relations between the respective artifacts. Before having a closer look at different types of qualitative spatial relations that can be recognized by comparing two Zones-of-Influence at a time, there are some issues which have to be considered when representing spatial relations in a qualitative way. According to [FR93], these include (among others): • the dimensionality of the represented space (2-D or 3-D) • the types of relations to be represented (e.g. distance or orientation changes) • the spatial primitives among which relations are defined (e.g. points or intervals)
3.2 Qualitative Representation
55
Figure 3.5: Orientation relations of nearby extended objects, after [Her94].
• the granularity and semantics of the qualitative representation (i.e. how is the space partitioned and which relations are represented) • the reference frame with respect to which a relation is defined (cf. Section 3.2.4) First, the dimensionality of space is determined mainly by application requirements or sensor limitations; in this thesis, 2-D projections of the physical space are used, which are sufficient for many application scenarios and conceptually similar to 3-D. For the representation of spatial relations, it is necessary to decide on a certain kind of spatial primitives first among which they are defined. With regard to positional and directional relations, mainly points [Fre92b, Mor04] and line segments [AEG94, MRW00] have been used in literature, the representation of extended objects for topological relations is discussed in [CBGG97, RCC92]. Similarly, we use spatial extensions for topological relations, as well as points for reasoning about positional relations between Zones-of-Influence (i.e. their anchor points) in Chapter 4, which have shown to be suitable abstractions for positional relations between distant objects during evaluation (cf. Chapter 7). However, once objects are placed in spatial proximity and do not have a uniform spatial extension, points may no longer be appropriate. Instead, it will be necessary to incorporate also the objects’ spatial extensions in order to recognize meaningful relations; respective approaches for orientation and distance relations can be found in [Her94] and [CFH97], respectively. Besides topological relations, for which of course spatial extensions are required, we also used distance relations between the anchor points and shapes of ZoIs for the evaluation scenario presented in Section 7.2.1. The problem of extended objects is exemplarily shown with orientation relations in Figure 3.5. In both cases, the primary object p is considered to be in the relation right to the reference object r if just their center points are used, although the relation front would be more appropriate in the left case. Possible solutions for dealing with orientation relations between rectangular shaped objects are presented in [Her94], which are based on an adaptation of the partitions of space in which
56
3 Representation of Space
Figure 3.6: Definition of static qualitative spatial relations in 2-D.
certain orientation relations hold – i.e. illustrated with equally sized sectors for the relations front and right in Figure 3.5 – to the shape of the reference object.
3.2.3 Static and Dynamic Spatial Relations As stated in Section 2.4, we distinguish between static and dynamic qualitative spatial relations; in this subsection, an overview and a respective classification of spatial relations will be given, which has been published in [Hol07a, Hol07b] already. With respect to the qualitative spatial reasoning literature we use the term object in the following, meaning Zones-of-Influence that are associated with digital artifacts. First, qualitative static spatial relations represent spatial relations between objects at certain points in time. We distinguish four types which are shown in Figure 3.6 by means of an exemplary spatial configuration, whereas two objects p (the primary object) and r (the reference object) are placed in two-dimensional Euclidean space; however, the relations can be defined correspondingly in three dimensions. In this figure, arrows represent the objects’ direction axes, and the resulting qualitative spatial relations for the given configuration are written boldface. For topological relations, we distinguish five possible relations between regions as defined in the RCC-5 Region Connection Calculus [CBGG97]; these are disjoint, overlaps, inside, contains and equal. A more precise qualitative representation is provided with the RCC-8 relations [CBGG97], which cover three additional cases where the regions have contact at their boundaries but do not overlap (cf. Figure 3.6); for real-world scenarios involving physical sensors however, the detection of touching boundaries has shown to be problematic due to their limited precision and accuracy. As tests for spatial relations between complex shaped regions can be
3.2 Qualitative Representation
57
Figure 3.7: Cone-/projection-based orientation and qualitative distance [CFH97, RM04].
computationally expensive (cf. [Leo98]), it should be noted that topological relations may not be feasible for the use with resource-constrained embedded systems. For orientation and directional relations, a cone-based qualitative representation which partitions the 360° range in four equally sized sectors is used, where each one of the respective regions is associated with a certain relation. In this regard, orientation relations describe where the primary object is placed relative to the reference object [CH01, Her94]; therefore, the space around the reference object is partitioned, and the relation is denoted by the region in which the primary object is located. Similarly, directional relations relate the direction of the primary object (as given by its intrinsic direction axis) with that of the reference object, such that the space around the primary object is partitioned according to the direction axis of the reference object, and the region in which the direction axis of the primary object points denotes the directional relation. The most common representation systems are cone- and projection-based [RM04] (cf. Figure 3.7), where it again depends on the application domain which one is used. Concerning applications for autonomous embedded systems, a cone-based representation seems to be most suitable as it easily allows to change the granularity by adding or removing axes and thus allows to cope with sensors of different accuracy. Moreover, sectors of different size can be defined (cf. [Mus99, Mus00] in order to take into account the domain-specific semantics of relations. Distance relations are a measure for the effort which is needed to go from a certain primary to the reference object [HCF95]. In this regard, our focus is on the Euclidean spatial distance; other measures like the time of travel are out of scope. We consider distance relations which partition the space around the reference object r in circular ranges, and assume an isotropic space in which points at the same distance are connected with concentric circles. Each one of the qualitative distances conforms to an interval of quantitative ones [CFH97], defining the qualitative relation between the reference and the primary object; the num-
58
3 Representation of Space
Figure 3.8: Definition of dynamic qualitative spatial relations in 2-D.
ber of intervals again determines the granularity of the relations. The distance relations shown in Figure 3.6 partition the space around the reference object r in circular ranges of the same size, except the outer range which is open; in contrast, [CFH97] defines the range of an outer distance relation as bigger than that of an inner one. For the example of Figure 3.4, the resulting relation of B to A would be medium-dist if the distance ranges have a size of e.g. 4m. Figure 3.7 shows common qualitative representations of orientation and distance relations, which partition the space in the eight cardinal directions north, east, south, west, northeast, south-east, south-west and north-west, and in the five distance levels very close, close, commensurate, far and very far, respectively [CFH97, RM04]. Correspondingly, four types of qualitative dynamic spatial relations shown in Figure 3.8 represent how an object’s spatial relation is changing in terms of metric values at a certain point in time. However, such changes need not result in changes of the corresponding static spatial relations; for example, two objects may be moving away from each other at a certain point in time, but the static spatial relation near may still exist. Dynamic topological relations represent a change in the scale of the primary with respect to the reference object, as for example that it is growing. For dynamic orientation relations, the space is partitioned in the same way as for static ones, but around the primary object; the region in which the thick arrow points denotes the dynamic relation. As opposed to dynamic orientation, dynamic distance and directional relations are one-dimensional properties which can just change through an increase or decrease of the metric distance (moving away or toward) and the angular difference of their direction axes (rotating clockwise or counterclockwise), respectively. From an implementation viewpoint, dynamic spatial relations are estimated by comparing the corresponding quantitative static spatial properties at two or more successive points in time. Additionally, there must
3.2 Qualitative Representation
59
Figure 3.9: Epsilon-neighborhood of a static qualitative orientation relation.
be means for representing the absence of relative motion, for which four further relations – stable-size (for topology), stable-orient (for orientation), stable-dist (for distance) and stable-dir (for direction) – are introduced; in this regard, it should be noted that just relative changes between the primary object and the reference object are represented, for which reason stable dynamic relations do not necessarily mean that there is no motion at all. The relations shown in the Figures 3.6 and 3.8 are so-called jointly exhaustive and pairwise disjoint (JEPD) relations, which means that the relations together cover all possible spatial situations and do not overlap (i.e. just one relation holds at a time); qualitative spatial reasoning (cf. Chapter 4) is commonly realized over such JEPD relations [WFDW07]. An important issue in real-world scenarios concerns the recognition of spatial relations from sensor data, which usually fluctuate although the sensed condition does not change; for example, the Cartesian coordinates reported by a location tracking system typically change within its specified accuracy. Such unstable sensor data may lead to frequent relation changes at the boundaries of two qualitative ranges, where e.g. a small change in the relative position of two objects may cause different qualitative distance or orientation relations. In order to avoid such fluctuations, the quantitative range of the currently existing static relation between two objects can be extended by a certain epsilon value as shown with the relation right in Figure 3.9, and dynamic relations are considered stable unless the quantitative change between two successive sensor values is significantly high. An alternative would be to use fuzzy relations as discussed e.g. in [CFH97, Dut88, Mus00], which is, however, not considered in this thesis.
60
3 Representation of Space
Figure 3.10: Intrinsic and extrinsic orientation relations.
3.2.4 Frames of Reference Another issue related to qualitative spatial representations are frames of reference, which influence the semantics of spatial relationships in that the same spatial situation can be expressed in different ways depending on the used reference frame. For example, the relation that a person p stands in front of another person r can have completely different meanings depending on whether it is related to the viewing direction of r, the shape of the room in which they are situated or the perspective of a third person. As for the spatial properties position and direction, which are given according to a certain reference system (cf. Section 3.1.2), spatial relations have to be represented with respect to a reference frame in order to ensure a common understanding across artifacts and their applications. Several classifications of reference frames can be found in literature, which mostly concern the representation of orientation relations [Mus00]. We adopt the classification given in [AEG94, CFH97], where orientation relations of a primary object p with respect to a reference object r are determined by one of the following three reference frames: • Intrinsic: the orientation is given by inherent properties of the reference object (e.g. the reference object’s front side is given by its axis of direction) • Extrinsic: the orientation is determined by external factors (e.g. the North pole or the object’s motion serves as an external reference direction) • Deictic: the orientation is represented from an external viewpoint (e.g. the viewpoint of an external observer from which the reference object is seen) The distinction between intrinsic and extrinsic reference frames for orientation relations is visualized in Figure 3.10. This classification has also been applied to
3.2 Qualitative Representation
61
distance relations in [CFH97, HCF95], whose scale can analogously be defined with respect to an intrinsic (e.g. depending on the size and shape of the reference object), extrinsic (determined by external factors like the travelling time) or deictic reference frame (i.e. the observer’s relative position influences the perceived distance). Another classification is given in [Kla98, Mus99], where (i) egocentric (i.e. relations between objects are represented with respect to the perspective of an observer, either an external observer in the deictic or the reference object in the intrinsic case) and (ii) allocentric frames of reference (i.e. relations between objects are given with respect to a fixed external coordinate system as in the extrinsic reference frame) are distinguished. Furthermore, [Lev96, Ten05] distinguish between intrinsic, relative and absolute reference frames, where the latter two are comparable to deictic and extrinsic reference frames, respectively. Our focus is on intrinsic and extrinsic frames of reference, as we only consider binary relations between the Zones-of-Influence of two artifacts at a time; relations between artifacts with respect to the viewpoint of a third one, as it is the case with a deictic reference frame [Ten05], are not considered. The reason for this is that our focus is on artifacts that are able to autonomously assess their own spatial situation with respect to their environment, namely by comparing local with received Zones-of-Influences as proposed in Section 3.2.2, for which reason deictic relations between surrounding artifacts are out of scope. As different reference frames are mainly relevant for orientation we consider extrinsic distance relations only, meaning that they are defined by the Euclidean distance between two artifacts – more precisely, between the anchor points or shapes of two Zones-of-Influence at a time.
3.2.5 Spatiotemporal Relations As mentioned in Section 2.4, we also consider time series of spatial relations in addition to their representation at certain points in time, which is simply due to the fact that actions take place in both space and time. Although it is obviously valuable to consider space and time in a combined fashion, there is only little existing work with regard to qualitative representation and reasoning. An approach for qualitative spatiotemporal reasoning about motion sequences is presented in [ZF96] for example, whereas more accurate inference results are derived by interpreting static representations as motions. A representation of the course of motion in the context of robotics using qualitative motion vectors is proposed in [MSE+ 98, Mus99], and [GGH06] qualitatively distinguishes movement patterns in order to recognize activities in a smart home setting. However, to the best of our knowledge, none of the existing approaches addresses the issue of application-
62
3 Representation of Space
specific combinations of spatial and temporal relations to high-level spatiotemporal relations. The consideration of time series of relations allows artifacts to recognize not only the current spatial relations to their surroundings, but also how they are changing over time like for example that another artifact is ”passing by on the right hand side”. Such a single spatiotemporal relation results from a logical and temporal combination of spatial relations, which distinguishes ours from existing work and will be explained in the following. In this regard, our focus is again on qualitative abstractions, which are a basis for qualitative reasoning about spatial relations over time; this section however deals just with the representation of spatiotemporal relations. In Chapter 5, a rule-based approach for inferring new spatial or spatiotemporal relations and triggering appropriate actions from the history of known relations will be presented, which has been published in [Hol07b] and constitutes a core component of the Zones-of-Influence Framework presented in Chapter 6. For representing and reasoning about qualitative spatial relationships among artifacts over time, a representation for the dimension time is required. A discrete time model is used for it, which is motivated by the fact that processing in computing systems is based on discrete clock cycles [Mus00], and thus also the recognition of relations – for which sensor readings and spatial data contained in received self-descriptions are used – happens at discrete points in time, either event-driven or at a fixed rate. Moreover, similar to spatial relations which are not always required with a high precision, it may be meaningful for an application that relations which occur “almost simultaneously” are assigned the same points in time – again depending on the application domain. As we are just interested in temporal order relations, we use a discrete time T with whole-numbered points ti ∈ T , and assume the time difference Δt between two consecutive points in time to be constant (i.e. ti+1 = ti + Δt). In the Zones-of-Influence Framework, discrete time is determined by an update thread with an application-dependent temporal resolution, which will be explained in detail in Section 6.2.3. This time model allows for associating spatial relations with discrete points in time, which is the basis for representing and reasoning about spatiotemporal relations. As with spatial relations, which are defined on spatial primitives like points or line segments (cf. Section 3.2.2), temporal relations have to be defined between certain temporal primitives. We use time intervals on the discrete time T in this thesis, where an interval [s, e] is defined by an ordered pair of a start-point s and an end-point e, and it equals a set of successive points in time: [s, e] = {t ∈ T |s ≤ t ≤ e}
(3.1)
3.2 Qualitative Representation
63
Figure 3.11: Qualitative temporal interval relations, cf. [All83].
We agree with the argumentations in [All83, For97, Fre92a], that the use of intervals corresponds to our intuitive notion of time, and – similar to qualitative spatial relations – intervals allow for qualitative abstractions of temporal relations such as “a occurred before b” or “a happened during b”, which is helpful whenever exact temporal relationships are unavailable or simply not of relevance. Spatiotemporal relations in this thesis are based on the qualitative temporal interval relations proposed in [All83], with the difference that we do not use a continuous but a discrete time. Alternative approaches which use time points rather than intervals can be found in [Fre92a, VK86]. According to Allen’s interval algebra [All83], 13 qualitative temporal relations between two intervals can be distinguished at a time, like for example before, equals and during. Figure 3.11 illustrates these relations between the intervals a and b with pictographic examples, whereas the markings on the time axes represent the discrete points in time t0 , . . .tn−1 at which events like e.g. the recognition of spatial relationships may occur, and exemplary time intervals a and b in which certain relations exist are shown with rectangles encompassing consecutive sequences of discrete points in time. Each line in the figure contains a circle representing the earliest discrete point in time at which the respective interval relation can be detected. As we have proposed in [Hol07b], time intervals in which certain spatial relations exist can be temporally related with such interval relations, like for example that near(y, x)[t,t+5Δt] overlaps with the relation le f t(y, x)[t+2Δt,t+11Δt] , where x and y denote Zones-of-Influence of artifacts and the subscript intervals
64
3 Representation of Space
represent the times in which the respective relations between them exist. Thus, the consideration of such temporal relations between time intervals in which spatial relations exist allows digital artifacts to adapt to spatiotemporal situations, which are characterized by corresponding spatial changes over time. As we do not consider a continuous but a discrete time, the semantics of the relationship definitions given in [All83] are different for the relations before, after, meets and met-by; with regard to Figure 3.11, before(a,b) and after(b,a) are defined with ea < sb in the continuous and ea + Δt < sb in the discrete case, and the relations meets(a,b) and met-by(b,a) are defined with ea = sb in the discrete and ea + Δt = sb in the continuous case (cf. [Köh03]). With the above definition of intervals it is also possible to represent points as intervals with the start- and endpoint being the same. Clearly, if points are related with other points or intervals, some of the qualitative relations shown in Figure 3.11 are not meaningful; these are overlaps, during, starts and finishes as well as their inverse counterparts. As stated above, time intervals in which certain relations exist can be related with qualitative temporal relations. To clarify this idea, Figure 3.12 shows exemplary time series of spatial distance and orientation relations between two artifacts x and y, which are illustrated as points and change over time due to changing spatial properties from which they are calculated. Some intervals are shown with boxes like in Figure 3.11, containing discrete points in time in with certain spatial relations or combinations of them exist. Arrows indicate temporal relations between such intervals; for example, the relation near(y,x)[1,2] overlaps with the relation left(y,x)[2,3] , and near(y,x)[6,6] is in a temporal meets relation to mediumdist(y,x)[7,8] . As can be seen in the figure, intervals of the same or different types of spatial relations can be temporally related. A distinct feature of our work is that temporally combined spatial relations can be interpreted as high-level spatiotemporal relations which exist in certain time intervals, for which reason they can in turn be temporally combined with other relations. Hence, the idea is that logical and temporal combinations of relations, which are relevant for an application, are explicitly represented with a new semantically meaningful name and can eventually be used like any other relation. We just consider combinations of relations between the same two artifacts at a time, which is necessary for using a combined relation like any other binary relation that is also defined between two artifacts. An example of temporally combined relations is given in Figure 3.12, where the three relations near(y,x)[6,6] , medium-dist(y,x)[7,8] and far(y,x)[9,9] – with the temporal meets relation between their intervals – form a new spatiotemporal relation leaving(y,x)[6,9] , which can again be temporally com-
3.2 Qualitative Representation
65
Figure 3.12: Qualitative spatiotemporal relations.
bined with other relations. In the example above, the resulting interval [s, e] of the temporal relation between two intervals a and b is defined as follows: [s, e] = [min(sa , sb ), max(ea , eb )]
(3.2)
However, although using the minimum and maximum of the start- and endpoints in time of the combined intervals makes sense for the given example, it is just one possibility as the resulting interval in general depends on the semantics of the combined relations and is thus up to the application programmer. In addition, time intervals can also be logically combined using and, or and not; as with temporally combined relations, the resulting relation of a logical and or an or combination of two relations is assigned a name reflecting the semantics of the new relation, and the interval in which the logically combined relation is fulfilled is given by an intersection or union of the two intervals, respectively. Thus, the logical combination of relations also results in a new relation and can be used like any other spatial or spatiotemporal relation. In the case of a logical not, the interval of the resulting relation equals the repository’s whole history, except the interval in which this relation holds. For the example of Figure 3.12, the relation combination back(y,x)[8,9] ∧ leaving(y,x)[6,9] can be interpreted as a new relation leaving-backward(y,x)[8,9] . Such combinations of relations lead to semantic problems in the case that disjunctions of alternatively possible relations (i.e. compound relations, cf. Section 4.1) are involved, which is thus disregarded with respect to spatiotemporal representations in the following.
66
3 Representation of Space
In Section 2.5, we stated that relations are stored in the artifact’s repository for further use, in particular for the purpose of rule-based reasoning which will be explained later on in Chapter 5. Due to the additional consideration of time, we have proposed in [HFH+ 08, Hol07b] that each relation is not only stored with a name which identifies the semantics of the relation by its type (e.g. static orientation) including its dimension and the frame of reference, a value which represents a certain relation of that type (e.g. right) and an identification of the reference and primary Zone-of-Influence between which the relation exists (i.e. just binary relations are considered), but also with its time interval which is defined by two discrete points in time. As with spatial properties, qualitative relations can be exchanged with other artifacts via self-descriptions, which is explained in more detail in Section 3.3.2.
3.3 Structure of Self-Descriptions In the Sections 2.2.1 and 2.3, we pointed out the importance of context-sharing by means of self-descriptions for the contextual interaction of digital artifacts and hence for their autonomous operation in the background. A self-description contains information which is of relevance for the application-level interaction among artifacts, as for example an artifact’s physical size, its provided services or the spatial context it is aware of. Based on the concept of spontaneous interaction between artifacts upon coming into spatial proximity which has been proposed in [FHM+ 04, FHR+ 06a], digital artifacts in this thesis exchange their self-descriptions within communication range and compare them with respect to certain application demands, in particular to compute spatial relations that are relevant for a running application from the context information contained therein. As mentioned in Section 2.2.1, the present thesis does not address how and under which circumstances the actual exchange of self-descriptions is performed, which depends on the implementation of the communication component of the framework described in Section 2.5. In order to support the open-world assumption mentioned in Section 2.2.1, namely to enable the interaction between artifacts which do not know each other in advance, we decided for a non-fully specified XML structure of the self-description which enables applications to include arbitrary relevant structured or non-structured information. XML has been chosen because it provides readability for humans and machines on the one hand, and allows to include spatial information represented with different XML dialects – such as those discussed in Section 3.3.1 for example – on the other hand. A notable disadvantage of using XML is its redun-
3.3 Structure of Self-Descriptions
67
dancy, which leads to higher costs concerning storage, processing and communication. The XML-structure of a self-description, which has first been presented in [HFH+ 08], is fairly simple: it consists of an XML root-element DADescription, which contains DADescriptionElement elements with information about the artifact. Different types of self-describing information are distinguished with unique type attributes, the structure of the data contained in a certain XML element however is application-specific and not further specified. The XML fragment of Listing 3.1 shows the structure of a self-description. According to the focus of this thesis, it can contain elements of type ZoI, where each one represents a single Zone-of-Influence of the respective digital artifact (cf. Section 3.1), and of type QualitativeRelation, each one representing the qualitative spatial or spatiotemporal relation relation between two Zones-of-Influence (cf. Section 3.2). The self-description is open for including arbitrary further information like an artifact’s unique identifier or information about its provided services, just by including the corresponding description elements. Details about how they are used in the Zones-of-Influence Framework will be given in Chapter 6. Listing 3.1: Structure of an XML-based self-description. < DADescription > < D A D e s c r i p t i o n E l e m e n t type = " ZoI " > < D A D e s c r i p t i o n E l e m e n t type = " Qu a l i t a t i v e R e l a t i o n " >
3.3.1 Quantitative Spatial Properties There are many ways for representing quantitative spatial properties like position, direction and shape in an artifact’s self-description. A large number of different XML-based markup languages emerged over the past years, which often focus on a certain application domain. Examples are the Keyhole Markup Language (KML) [Goo08], which is designed for geographic annotations with e.g. images, polygons or three-dimensional models in earth browsers or maps, as well as the Geography Markup Language (GML) [CDL+ 04], which is a comprehensive modelling language and interchange format for geographic systems. Another proposed standard is the Physical Markup Language (PML) [FAOH03], an XML-based language for
68
3 Representation of Space
representing information about physical objects with a focus on Electronic Product Code (EPC) network enabled objects. In general, depending on the application requirements, self-descriptions may contain spatial information of arbitrary XML-based markup languages; in particular, all of the above mentioned languages could be used for representing Zonesof-Influence. This flexibility is supported by the software framework presented in Chapter 6 insofar as it allows for an easy exchange of the components that are responsible for parsing self-descriptions. However, the focus of our work is not on the investigation of markup languages for representing spatial information but rather on concepts for making autonomous embedded systems spatially aware, for which reason we have developed a simple data format that is tailored to the description of spatial properties and qualitative relations within the scope of this thesis. For the sake of simplicity, just as many information as required for evaluating the proposed concepts for spatial awareness is represented; it has been derived straightforward from the specification of Zones-of-Influence, and will be explained in the following. As described in Section 3.1, a Zone-of-Influence is specified by its position, direction and shape; thus, an XML-element DADescriptionElement of type ZoI in an artifact’s self-description contains the following information: • Name: Serves as an identification of the ZoI as well as a semantic identifier for applications. The name (element Name) must be unique per artifact. • Position: Defines the position (element Position) of the so-called anchor point of the ZoI (e.g. using WGS84 ECEF coordinates), which specifies a reference point that is important for recognizing relations like distance and orientation to other Zones-of-Influence. It is typically the value sensed by some position sensor; however, it can also be determined by other static or dynamic contexts. Details about the integration of dynamic positions provided by sensors will be given further below. It should be noted that this point need not be the physical position of the artifact or the center of the ZoI’s shape. In order to be able to compare values from different sensors with different reference systems, the used reference system has to be specified using the XML attribute refSystem – similarly to the GML markup language [CDL+ 04]. • Direction: Defines the direction (element Direction) of the anchor point – and thus of the whole Zone-of-Influence – in space, which determines how the overall ZoI is oriented in space (e.g. using Euler angles). As for
3.3 Structure of Self-Descriptions
69
position, the direction can be static or dynamic, and the reference system has to be specified using the attribute refSystem. • Shapes: Each ZoI consists of one or more shapes (element Shape) defining its spatial extension. As stated in Section 3.1.1, we use the term shape synonymously with the spatial extension of a ZoI. Like in PML [FAOH03] also, a shape can be a predefined basic shape like a sphere or a cuboid, but it can also be an arbitrary freeform shape that is defined by a sequence of points. Each shape requires specific parameters describing its characteristics, as for example the width, height and depth of a cuboid. The position and direction of a single shape are given relative – and according to the respective reference system – to the position and direction of the anchor point by using the XML elements RelativePosition and RelativeDirection, respectively. In the following, examples for representing Zones-of-Influence in self-descriptions are given, which should provide a better understanding of the framework and evaluation details presented in the Chapters 6 and 7. The XML fragment in Listing 3.2 describes the static Zone-of-Influence ExampleStaticZone, which means that its position and direction are fixed (i.e. not given by sensor values) and thus do not change over time. Moreover, the XML code shows a fragmentation of the ZoI with two shapes: a cuboid and a point. However, other basic and freeform shapes can also be represented in the description of a Zone-of-Influence. Position and direction in this example are given in Cartesian ECEF coordinates according to the WGS84 reference system and in NED Euler angles, respectively. In this regard, we assume positions to be given in meters and angles in degrees. Listing 3.2: Representation of a static and fragmented Zone-of-Influence. < D A D e s c r i p t i o n E l e m e n t type = " ZoI " > < Name > ExampleStaticZone < Position refSystem = " WGS84ECEF " > <X > 4094824 1189656 4727278 < Direction refSystem = " NEDEuler " > < Yaw > 0.4 < Pitch > -88.5 < Roll > 1.2 < Shape type = " Cuboid " > < RelativePosition > <X > 0.2 1.4 42 < RelativeDirection > < Yaw > -3.7 < Pitch > 4.5 < Roll > 0.0 < Width >2 < Height >2 < Depth >2 < Shape type = " Point " >
70
3 Representation of Space
< RelativePosition > <X >2 0 0
An important aspect of a Zone-of-Influence is that its spatial properties (i.e. position, direction and shape) can be dynamic, since the properties of moving artifacts are subject to changes. A ZoI is called dynamic if at least one spatial property depends on a dynamic context, as for example when its position is determined by a physical sensor like the IS-900 tracking system [Int08]. In order to allow for modelling dynamic Zones-of-Influence, a dedicated DADescriptionElement of type SensorData is used, which specifies both where the data can be acquired from and how the values have to be referenced for their use in the description of a Zone-of-Influence. An exemplary dynamic ZoI is shown in Listing 3.3, where the position values are given by the name of the sensor data – which is PosDir in this example – together with an @ prefix (e.g. @PosDir.x). Detailed explanations of the specification of sensor data accessors in self-descriptions (cf. Listing 6.1) and the corresponding implementations in the Zones-of-Influence Framework can be found in Section 6.2.2. Listing 3.3: Representation of a dynamic Zone-of-Influence. < D A D e s c r i p t i o n E l e m e n t type = " ZoI " > < Name > ExampleDyna m i c Z o n e < Position refSystem = " IS900AtJKU " > <X > @PosDir . x @PosDir . y @PosDir . z < Direction refSystem = " IS900AtJKU " > < Yaw >0 < Pitch >0 < Roll >0 < Shape type = " Sphere " > < Radius >5
The above examples showed how static, dynamic and fragmented Zones-ofInfluence can be defined. As mentioned in Section 3.1.1, an artifact can also be associated with multiple ZoIs; this is implicitly supported by the general structure of self-descriptions, which may contain multiple DADescriptionElement elements of the same type.
3.3.2 Qualitative Spatial Relations A self-description can not only contain quantitative descriptions of Zones-of-Influence, but also qualitative spatial relations between them. Thus, artifacts are able to exchange relations they are aware of among each other, which is a precondition for inferring relations to other artifacts to which none could be determined
3.3 Structure of Self-Descriptions
71
by a comparison of Zones-of-Influence. This is the case if the respective artifacts are out of communication range (and therefore cannot directly exchange their self-descriptions) or when the Zones-of-Influence contained in exchanged selfdescriptions are based on sensor-data which cannot be interpreted by the receiving artifact (e.g. for sensors with unknown or incompatible reference systems), among other reasons. In Section 4.3, an approach for inferring relations among artifacts over multiple hops by using transitive inference is presented, which relies on a local exchange of qualitative relations they are aware of. In contrast to the representation of ZoIs in self-descriptions, the exchange of qualitative relations has not been integrated in the Zones-of-Influence Framework (cf. Chapter 6) at the time of writing this thesis yet, for which reason simulation is used for the purpose of evaluation in the next chapter. As described at the end of Section 3.2.5, qualitative relations between Zones-ofInfluence are stored in the artifact’s repository with a name and a value as well as an identification of the two ZoIs between which the relation exists and the respective time interval. They are thus – by using DADescriptionElement XML elements of type QualitativeRelation as shown by means of an example in Listing 3.4 – represented with the following information in self-descriptions: • Name: Identifies the type of the relation (e.g. StaticOrientation as defined for 2-D in Figure 3.6), which must be unique for the group of artifacts among which self-descriptions are exchanged. With this name, the relation semantics are fully specified, including the dimensionality, the used spatial primitives, its granularity and the frame of reference (cf. Section 3.2.2). • Value: Represents a certain relation in a qualitative way (e.g. right), whose meaning is defined by the corresponding relation type and which must be unique for that type. • Reference ZoI: Specifies the reference Zone-of-Influence of the relation. Since a ZoI must be unique per artifact (cf. Section 3.3.1), it is given by the schema <artifact identification>:. • Primary ZoI: Specifies the primary ZoI of the relation in the same way like the reference ZoI, which is usually a Zone-of-Influence of the artifact from which the self-description has been received. The time interval in which a qualitative relation exists is another important information which has to be stored in the relations repository in order to enable spatiotemporal reasoning. However, just the currently existing relations without any temporal information are exchanged with self-descriptions, which is mainly for
72
3 Representation of Space
two reasons. First, in order to take advantage of other spontaneously networked artifacts’ relations which are associated with certain intervals in time, a synchronization of the clocks among them – which requires additional computing and communication resources – would be necessary. Secondly, and most importantly, a synchronization of clocks is contrary to our general focus on autonomously operating digital artifacts, which act independently of other artifacts and on the basis of locally perceived information only. Listing 3.4: Representation of a qualitative relation. < D A D e s c r i p t i o n e E l e m e n t type = " Qu a l i t a t i v e R e l a t i o n " > < Name > StaticOrientation < Value > right < ReferenceZoI > 00 -00 -00 -70 -05 -75 - D3 -49 :PhysicalShape < PrimaryZoI > 00 -00 -00 -70 -05 -75 -67 -1 B:PhysicalShape
3.4 Summary In this chapter we have discussed representations of spatial aspects, which are the basis for providing spatial awareness to spontaneously interacting artifacts. Zones-of-Influence have been introduced as an approach for representing quantitative spatial properties which are relevant for an application. Each artifact can be associated with one or more of such Zones-of-Influence, which are geographic regions of a certain shape and size that can be defined relative to an artifact’s position and direction in space. Just static spatial properties are taken into account, and their position and direction is represented with respect to a fixed, external frame of reference. Extensions of this concept by additionally including dynamic spatial properties such as the artifact’s angular acceleration or linear velocity are of course possible, but not considered within this thesis. For the sake of simplicity, we are mainly concerned with two-dimensional Zones-of-Influence; however, the presented concepts are conceptually similar for spatial properties in three dimensions. Zones-of-Influence are shared among artifacts with so-called self-descriptions, which enables them to recognize spatial relations by a pairwise comparison of their local with received Zones-of-Influence. Our focus is on qualitative relationship abstractions, which have advantages whenever the spatial cognition of humans or resource-constrained systems are involved, among others. A crucial point is the observation that the semantics of qualitative relations – including the used spatial primitives, the way how the space is partitioned and the reference frame
3.4 Summary
73
with respect to which they are defined – depends on the spatially aware application that makes use of them. There are often multiple ways for determining a certain relation; for example, distance relations can be determined by relating point abstractions of two artifacts, or by defining Zones-of-Influence whose spatial extensions correspond to qualitative distances and recognizing topological overlaps relations between them. We have discussed a classification of spatial relations with respect to the topology, position and direction of two Zones-of-Influence at a time by means of examples. In this regard, static relations such as overlaps and left as well as dynamic relations such as moving-away or growing, which we consider valuable for characterizing the spatial situation of real-world objects with respect to their surroundings, have been distinguished; however, cognitive aspects concerning the semantics of relations are disregarded in the present thesis. We use just binary relations (i.e. relations between two Zones-of-Influence), and thus only intrinsic and extrinsic reference frames for the representation of orientation. For the consideration of time-series of qualitative spatial relations, we propose their logical and temporal combination to spatiotemporal relations, which allows for dealing with trajectories of spatial relations that have a certain meaning for the application in a qualitative way. Therefore, relations are associated with discrete time intervals in which they exists, which can be related with qualitative temporal relations such as before or during. Each relation, which results from a combination of other relations, is explicitly represented with a semantically meaningful name as well as a time interval in which it exists; a rule-based approach for inferring high-level relations by a combination of known relations will be presented in Chapter 5. Thus, an inferred relation can in turn be combined with other relations, which constitutes a distinguishing feature of our work. Compound relations – i.e. disjunctions of alternative relations which are commonly the result of composition operations (cf. Chapter 4) – are disregarded with respect to relationship combinations. XML-based self-descriptions are used for the exchange of information which is relevant for the application-level interaction between artifacts, whereas our focus is on quantitative spatial properties as well as on qualitative spatial relations. We have proposed and exemplarily discussed a structured data format for their representation within self-descriptions in this chapter, which is kept as simple as possible so as to be able to realize and evaluate the developed concepts for spatial awareness in Chapter 7. The semantics of exchanged spatial information (e.g. given by the reference system of a Zone-of-Influence or the type of a qualitative relation) must be known by all interacting artifacts to be able to use them. For the exchange of qualitative relations, just currently existing relations without any temporal information
74
3 Representation of Space
are considered in order to cope with the autonomy requirement of spontaneously interacting digital artifacts.
4 Distributed Spatial Reasoning It is quality rather than quantity that matters. Lucius Annaeus Seneca, c. 4 BC - 65 AD
Reasoning about perceived context information is fundamental for autonomous systems to draw conclusions about their spatial contexts and adapt to changing situations at runtime. According to our focus on qualitative spatial relations, this chapter builds on the qualitative abstractions presented in Chapter 3 and is concerned with inferring new from known relations. In Section 4.1, we first give an overview of approaches for qualitative spatial reasoning, with a focus on binary relations and their properties as well as compositional reasoning about them. In Section 4.2, we discuss reasoning about positional and directional relations in more detail and present composition tables for distance and orientation relations. The main part of this chapter is about an algorithm for inferring and distributing spatial relationships among autonomous digital artifacts over multiple hops, which is presented in Section 4.3 and evaluated by simulation means in Section 4.4. The general idea is that artifacts recognize spatial relations to others in their vicinity by sharing spatial context information, and infer relations to artifacts out of communication range (i.e. data cannot be exchanged directly, but just via other artifacts) by means of qualitative spatial reasoning techniques which are discussed in this chapter. In particular, the algorithm makes use of composition tables or the properties of binary relations in order to provide artifacts, which are out of communication range, with an awareness about their spatial relationships to each other. Section 4.5 eventually discusses benefits and drawbacks of the presented qualitative spatial reasoning approaches and elaborates on their use for autonomous embedded systems.
4.1 Overview of Qualitative Approaches Qualitative spatial reasoning in general builds upon spatial relations as discussed in Section 3.2, and aims at providing computers with the ability to draw conclusions in a qualitative manner. With regard to our focus on autonomous embedded
76
4 Distributed Spatial Reasoning
systems, qualitative spatial reasoning allows digital artifacts to infer new relations based on known ones, and thus improves their spatial awareness in that implicitly available knowledge is made explicit and can be directly used at the application level. It is typically realized with calculi over sets of qualitative spatial relations, from which a considerable number has been developed within the past two decades. They focus on different spatial properties, where most of the research efforts have been concerned with topological reasoning about regions as well as reasoning about distance and orientation relations on points or line segments [DFWW06, KB99]. In the following, the foundations of qualitative spatial reasoning will be discussed as far as they are relevant to understand the proposed approach for relationship distribution presented in Section 4.3; starting points for further information on this research field can be found in several survey papers [CH01, FR93, HN02] and books [For97, Her94]. As stated in Section 3.2.2, this thesis is concerned with binary relations between artifacts – or, more precisely, between Zones-of-Influence which are associated with them; we use object synonymously for Zone-of-Influence in the following, as it is the common term in the research field of qualitative spatial representation and reasoning. A relation R between two objects x and y (i.e. (x, y) ∈ R) is often denoted as R(x, y), where x is referred to as the primary and y as the reference object. According to [DFWW06], a binary spatial calculus consists of (i) a domain D containing the spatial primitives (e.g. points in two-dimensional space representing physical objects), (ii) a finite set BR of typically jointly exhaustive and pairwise disjoint 2-ary base relations on that domain, as well as (iii) a set of operations which are closed over the powerset R of these base relations. Thus, the result of an operation may be a compound relation (i.e. the union of multiple base relations), which allows for expressing uncertainty (e.g. that an object x is either left or front of y); for this reason, the operations of a calculus have to be defined for all possible unions of base relations (cf. Section 4.2.2). A special case in this regard is the universal relation, which is a disjunction of all base relations and represents the complete lack of knowledge about the relation between the respective two objects [NS02, WFDW07]. The following operations union (∪), intersection (∩) and composition (◦) on binary relations are relevant for our approach, where R, S ∈ R [DFWW06]: R∪S
= {x|(x ∈ R) ∨ (x ∈ S)}
(4.1)
R∩S
= {x|(x ∈ R) ∧ (x ∈ S)}
(4.2)
R◦S
= {(x, z)|∃y ∈ D : (x, y) ∈ R ∧ (y, z) ∈ S}
(4.3)
4.1 Overview of Qualitative Approaches
77
In the Sections 4.1.1 and 4.1.2, closures of binary relations and compositional reasoning are discussed, which build up the basis for the relationship distribution algorithm presented in Section 4.3. In this regard, our focus is on static positional and directional relations, topological and temporal relations will not be considered in the remainder of this chapter.
4.1.1 Properties and Closures of Binary Relations Once an artifact has become aware of certain qualitative relations to and between other artifacts, it may be able to infer new relations from them, which is the subject of qualitative spatial reasoning. One possibility is to exploit the properties reflexivity, symmetry and transitivity of binary relations R over a set D, where any of them may but need not be fulfilled for a given relation: ∀x ∈ D : ∀x, y ∈ D : ∀x, y, z ∈ D :
(x, x) ∈ R (reflexive)
(4.4)
(x, y) ∈ R ⇒ (y, x) ∈ R (symmetric)
(4.5)
(x, y) ∈ R ∧ (y, z) ∈ R ⇒ (x, z) ∈ R (transitive)
(4.6)
Table 4.1 shows a selection of spatial orientation and distance relations as presented in Section 3.2.3, together with their properties. These relations are extrinsic relations, i.e. referring to an external reference like the earth or a building (cf. Section 3.2.4). For example, the relation left is transitive, as an artifact x, which is left of y, is also left of z if y is left of z, but it is neither symmetric nor reflexive. On the other hand, the relation near is both symmetric (if x is near y, then y is also near x) and reflexive (as x is inherently near to itself), but not transitive as the distance between x and z not only depends on the distance between x and y as well as between y and z, but also on the orientation relation between those artifacts [CFH97]. The properties of intrinsic orientation relations however are not transitive, which is for the following reason (cf. [Fra98]). In the above example that x is left of y, the orientation relation is defined with respect to the walls of a room for example, but independent of the artifact’s direction in space. In this extrinsic case, it does not matter which one of its sides is directed upwards for example. On the other hand, if the artifact’s intrinsic front side is of relevance for the application – e.g. given by an embedded sensor which determines the artifact’s axis of direction, which we have utilized e.g. in [FRH06, FRHR05] – the relation left is no longer transitive as it depends on the reference artifact’s spatial direction (i.e. the artifacts y and z in the above case). The properties of binary relations can be exploited for inferring relations between artifacts without reverting to quantitative spatial information, which we use
78
4 Distributed Spatial Reasoning Table 4.1: Extrinsic orientation and distance relations with their properties.
reflexive
symmetric
transitive x x x x
front right back left near medium-dist far
x
x x x
for our relationship distribution algorithm that has first been published in [HF07]. The general idea is to process the transitive, symmetric and reflexive closure for all relations over a group of artifacts, which are defined as follows: r(R) = R ∪ Δ with Δ = {(x, x)|x ∈ D} (reflexive closure)
(4.7)
s(R) = R ∪ R−1 with R−1 = {(y, x)|(x, y) ∈ R} (symmetric closure) (4.8) t(R) =
∞
Ri with Ri+1 = Ri ◦ R (transitive closure)
(4.9)
i=1
From an implementation viewpoint, the closure with respect to a certain relation property is processed iteratively for every relation R over a group of artifacts. As the artifacts are distributed in space, each artifact x shares both quantitative spatial properties and qualitative spatial relations it is aware of via self-descriptions with others in communication range, and infers additional relations R as defined by the respective closure. Like those relations which have been recognized by a comparison of quantitative spatial properties, inferred relations are eventually added to the artifact’s relations repository and included in its self-description. To find the reflexive closure, x infers the relation (x, x) ∈ R. For the symmetric closure, x infers (x, y) ∈ R for each relation (y, x) ∈ R it is aware of. In both cases, inferred relations concern the direct neighbors of an artifact only. For the transitive closure, artifact x infers the relation (z, x) ∈ R if it is already aware of (y, x) ∈ R and acquires the qualitative relation (z, y) ∈ R through a received self-description of artifact y. The transitive closure provides efficient means for inferring relations between artifacts out of range, namely without routing self-descriptions over multiple hops; a detailed explanation will be given in Section 4.3.1. A precondition for process-
4.1 Overview of Qualitative Approaches
79
ing closures is that the involved artifacts have knowledge about the properties of exchanged relations.
4.1.2 Compositional Reasoning Compositional reasoning has received much attention in the research community, as for example in [All83, Fre92b, Her94, Sha96]. Basically, the fundamental question involved is the following: “Given the relation between two objects x and y, as well as between y and z, what is the relation between x and z?”. The result of a composition operation depends on the meaning of the relation, that is it has to be developed from the relation’s geometric semantics [MDF05], and it is often a compound relation (i.e. a set of alternatively possible base relations). The results are commonly stored in composition tables, which define the resulting relations for all possible compositions of base relations; compound relations can be computed as the union of the compositions of base relations. Usually, an iconic notation is used for it, which represents the neighborhood structure of relations; two relations are referred to as conceptual neighbors if and only if they can be directly transformed into each other (i.e. without passing other relations) [Fre91]. A detailed explanation of the composition of positional relations on the basis of composition tables will be given in Section 4.2. The transitivity of relations can be seen as a special case of the composition operation, where all three relations – the two composed ones as well as the resulting relation – are the same [Sha96]. This is obvious from the explanations in Section 4.1.1, where the transitive closure is defined as the union of repeated composition operations on pairs of objects between which the same relation R holds. Although this approach is universal in the sense that it can be applied to arbitrary relations whose properties are known, it is quite limited as many relations like distance and intrinsic orientation are not transitive. Based on the exploitation of transitivity, we thus proposed an approach for distributing relationship information throughout a network of artifacts in [Hol07a] by using compositional reasoning, which will be presented in Section 4.3.2. To the best of our knowledge, the distribution of qualitative spatial relations among digital artifacts using transitive inference as described above is a novelty of this thesis. Figure 4.1 should clarify the idea of compositional reasoning among digital artifacts by means of a simple theoretical example. It shows a distribution of physical objects o1, . . . o8 in space, as well as a person p which is interested in its orientation and distance relations to them. We assume that the person and all objects are equipped with digital artifacts that are able to localize themselves according to an extrinsic frame of reference (i.e. represented with arrows) and communicate
80
4 Distributed Spatial Reasoning
Figure 4.1: An example for compositional reasoning.
with each other within a short range (i.e. represented with dotted lines between them). However, although artifact p is able to communicate with o3 only, it should acquire the location of all other objects relative to its own position. This can be solved trivially by letting each artifact exchange its position information with all other artifacts. In the following sections, we will tackle the issue of how compositional reasoning can be exploited to answer such questions with purely qualitative relations and by exchanging spatial information with direct neighbors only. The idea is that artifacts determine relations to their direct neighbors (e.g. o4 recognizes that o6 is in the relations front and medium-dist, and o3 recognizes that o4 is front and near to itself), exchange this knowledge with them (e.g. o4 tells o3 about its relation to o6) and thus allow their neighbors to infer relations to artifacts out of range using appropriate composition tables (e.g. o3 infers that o6 is front and either in a medium-dist or far distance relation according to Table 4.5).
4.2 Reasoning about Positional and Directional Relations In this section, we will have a closer look at compositional reasoning, where the focus is on positional (i.e. orientation and distance) and directional relations as previously shown in Figure 3.6. First, a literature survey on related work is presented, which focuses on popular spatial calculi that are closely related to our work; due to the huge number of calculi however, the survey does not claim to be exhaustive. Afterwards, a new spatial calculus that has been published in [Hol07a] is presented, which allows for reasoning about static orientation and distance rela-
4.2 Reasoning about Positional and Directional Relations
81
Table 4.2: Comparison of approaches for reasoning about static spatial relations.
orientation
a no
distance
Freksa Ligozat et al. Hernández et al. Abdelmoty et al. Moratz et al. Renz et al. Zimmermann et al.a Clementini et al.a
x x x x x x x
x x
Proposed approach
x
x
direction
x x x x
xb
calculus is specified considered for the relationship distribution algorithm of Section 4.3
b not
tions in a combined fashion. We consider the proposed approach valuable for the use with resource-constrained autonomous embedded systems, and it is eventually exploited for our relationship distribution algorithm presented in Section 4.3.
4.2.1 Related Approaches Table 4.2 shows a comparison of popular qualitative spatial calculi which are classified according their consideration of (i) orientation, (ii) distance and (iii) directional relations. As stated above already, topological and temporal relations are not taken into account in this regard. Many spatial calculi incorporate orientation only, like for example Freksa’s ternary double-cross calculus [Fre92b] and the flip-flop calculus of Ligozat et al. [SN04], which describe the deictic orientation of a point in the plane with respect to a vector that is given by two further points. However, they can represent intrinsic or extrinsic orientation relations also by viewing them as positional binary relations between a dipole and an isolated point at a time. On the other hand, there are only a few calculi dealing with spatial distance only, as for example that of Hernández et al. [HCF95] who particularly addressed the composition of distances depending on intrinsic orientation relations. Early work that combines orientation and directional relations is that of Abdelmoty et al. [AEG94], which represents them by computing the intersections of orientation lines. The binary dipole relation algebra of Moratz et al. [MRW00] uses
82
4 Distributed Spatial Reasoning
straight line segments, which are each determined by a pair of points in order to represent orientation relations between objects. A continuative calculus of Moratz et al. is the oriented point relation algebra [MDF05], where oriented points are used instead of dipoles, and the granularity is adjustable with a single parameter; the exact set of base relations thus depends on the chosen level of granularity. A similar approach with arbitrary granularity is presented by Renz et al. in [RM04], who developed the star calculus for relating two points in a plane with respect to a reference point or direction. However, there is only few existing work about the combination of orientation and distance relations. Zimmermann et al. [ZF96] add distance to their ternary calculus for representing orientation relations, and show how distance information restricts the possible orientations; this dependency is also discussed in [Sha96]. Clementini et al. [CFH97] show the interplay between orientation and distance relations, but also do not present composition tables for reasoning about them in combination. To the best of our knowledge, there is no existing work which deals with compositional reasoning about combined binary orientation and distance relations as presented in Section 4.2.2, neither taking into account directional relations nor without considering them. Moreover, in Section 4.3 we use it for distributing relationship information among autonomous artifacts, which also seems to be new.
4.2.2 Composition of Positional Relations Building on the fundamentals of compositional reasoning explained in Section 4.1.2, we specifically address the composition of static positional relations in the following. We consider them the most valuable ones with respect to relating spatially distributed digital artifacts, as they can be processed with comparatively little computing efforts and are thus suitable for resource-constrained embedded systems. Moreover, most topological relations are not meaningful for physical objects (in contrast to Zones-of-Influence which may be associated with them), and directional relations – as for example that two objects are directed similarly – are often not relevant for spatially aware applications. This estimation is also reflected by the related work survey in Section 2.6, where only few projects take advantage of relations between the directions of artifacts. We have developed composition tables for the four orientation base relations front, right, back and left as well as for the three distance base relations near, medium-dist and far, according to their definitions in Section 3.2.3. Composition tables specify the possible relations RS(z, x) of a certain relation type between two objects x and z by taking into account their relations R(y, x) and S(z, y) to a com-
4.2 Reasoning about Positional and Directional Relations
83
mon third object y, separately for all possible combinations of base relations. As mentioned already, the composition of base relations has to be computed depending on their semantics, which means that we have to consider how exactly they partition the space; in the following, this is in four equally sized sectors for orientation and three circular ranges of the same size (except the outer range which is open) for distance relations as shown in Figure 3.6. In general, it can be said that the construction of composition tables is an error-prone and time-consuming process [DFWW06], but they make efficient reasoning by simple table lookups possible once they are available. Although not considered in this thesis, composition tables can also be used for composing relations of different types, for example to infer topological from directional relations as has been done in [Sha96]. An important issue regarding orientation relations is the considered reference frame, as the result of a composition operation depends on the directional relation between the objects x and y. For an extrinsic frame of reference (cf. Section 3.2.4), an external reference direction is used for all objects involved in the reasoning process, which means that they have exactly the same direction in space. In the case of an intrinsic frame of reference, the orientation relations depend on the direction of the respective reference objects x for the relation R(y, x) and y for the relation S(z, y) (cf. [Her94]). For qualitative compositional reasoning about intrinsic orientation relations, it is thus necessary to integrate qualitative directional information in the reasoning process also. We therefore distinguish the relations similar-dir, right-dir, opposite-dir and left-dir, which have been defined in Section 3.2.3. Consequently, dedicated composition tables for intrinsic and extrinsic positional relations are required, with 12 base relations in the former (due to the four orientation and three distance relations) and 48 base relations in the latter case (due to the four additional directional relations). A finer granularity by distinguishing more relations is of course possible; however, changes in the number and semantics of relations require a reconstruction of the composition table. How the composition of positional relations among three artifacts x, y and z is acquired can be seen in Figure 4.2, exemplarily for the two positional relations front(y,x) ∧ near(y,x) and right(z,y) ∧ near(z,y), and for an extrinsic reference frame at the top as well as for a similar direction between the objects x and y – i.e. similar-dir(y,x) – at the bottom of the figure. Due to the abstraction of space in discrete partitions, a composition operation usually results in a compound relation, which means that not a single but multiple alternatively possible relations are provided. While the former composition results in a disjunction of just four relations – these are front(z,x) ∧ near(z,x), front(z,x) ∧ medium-dist(z,x), right(z,x) ∧ near(z,x) as well as right(z,x) ∧ medium-dist(z,x) – the latter results in a set of even six relations as a consequence of the additional range of possible alternative
84
4 Distributed Spatial Reasoning
Figure 4.2: Composition of positional relations with same (top) and similar directions.
directions of object y with respect to x. It can be seen that intrinsic orientation relations lead to many disjunctions and are thus practically less relevant, for which reason our focus is on extrinsic orientation and distance in the following. The results of intrinsic composition will become more specific if the number of distinguished qualitative directional relations is increased, but at the same time lead to bigger tables due to the higher number of base relations. The Tables 4.3 and 4.4 show the separate composition for extrinsic orientation with four and distance with three base relations, which means that orientation and distance are composed independently. An iconic notation is used, which represents the neighborhood structure among the positional relations (cf. Section 4.1.2). The four orientation base relations are visualized with black dots indicating their orientation, and the three distance base relations are visualized with filled areas indicating their spatial ranges; disjunctions of base relations, which are often the result of a composition operation and represent possible alternative relations, are visualized by superimposing their icons like for example in [Fre92b]. The results in the main diagonal of Table 4.3 equal the corresponding orientation base relations, which implies that static orientation is a transitive relation. As stated in Section 4.1, the composition of compound relations can be computed by a composition of the respective base relations and a subsequent union of the resulting – possibly compound – relations. From these tables, it can be seen that the composition operation often leads to coarse results, which is also depicted in the literature
4.2 Reasoning about Positional and Directional Relations
85
Table 4.3: Composition table for static orientation relations.
Table 4.4: Composition table for static distance relations.
[Her94, Sha96]; for example, the compositions of front and back or of near and medium-dist result in the union of all orientation or distance base relations respectively, which are referred to as universal relations. The composition table for combined orientation and distance relations that has been published in [Hol07a] is shown in Table 4.5, partially for six base relations at a time; all others can be determined by simply rotating the table. The composition table has been created by moving an object y within the range of a particular quali-
86
4 Distributed Spatial Reasoning Table 4.5: Composition table for combined orientation and distance.
tative relation R(y, x) to x as shown in Figure 4.2, and determining which positions are possible for a third object z in order to fulfill a second base relation S(z, y) to y; these possible positions eventually define the resulting compound relation RS(z, x). The upper example of Figure 4.2, namely static orientation for an extrinsic reference frame, is highlighted in Table 4.5. As can be seen, more accurate results are achieved by a combined consideration of orientation and distance; for example, composing the extrinsic distance relations medium-dist(z,y) and far(y,x) only yields the universal relation, but additionally taking into account e.g. the orientation relations front(z,y) and front(y,x) results in just one distance relation, namely far(z,x). Another example are the orientation relations front(z,y) and back(y,x), whose composition again results in the universal orientation relation, but much more specific results can be inferred in combination with the according distance relations.
4.3 Spatial Relationship Inference and Distribution
87
4.3 Spatial Relationship Inference and Distribution This section presents an approach for inferring and distributing qualitative spatial relations between digital artifacts, with the aim of providing a group of artifacts with an awareness about spatial relations among each other. Artifacts hence cooperate by sharing their spatial contexts, which allows them to recognize relations to artifacts in vicinity and infer further relations – even to others that are out of range – using qualitative spatial reasoning techniques as discussed in Section 4.1. After presenting the general concept by taking advantage of the properties of binary spatial relations, an algorithm is proposed which is based on compositional inference and thus allows for inferring more significant conclusions. An evaluation of the algorithm with different topologies of networked groups of artifacts by simulation means will be given afterwards in Section 4.4.
4.3.1 General Concept by Exploiting Relation Properties We distinguish two general approaches for providing a group of artifacts, which are either connected directly (i.e. they are within communication range) or via other artifacts, with a mutual spatial relationship awareness: (i) exchanging quantitative spatial properties between each pair of artifacts or (ii) sharing spatial properties with directly connected artifacts only, together with their knowledge about qualitative spatial relations to other artifacts. While the former can be done by simply broadcasting spatial properties throughout the network, which means that selfdescriptions are exchanged between distant artifacts over multiple hops, we developed a new approach for the latter which is based on locally perceived information only. The proposed approach exhibits self-organizing behavior in that artifacts autonomously, and in an ad-hoc manner, establish a collective spatial awareness of each other (cf. Section 2.3). Although there exist several approaches for localizing nodes in ad-hoc networks (cf. [LR03]), as for example [BSHK06] and [CHH02] which are both based on quantitative distance estimates between nodes, ours seems to be the only one which is solely based on qualitative spatial relations. The exchange of self-descriptions is the basis for relationship inference and distribution among digital artifacts. As explained in Section 2.2.1, the self-description of a particular artifact contains both quantitative spatial properties represented with Zones-of-Influence and qualitative spatial relations it is aware of (e.g. acquired by a pairwise comparison of its local with received self-descriptions), as well as any additional information which is required by a particular application. In the case of our relationship distribution algorithm presented below, this is an identification of the artifact as well as a sequence number which is unique for a particular process
88
4 Distributed Spatial Reasoning
of inferring and distributing relations among a group of artifacts. Each qualitative relation is given by (i) a unique name identifying its type, (ii) a value which represents a certain relation of that type as well as (iii) the reference and primary artifact between which the relation exists; in contrast to Section 3.2.5 however, no time intervals are stored as temporal relations are not considered in the following. Due to our focus on positional relations, digital artifacts are abstracted using point-shaped Zones-of-Influence, to which we refer to as artifacts in the following as they simply represent the artifacts’ positions. Moreover, we assume that their qualitative relations to each other do not change during the process of relationship inference and distribution. The simplest way for achieving mutual spatial relationship awareness is a flooding of self-descriptions, which enables each artifact to recognize relations by a pairwise comparison of its own spatial properties with those contained in received self-descriptions. We consider it the most feasible approach for the case that the self-description of each artifact has to be sent to all other artifacts of a group. We are aware of alternative broadcasting techniques surveyed in [WC02], which are based on (i) an assignment of probabilities for rebroadcasting messages, (ii) the assumption that nodes have common transmission distances or (iii) the maintenance of knowledge about its neighborhood by each node, and therewith aim at reducing the number of redundant broadcasts; however, as they either do not guarantee that all nodes are reached or require topology information obtained from periodic "hello" packets, we decided for using simple flooding. Nevertheless, flooding (i) induces a high amount of traffic as will be seen in Section 4.4, which is because each artifact broadcasts all self-descriptions it receives for the first time, and (ii) it requires much processing power due to the fact that all relations are calculated from the quantitative spatial properties. With regard to battery-powered embedded systems with constrained computing resources, flooding may thus simply exceed their computing and communication capabilities. An alternative we have proposed in [HF07] first is to exploit the relation properties reflexivity, symmetry and transitivity as defined in Section 4.1.1 in order to infer certain relations with purely qualitative means. Assuming that an artifact also shares its qualitative spatial relations and is aware of those relations’ properties, it is able to acquire relations without having to process them from quantitative spatial properties (which are possibly unavailable). By sharing inferred relations with their own self-descriptions, in particular the transitive closure with respect to a certain relation can be processed, which allows for acquiring relations between distant artifacts without an exchange of spatial contexts between them.
4.3 Spatial Relationship Inference and Distribution
89
In the remainder of this section, the general concept of our approach for inferring and distributing relations is explained, which comprises the following four steps: 1. A certain artifact y broadcasts its self-description, e.g. in regular intervals or due to significantly changed sensor values as a result of movement, to surrounding artifacts in communication range. 2. Each artifact x, which receives the self-description of the broadcasting artifact y for the first time (i.e. determined by the sequence number mentioned above), recognizes spatial relations R(y, x) to y according to the application demands (cf. Section 3.2.2) from the quantitative spatial properties contained therein, and adds them to its own self-description. 3. It then infers further relations from the already known and newly recognized or received ones (i.e. contained in the self-description of y) by taking advantage of qualitative spatial reasoning techniques – either based on relation properties (cf. Section 4.1.1) or on compositional inference (cf. Section 4.1.2) – and adds them to its self-description, too. 4. If either the relations in the self-description of x changed (i.e. new relations have been added) or it is the first time a self-description has been received, x broadcasts its self-description containing current quantitative spatial properties – in the case of limited communication bandwidth they can be included just for the first broadcast – and the qualitative spatial relations it is aware of, together with the sequence number of the received self-description. More detailed information about the algorithmic implementation of the above approach will be given in Section 4.3.2. The Figures 4.3 and 4.4 demonstrate how the algorithm works for a two-dimensional Euclidean space. Although possible, we do not consider a hop-count in the following, which would allow for restricting the spatial range in which relations are distributed. The vertices of the graphs represent a group of five wirelessly communicating artifacts a, . . . e, and edges between them show that the respective artifacts are within communication range. Broadcasts of self-descriptions are illustrated with curved arrows from one vertex to another, and the relations below them show the spatial relationship awareness of the respective artifacts at certain points in time during the relationship distribution. A projection-based qualitative abstraction has been used for the transitive orientation relations, which partitions the space in the four half-planes left, right, front and back using two orthogonal axes. They are aligned according to an external reference frame that is indicated by the x- and y-axes at the top left of the figures.
90
4 Distributed Spatial Reasoning
Figure 4.3: Distribution of the transitive relation left.
The reflexive and symmetric distance relations are defined with concentric circles as described in Section 3.2.3, where the four distances near, medium-dist and far are distinguished. To keep the examples short and clear, we will consider just the relations left and near. They are defined as follows, where d is a certain distance which equals the range of near and xi , yi ∈ R indicate the Euclidean position of artifact i: left(a,b)
⇔
xa < xb
near(a,b)
⇔
(xa − xb ) + (ya − yb ) < d
(4.10) 2
2
2
(4.11)
In step (1) of Figure 4.3, artifact d first broadcasts its self-description with a unique sequence number, which is received by the artifacts c and e. They thereupon perform the recognition, inference and possible broadcast as described above. While artifact e recognizes that artifact d is on the left hand side, i.e. e has become aware of the relation left(d,e), artifact c clearly does not recognize this relation as it is to the right of d. Both artifacts c and e then broadcast their own self-description in step (2) as they have received it for the first time, whereby the self-description of e already contains the newly recognized relationship to d. This distribution of relations continues in step (3), where more and more relations are being recognized.
4.3 Spatial Relationship Inference and Distribution
91
Figure 4.4: Distribution of the symmetric and reflexive relation near.
In step (4) of Figure 4.3, artifact d infers the two relations left(a,d) and left(b,d) from the relations left(a,c) and left(b,c) it received from artifact c as well as the knowledge about the relation left(c,d) and that left is a transitive relation. Due to the changed relations, artifact d again broadcasts its self-description, now containing all three relations it is aware of. This in turn also makes artifact e aware about its relations to a and b in step (5). Finally, this distribution process terminates in step (6), as no further relations are inferred. Figure 4.4 shows a similar picture for the reflexive and symmetric relation near, again with artifact d broadcasting its self-description first. Artifact c, which receives the self-description of d, recognizes the relation near(d,c) by a comparison of the spatial properties of c and d, and then infers the relations near(c,d) due to the symmetry of the relation near as well as near(c,c) due to its reflexivity. According to the distribution approach described above, we assume that the relationship inference always takes place upon receiving a self-description; for this reason, also the relation near(d,d) is not recognized until step (2), although the overall reflexive closure could be processed by each artifact without requiring information contained in received self-descriptions. For both examples, the result is a group of artifacts in which every one is aware of its left and near relations to each other, as far as they could be recognized and inferred by taking advantage of their properties. From a mathematical viewpoint, the transitive closure of the relation left as
92
4 Distributed Spatial Reasoning
well as the reflexive closure and the symmetric closure of the relation near have been calculated (cf. Section 4.1.1). It should be noted that the figures are simplified in that all nodes broadcast at the same time; in fact, artifacts broadcast their self-descriptions independently from each other. In contrast to the transitive orientation example, no relations are inferred over multiple hops in case of the reflexive and symmetric distance relation. However, as we are interested in inferring relations between distant artifacts which cannot be acquired directly, like for example if they are out of sensor- and communicationrange, our focus will be on transitive inference in the following. Exploiting the transitivity property for reasoning purposes has the advantage that it can be applied to arbitrary binary relations and is not restricted to spatial ones; for example, all relations with the semantics “equals”, “greater than” or “implies” are transitive. Moreover, it reduces the communication required between artifacts in comparison to flooding, as no self-descriptions are routed through the network of artifacts and local self-descriptions are broadcast upon changed relations only (cf. Section 4.4). On the other hand, a major disadvantage is that just a subset of the relations which hold between artifacts can be detected compared with an availability of quantitative sensor data between all pairs of artifacts, which is for two reasons. First, many relations – such as intrinsic orientation and static distance (cf. Section 4.1.1) – are not transitive, and thus no transitive closure can be processed for them. Second, processing the transitive closure requires that all three relations – i.e. the two composed as well as the resulting one – have the same value, for which reason e.g. no front or back relation can be inferred between the artifacts b and d in our example.
4.3.2 Distribution Algorithm Using Compositional Inference Although the distribution approach described above allows for reasoning about qualitative spatial relations under certain circumstances, we argued that it is quite limited as it works for the same relation values at a time only and many relations are not transitive. As stated in Section 4.1.2, transitivity can be seen as a special case of composition where the two composed as well as the resulting relation are the same. As a logical next step, we thus replaced the reasoning step with compositional inference using pre-defined composition tables such as those shown in Section 4.2.2, which often allow for inferring more accurate results. Given appropriate tables, it is possible to infer new from two existing relations which (i) need not have the same value (e.g., the orientation relations front and right can be composed) and (ii) need not be transitive (e.g. a composition of distance relations is possible). Moreover, composition allows for reasoning about compound relations,
4.3 Spatial Relationship Inference and Distribution
93
which means that uncertainty can be taken into account which is also not the case if just the transitivity property is exploited. Algorithm 1 Operations of artifact x upon receiving the self-description of y. 1: if self-description of y received the first time then 2: recognize qualitative relation of y to x and put it to self-description of x 3: end if 4: R ← get qualitative relation of y to x from self-description of x 5: for all artifacts z which are in relation to y do 6: if z = x then 7: S ← get qualitative relation of z to y from self-description of y 8: RS ← perform composition R ◦ S 9: if self-description of x already contains relation of z to x then 10: intersect known relation with composition result RS 11: else 12: put composition result RS to self-description of x 13: end if 14: end if 15: end for 16: if first self-description received or relations in self-description of x changed then 17: broadcast self-description of x 18: end if According to the approach for inferring and distributing relations described in Section 4.3.1, we have developed an algorithm and implemented it for the purpose of evaluation (cf. Section 4.4). Algorithm 1, which has been published in [Hol07a], describes the operations an artifact x performs upon receiving the self-description of artifact y. At the first time artifact x receives the self-description of y during the same process of inferring and distributing relations, which is determined by a unique sequence number contained in the self-description of y, it recognizes the relation R(y, x) and puts it to its own self-description. Afterwards, artifact x infers relations RS by composing the relation R with each relation S contained in the received self-description, whereas R and S can be compound relations. The composition operation is thus repeatedly applied to triples of artifacts, and the results are again added to the self-description of x. If the composed relations R and S are compound ones, the composition result RS is the union of the compositions of base relations (cf. Section 4.2.2). In order to retain the most accurate result, RS is eventually intersected with relations to
94
4 Distributed Spatial Reasoning
Figure 4.5: Recognition, composition and intersection of qualitative positional relations.
the respective artifact that are possibly known by x already (i.e. contained in its self-description); however, it should be noted that intersection leads to meaningful results for JEPD relations only (cf. Section 3.2.3). In the case that no further relations can be inferred, no more self-descriptions are being broadcast and the distribution process terminates. The broadcast can be delayed for a certain amount of time such that it is not performed separately for each received self-description, but only once for several changes, which reduces the induced traffic as will be seen in Section 4.4. It should be noted that local communication policies of artifacts, which concern privacy and security aspects as discussed in [MOFH03] for example, may have an impact on the recognition and inference of relations but are not considered in this work. Figure 4.5 shows the algorithmic steps recognition, composition and intersection by means of a simple topology with four artifacts a, . . . d which are placed in
4.3 Spatial Relationship Inference and Distribution
95
two-dimensional space. Each vertex represents a digital artifact, edges between two vertices indicate that the respective artifacts are within communication range, and arrows represent broadcasts of self-descriptions. For the composition operation, a combined consideration of orientation and distance relations using the composition table of Table 4.5 is used. First, after an initial broadcast of artifact d, the artifacts a and c receive the self-description of d and recognize qualitative positional relations by comparing their own position with that contained in the received self-description. For example, artifact c recognizes that d is front and near, which results in the relation front(d,c) ∧ near(d,c). Subsequent broadcasts of the self-descriptions of a and c enable b and d to recognize their relations to the broadcasting artifacts also, as for example that c is right and near to b. Afterwards, in the second step of the algorithm, artifact b infers additional relations from the received and previously recognized ones. For example, b composes the relations front(d,c) ∧ near(d,c) (i.e. received from artifact c) and right(c,b) ∧ near(c,b) (i.e. recognized by a comparison with the position of artifact c), which results in a disjunction of the four relations front(d,b) ∧ near(d,b), front(d,b) ∧ medium-dist(d,b), right(d,b) ∧ near(d,b) and right(d,b) ∧ medium-dist(d,b). Another composition artifact b performs concerns the positional relations between d and a as well as those between a and b, which results in a further composed relation between d and b (cf. Figure 4.5). Artifact b finally intersects the two composition results, which leads to a disjunction of just two relations, namely front(d,b) ∧ near(d,b) and front(d,b) ∧ medium-dist(d,b). We distinguish three variants of the distribution algorithm, which differ in how the composition operation is implemented. The first one, which corresponds to the pseudocode of Algorithm 1 and has been used for the example in Figure 4.5, is a combined composition of positional relations. It produces the best results of all variants, namely those which constrain the resulting compound relations most, as will be shown by simulation in the next section. In the second variant, which is a small modification of the above algorithm, the composition of orientation and distance relations is performed separately by using dedicated composition tables for orientation (cf. Table 4.3) and distance (cf. Table 4.4). As we have discussed in Section 4.2.2 already, using separate composition tables for orientation and distance often leads to comparatively coarse results, in particular with regard to distance relations. On the other hand, the composition tables are much easier to create and smaller than in the combined case. The third and last variant processes the transitive closure of orientation relations, which means with respect to Algorithm 1 that just orientation relations are considered for the inference, and that the composed relations R and S are not compound relations and do have the same relation value at a time. This variant requires the least administrative and computing
96
4 Distributed Spatial Reasoning
efforts, but – for reasons mentioned in the previous section – provides the coarsest results. A comparison of these three variants and the flooding approach with respect to their accuracy and the induced traffic by means of different network topologies will be given in the following section. An early algorithm from the research field of qualitative spatial representation and reasoning, which has been introduced in [All83], computes the closure of a set of temporal interval relations among nodes every time a new relation is inserted; a detailed explanation of a corrected version of this algorithm can be found in [Her94]. It first intersects a possibly already existing with the new relation between two nodes i and j, and then computes the closure by propagating the effect of the new relation in the case it has changed. For this purpose, it checks whether the new relation can be used to constrain (i.e. to make them more specific) the relations between i and each other “comparable” node k as well as between each other node k and j by using composition table lookups, which in turn may lead to changed relations that are to be propagated; in this regard, propagation can be controlled by defining which nodes are “comparable” and which ones are not. The main difference to the algorithm proposed in this thesis is that the maintenance and inference of relations is centralized and not distributed across spontaneously interacting artifacts, which however is fundamental for their autonomous operation as we have argued in Section 2.2. Therefore, the recognition and maintenance of relations as well as the inference of new ones is conducted by the artifacts independently from each other, and it is solely based on their local perceptions and the exchange of them with other artifacts in communication range. According to the analysis of the closure algorithm presented in [All83], the upper bound of the operation time of our algorithm is also O(n2 ) for a fixed number of n artifacts, as at most n ∗ (n − 1) compound relations can hold between them, and each one consists of up to 12 positional base relations which cause a constant maximum number of 12 self-description changes per artifact.
4.4 Simulation Results The algorithm presented in Section 4.3 is an efficient means for providing spatially distributed digital artifacts with an awareness about qualitative spatial relations among each other. The key contribution in this regard is the inference of relations over multiple hops without the need for numeric calculations, given the availability of appropriate composition tables which allow for resource-efficient reasoning with simple table-lookups. In order to evaluate the algorithm and its implementation variants, we simulated it with different network topologies and
4.4 Simulation Results
97
varying numbers of artifacts by using the J-Sim simulation environment [Uni04]. In J-Sim, artifacts are implemented in Java as individual software components which communicate with each other via ports. This suits our work on autonomous digital artifacts well, which interact with their local neighborhood by exchanging self-descriptions. The Tcl script language is used in J-Sim for composing the components and thus defining the overall system behavior. The evaluation was done without taking into account certain wireless communication technologies or transmission protocols, just the logics of the algorithm have been simulated. The aim was to compare the implementation variants of our algorithm and the flooding approach both with regard to (i) their accuracy in terms of acquired qualitative relations after termination of the distribution process and (ii) the induced traffic which is generated by broadcasts of self-descriptions. We have implemented (i) the flooding variant, as well as the proposed algorithm with its three ways for composing spatial relations (cf. Section 4.3): (ii) transitive closure, (iii) separate composition and (iv) combined composition. Each one of the composed J-Sim components, which represents a digital artifact executing one of these algorithms, is able to recognize and infer qualitative relations to others, put the results to its self-description and exchange it with others via the communication ports. In this regard, the three composition tables have been fully implemented in order to allow for simulating arbitrary topologies. We only used positional (i.e. orientation and distance) relations, the algorithm is however applicable for other relation types also as long as composition tables are available or the relation properties are known. First, we investigated the spatial awareness which is achieved by artifacts during a whole relationship distribution process. We used the example topology with n = 5 artifacts shown in Figure 4.6 for it, and visualized the whole process step-by-step for the case that artifact b initially broadcasts its self-description and that combined composition is used. In the simulation, the five artifacts are independently executed components, which are connected via their ports as indicated with edges between vertices in the figure. The achieved spatial awareness for this topology after termination of the relationship distribution can be seen in Table 4.6, depending on the algorithm used in each case. In the following, we will have a closer look at the simulation results. With the flooding approach, each artifact gets to know the self-descriptions of all others in the network, and it is possible to compute exactly one base relation from the sensor data of each artifact. Flooding is thus the most accurate variant, resulting in n ∗ (n − 1) = 20 relations (i.e. 4 base relations per artifact), and it causes a total number of 50 received self-descriptions over all n artifacts. The transitive closure algorithm on the other hand is the least accurate one, mainly for two reasons. First, all three relations have to be the same for processing the transitive closure,
98
4 Distributed Spatial Reasoning
Figure 4.6: Compositional inference and distribution started at artifact b.
Artifact b R(a,b): medium-dist ∧ left R(c,b): near ∧ back R(d,b): near ∧ right R(e,b): medium-dist ∧ right R(a,b): medium-dist ∧ left R(c,b): near ∧ back R(d,b): near ∧ right R(e,b): right R(a,b): medium-dist ∧ left R(c,b): near ∧ back R(d,b): near ∧ right R(e,b): (near ∨ medium-dist) ∧ right R(a,b): medium-dist ∧ left R(c,b): near ∧ back R(d,b): near ∧ right R(e,b): (near∧right) ∨ (medium-dist∧right)
Artifact a
R(b,a): medium-dist ∧ right R(c,a): medium-dist ∧ right R(d,a): far ∧ right R(e,a): far ∧ right
R(b,a): medium-dist ∧ right R(c,a): – R(d,a): right R(e,a): right
R(b,a): medium-dist ∧ right R(c,a): right ∨ back R(d,a): right R(e,a): right
R(b,a): medium-dist ∧ right R(c,a): (near∧right) ∨ (medium-dist∧right) ∨ (far∧right) ∨ (near ∧ back) ∨ (medium-dist ∧ back) ∨ (far∧back) R(d,a): (medium-dist ∧ right) ∨ (far∧right) R(e,a): (medium-dist ∧ right) ∨ (far∧right)
Algorithm
Flooding
Transitive Closure
Separate Composition
Combined Composition R(a,c): (near∧left) ∨ (medium-dist∧left) ∨ (far∧left) ∨ (near ∧ front) ∨ (medium-dist ∧ front) ∨ (far ∧ front) R(b,c): near ∧ front R(d,c): near ∧ right R(e,c): (near∧right) ∨ (medium-dist∧right)
R(a,c): left ∨ front R(b,c): near ∧ front R(d,c): near ∧ right R(e,c): (near ∨ medium-dist) ∧ right
R(a,c): – R(b,c): near ∧ front R(d,c): near ∧ right R(e,c): right
R(a,c): medium-dist ∧ left R(b,c): near ∧ front R(d,c): near ∧ right R(e,c): medium-dist ∧ right
Artifact c
R(a,d): (medium-dist ∧ left) ∨ (far∧left) R(b,d): near ∧ left R(c,d): near ∧ left R(e,d): near ∧ right
R(a,d): left R(b,d): near ∧ left R(c,d): near ∧ left R(e,d): near ∧ right
R(a,d): left R(b,d): near ∧ left R(c,d): near ∧ left R(e,d): near ∧ right
R(a,d): far ∧ left R(b,d): near ∧ left R(c,d): near ∧ left R(e,d): near ∧ right
Artifact d
R(a,e): (medium-dist ∧ left) ∨ (far∧left) R(b,e): (near∧left) ∨ (medium-dist∧left) R(c,e): (near∧left) ∨ (medium-dist∧left) R(d,e): near ∧ left
R(a,e): left R(b,e): (near ∨ medium-dist) ∧ left R(c,e): (near ∨ medium-dist) ∧ left R(d,e): near ∧ left
R(a,e): left R(b,e): left R(c,e): left R(d,e): near ∧ left
R(a,e): far ∧ left R(b,e): medium-dist ∧ left R(c,e): medium-dist ∧ left R(d,e): near ∧ left
Artifact e
4.4 Simulation Results 99
Table 4.6: Simulation results for the topology of Figure 4.6.
100
4 Distributed Spatial Reasoning
for which reason e.g. artifact a is not able to infer any relation to artifact c and vice versa, as the orientation on the path from a to c changes (i.e. right(b,a) and back(c,b)). Second, the spatial relations have to be transitive, which is not the case for distances; thus, no distance relation can be inferred from the artifacts c, d and e to a as well as from e to b and c. It results in 58 base relations, as a missing distance or orientation relation corresponds to 3 or 4 base relations (i.e. the respective universal relations, cf. Section 4.1), and no positional relation at all (e.g. from c to a) corresponds to their product with 12 alternative positional relations. The simulation also showed that a traffic of 36 received self-descriptions is induced over all artifacts. With delayed broadcasts, which avoids multiple broadcasts due to successively received self-descriptions by performing them in short fixed intervals only, it could be reduced to just 25 self-descriptions, which is 50% compared with flooding. In both cases, the simulation was started once from each artifact, and the average of the resulting traffic was taken. Additionally, we investigated the relative accuracy achieved by a certain algorithm, which we define as the complement of the ratio between the (i) difference of the actual (i.e. 58) and the minimum (i.e. 5 ∗ 4 = 20) number of relations, and (ii) the difference of the maximum (i.e. 5 ∗ 4 ∗ 12 = 240) and the minimum number of relations. For the transitive closure algorithm, this is 1 − (58 − 20)/(240 − 20) = 83% compared with flooding. Taking into account the separate composition of orientation and distance using the composition Tables 4.3 and 4.4, more accurate conclusions can be drawn. For example, artifact a is now able to infer the two alternatively possible base relations right(c,a) and back(c,a), and artifact c is able to narrow down the possible distance relations to artifact e to near(e,c) and medium-dist(e,c). The implementation variant with separate composition results in a total number of 42 relations and induces a traffic of 38 self-descriptions without and 26 by using delayed broadcasts. The relative accuracy raises to 90% and the traffic for delayed broadcasts to 52%. Nevertheless, the composition of distances often results in the universal relation, which does not provide any information about the spatial distance between artifacts. With a combined composition of orientation and distance relations as shown in Table 4.5 however, the accuracy of the relations between some artifacts can be increased further. For example, while artifact a is only able to infer the relation right(e,a) due to the resulting universal relation by composing the distance relations mediumdist(b,a), near(d,b) and near(e,d) without taking into account orientations, their combined consideration allows to exclude the distance relation near(e,a). The total number of relations can eventually be refined to 38, the induced traffic is the same with 38 and 26 self-descriptions, respectively. The accuracy thus raises to
4.4 Simulation Results
101
92% and the traffic remains 52%, which means that a higher accuracy is achieved with the same traffic. We simulated the four implementation variants from above with the common network topologies full binary tree, line, mesh and ring, and for varying numbers of artifacts which are positioned as shown in Figure 4.7. The induced traffic for the different topologies and depending on the number of nodes can be seen in Figure 4.8, whereas several observations can be made. First, it can be seen that it is in any case reduced by using qualitative composition in comparison to flooding. Second, the transitive closure algorithm is always equal or below the traffic induced by a separate or combined composition of positional relations, which is due to the smaller number of inferred relations and thus the fewer broadcasts. Third, the induced traffic for separate and combined composition is quite different for the binary tree topology, whereas it is virtually the same for line, mesh and ring topologies. Fourth, using delayed broadcasts significantly reduces the traffic with an increasing number of nodes, for example to less than 40% for the mesh topology with 15 and the binary tree topology with 16 nodes. We also experimented with complete graphs and star topologies which lead to similar results.
Figure 4.7: Simulated topologies full binary tree, line, mesh and ring.
From the above evaluation it can be seen how many broadcasts are caused by the different algorithm implementations, especially in comparison with the flooding approach. In order to provide a group of connected artifacts with an awareness about their qualitative spatial relations among each other, flooding requires the highest amount of communication. However, by relying on qualitative inference between artifacts which are not directly connected, not all relations between them can be stated with certainty; instead, usually disjunctions of spatial relations are acquired between distant artifacts. In the following, the relative accuracies – i.e. with respect to the flooding approach – for the four topologies are finally shown in Figure 4.9, from which several things can be seen. As in the above example, 100% accuracy means that each artifact knows the exact base relations to all other artifacts, 0% means that just universal relations are known (i.e. all 12 positional
102
4 Distributed Spatial Reasoning
Figure 4.8: Traffic induced by different algorithms depending on the topology.
4.5 Findings and Discussion
103
Figure 4.9: Relative accuracy of different algorithms compared with flooding.
relations are possible between any pair of artifacts) and 50% means that just half of the alternative base relations which may exist between artifacts are possible and the others can be excluded. First, the transitive closure approach leads to the least accuracy, which is clear as it only supports a subset of the possible compositions. Second, the accuracy decreases with an increasing number of nodes, which is due to the coarser composition results coming along with the higher number of hops between artifacts. Third, the percentage-wise reduction of traffic is in all simulated scenarios higher than the percentage-wise loss of accuracy; for example, the induced traffic for the mesh-topology with 16 nodes drops to 27% in the case of separate composition, whereas the accuracy is reduced to 73% only.
4.5 Findings and Discussion In this chapter, we first discussed fundamentals of qualitative reasoning about binary spatial relations, and presented means for inferring new from known positional relations by (i) exploiting the properties reflexivity, symmetry and transitivity, as well as by using composition tables for (ii) a separate or (iii) a combined consideration of orientation and distance relations. As for many relation types none of the three properties is fulfilled and closures can be processed for equal relation values only, our focus was on compositional reasoning which allows for overcoming these restrictions. We proposed an algorithm for inferring and distributing relations among autonomously operating digital artifacts over multiple hops by taking advantage of those qualitative reasoning techniques, namely without numeric calculations or the need for exchanging spatial contexts between artifacts that are out of communication range. The aim was to provide a group
104
4 Distributed Spatial Reasoning
of artifacts – which are either connected directly or via other artifacts – with an awareness about their relations to each other; therefore, self-descriptions containing both quantitative spatial properties and qualitative spatial relations are shared between neighboring artifacts. In this regard, our focus was on static positional relations in 2-D between two stationary artifacts at a time, which are given with respect to an extrinsic reference frame. It should be noted that also other than positional relations could be used, given that the respective composition tables are available. The presented approach allows for resource-efficient reasoning with simple table lookups, and is thus – among other reasons for using qualitative relations mentioned in the previous chapter – valuable whenever relations between artifacts out of sensor- and communication-range have to be acquired, or the artifacts’ resources – in particular their computing and communication capabilities – are limited. However, a mentionable disadvantage is the time-consuming construction of composition tables as well as the necessity of their reconstruction upon changes in the number or semantics of the corresponding relations. We have implemented the algorithm variants depicted at the beginning of this subsection, and evaluated them in a Java-based simulation environment for different topologies and numbers of networked artifacts. Wireless communication aspects such as delays or packet losses as well as conflicting or inaccurate sensor data have been disregarded in the simulation, which we however consider interesting with respect to future work. Another issue of future work would be to extend the presented approach to moving artifacts with constantly changing relations between them, which is not covered in this thesis but can be important with regard to dynamic real-world scenarios. The simulation of our algorithm showed that the traffic induced by broadcast self-descriptions is significantly reduced compared to an exchange of self-descriptions between each pair of artifacts by means of flooding. However, especially between distant artifacts which are several hops away from each other, often just coarse results in the sense of alternatively possible relations can be acquired, which is also reflected by a decrease of the overall accuracy with an increasing number of artifacts in the simulation. In this regard, a combined composition of distance and orientation leads to more accurate results than their separate composition, which in turn provides more accurate results than exploiting just the transitivity property of binary relations. It should be noted that the proposed approach of inferring relations by utilizing qualitative spatial reasoning techniques is not only meaningful for acquiring relations among a group of connected artifacts as presented in this chapter, but also for “en passant” inference, meaning that an artifact which is already aware of its relations to others around shares this information with artifacts passing by.
5 Rule-Based Spatial Awareness All our knowledge begins with the senses, proceeds then to the understanding, and ends with reason. There is nothing higher than reason. Immanuel Kant, 1724-1804
In the preceding chapters, qualitative abstractions of spatial and spatiotemporal relations, as well as the inference of new relations from them by means of compositional reasoning, have been discussed within the scope of spontaneously interacting digital artifacts. They are able to acquire spatial properties characterizing their situation on the one hand and recognize relations by a pairwise comparison of their own with the spatial properties of artifacts in range on the other hand, which are – together with recognized and inferred relations – shared among them by using ad-hoc communication. However, what we have not considered by now are programmatic means for achieving spatially aware behavior of artifacts. Essential in this respect is their ability to reason about the history of qualitative relations they are aware of, for which purpose a rule-based solution is proposed in this chapter. First, in Section 5.1, a motivation for and an introduction in the use of rules for spatially aware digital artifacts are given. Afterwards, Section 5.2 presents our approach for rule-based reasoning about qualitative spatial relations, in particular the inference of new relations and the triggering of appropriate application-level actions as a consequence of observing certain defined patterns on the known relations. It builds on qualitative relations that have been recognized and abstracted depending on application demands, and allows for achieving spatially aware behavior using declarative programming. As will be seen in Chapter 6, rule-based reasoning constitutes an integral component of the Zones-of-Influence Framework; in this regard, it serves as a layer of abstraction between the low-level relationship recognition and the application programming layer. Finally, a proof-of-concept evaluation with a motion tracking system is provided in Section 5.3, and Section 5.4 summarizes the proposed approach and points out open issues for future work.
106
5 Rule-Based Spatial Awareness
5.1 Achieving Spatially Aware Behavior The focus of the present thesis is on the spatial awareness of autonomous embedded systems that are integrated into everyday physical artifacts, so as to be able to develop spatially aware applications which rely on the artifacts’ spatial relations to one another as well as on relationship changes over time. We have presented an architecture for spatial awareness in Section 2.5, whose core responsibilities are the recognition of spatial relations to artifacts in communication range and their qualitative abstraction for the application-level use, depending on application demands and the availability of required spatial information. The representation of qualitative static and dynamic spatial relations on the one hand, and their combined consideration with qualitative temporal relations between intervals in which they hold on the other hand, have been extensively discussed in Section 3.2. For achieving spatially aware behavior of digital artifacts, which means in particular that they are able to determine and use spatial relations among them, an important issue is their ability to reason – i.e. to draw conclusions – about those relations and their changes over time. It allows to make knowledge explicit for applications which is implicitly contained in the known relations and their semantics only, and thus contributes to the autonomous adaptation of artifacts to their spatial environment. In Chapter 4, we were concerned with compositional reasoning, which commonly uses predefined composition tables for inferring new from known relations. Although such a reasoning technique contributes to the development of spatially aware applications in that the number of relations which can be used is extended, it does not provide programmatic means for achieving spatially aware behavior. Hence, the upcoming issue is to find an appropriate software solution which supports programmers in defining the actions to be executed for certain combinations of relations. In this regard, our scope is on reactive systems whose output depends on the current and past states; proactive behavior as e.g. addressed in [May04] is not considered.
5.1.1 Reasoning with Rules In general, two programming paradigms can be distinguished: imperative and declarative programming [BIK+ 04]. While the former is concerned with how to do something, the latter allows for describing what to do. We decided for a declarative approach using rules in this thesis, which is for several reasons. First, a rulebased approach provides a higher degree of flexibility with regard to customizing the behavior of spontaneously interacting digital artifacts [BCFM03, TTH+ 04], as both the rules and the data on which they operate can change at runtime. The sec-
5.1 Achieving Spatially Aware Behavior
107
ond reason is that the utilization of rules allows for a more natural and intuitive modelling of the behavior of networked mobile devices, and is thus also suitable for non-technical users and domain experts [BCFM03, HR85]. A third reason for choosing a rule-based approach is the comparatively high speed and scalability of some rule engines, especially when the data does not change entirely between rule executions [PNF+ 08]; due to their abstraction from the more fluctuating lowlevel sensor readings, this is usually the case for qualitatively represented spatial relations. In the field of pervasive computing, rules have been used for behavior modelling in many projects. For example, [SAW94] distinguishes context-triggered actions as one of four categories of context-aware applications, and uses if-then rules to specify in the “contextual reminders” application which message should appear depending on different contexts such as time, location and the identity of people. Another example is given in [BGS01], where rules are applied to a history of contexts acquired from temperature and movement sensors in order to detect high-level situations such as “drinking out of the cup”. [BCFM03] and [TTH+ 04] use a rule-based approach for programming the behavior of mobile systems in a more natural way, and [SGKK04] takes advantage of rules both for inferring further based on existing knowledge – which can be built-in domain knowledge, observed or previously inferred knowledge – as well as for determining an artifact’s behavior. According to [BBB+ 07], two types of rules for programming reactive systems can be distinguished. The first one are ECA (Event-Condition-Action) rules, which have the structure on <event> if then and thus define which action (also referred to as consequence) is executed when the specified event occurs and the condition holds. The second type are production rules, which have the structure when then and specify the action to be executed when the condition becomes true due to an update on the database. Instead of events in the case of ECA rules, which makes them particularly suitable for developing distributed applications (like e.g. in [BCFM03, TTH+ 04]), production rule languages have the concept of a working memory containing data and an update statement for explicitly notifying about changes in data, which makes them the preferred choice for managing the state of a system and defining its behavior based on domain-specific knowledge [BBB+ 07]. As the focus of this work depicted in Section 1.1 is on making digital artifacts spatially aware about their environment, which means that they have to be able to maintain a spatial model, draw conclusions about it and adapt their behavior accordingly, we have chosen to use production rules and refer to them simply as rules in the following.
108
5 Rule-Based Spatial Awareness
Figure 5.1: Rule-based reasoning [PNF+ 08].
With regard to this thesis, the data are qualitative spatial or spatiotemporal relations, and actions which result from fulfilled conditions on that data are primarily the inference of new high-level relations or the triggering of application-level actions. This allows applications to adapt to their spatial environment and thus exhibit spatially aware behavior, which is presented and evaluated in the Sections 5.2 and 5.3, respectively. In order to be able to use production rules however, a rule engine software is required, which matches data against rules and infers conclusions that result in corresponding pre-defined actions. For a better understanding of rule-based spatial awareness discussed in detail afterwards, the next subsection gives a short overview on the fundamentals of rule engines.
5.1.2 Using a Rule Engine Due to our choice for production rules, we use a so-called production rule system for reasoning about spatial relations, which is often simply referred to as rule engine. As can be seen in Figure 5.1, it basically consists of a knowledge base which contains both rules (i.e. the logic) and facts (i.e. the data), and an inference engine that infers conclusions from them which result in certain actions [HR85]. The rules are constructed by a human domain expert and stored in the production memory of the rule engine, and facts are inserted in the working memory where they can be retracted (i.e. removed) or modified [PNF+ 08]. Rules consist of a conditional part which specifies patterns on the data using first order logic, as well as a consequence part which defines one or more actions that are executed when the condition is fulfilled; in that case, the rule is said to be fired.
5.2 Rule-Based Qualitative Spatial Reasoning
109
The process of matching facts against rules is called pattern matching, and it is performed by the inference engine which can be regarded as the core component of the rule engine. Facts are evaluated for matches against rules upon insertion in the working memory, and added to the so-called agenda if they match; the actual execution of the rules however, which is performed for each combination of facts for which they are fulfilled, has to be triggered explicitly. The agenda is responsible for resolving conflicts in the case that multiple rules are scheduled for execution; for this purpose, it defines an execution order of the rules, which may modify the working memory and thus possibly cause some other rules to be removed from the agenda for example. An important aspect of rule-based systems is that rules can be changed without recompilation of program code, which allows for adapting the artifact behavior at runtime. Within the scope of this thesis we decided for using the Java-based rule engine JBoss Drools 4.0 [PNF+ 08], whereby the facts stored in the working memory are Java objects representing qualitative relations with their intervals. It uses an extension of the Rete algorithm for pattern matching, which minimizes the number of patterns to be evaluated by compiling the rule conditions into a network [BBB+ 07], and thus scales to large numbers of rules and facts.
5.2 Rule-Based Qualitative Spatial Reasoning In Section 5.1, we have pointed out the importance of reasoning about qualitative spatial relations over time in order to achieve spatially aware behavior, and argued for the use of production rules which provide declarative means for defining the actions to be executed for certain combinations of relations. The proposed rule-based approach we have published in [Hol07b] builds on a repository of qualitatively abstracted relations (i.e. the rule engine’s working memory), which have been – according to the architecture for spatial awareness presented in Section 2.5 – either (i) recognized by a pairwise comparison of Zones-of-Influence, (ii) received with the self-description of a digital artifact in range or (iii) inferred from already known relations, and it serves as a layer of abstraction for the development of spatially aware applications. As rules are applied to the relations repository for detecting arbitrary patterns that can be formulated with first order logic, and executing any actions when the specified patterns have been found, they can serve a number of different purposes. For achieving spatial awareness, we distinguish the following two types of rules: • Maintenance rules: perform maintenance operations on the repository, which can result in modifications or retractions of relations
110
5 Rule-Based Spatial Awareness
• Inference rules: infer relationships or application-level actions depending on specified patterns of relations observed in the repository In the following, these two types of rules and their use with the rule engine are presented, whereas a detailed discussion on inference rules – which actually define an artifact’s spatially aware behavior – will be given in Section 5.2.1. An important aspect that has to be considered first however is the timing concerning (i) the recognition of relations, (ii) an update of the repository therewith and (iii) the subsequent execution of rules. The starting point is the recognition of relations, for which we distinguish two variants: • Periodic recognition of all relations, for which reason each Zone-of-Influence of an artifact has to be compared with all other ZoIs it is aware of • Event-based recognition for each changed or new Zone-of-Influence, which has to be done just for these ZoIs upon notification Afterwards, the repository is updated with the recognized relations, which can be a modification of the time intervals of existing or an insertion of newly recognized relations. The actual execution of rules – i.e. of the actions in the consequence parts of rules that match – has to be triggered explicitly, which is done immediately after updating the repository. For the purpose of a first feasibility study of our rule-based approach by using two Zones-of-Influence in Section 5.3, all relations are periodically recognized and inserted in the repository, whereas a relation that is recognized at time t is inserted with the interval [t,t] – meaning that it holds at time t – in the repository; all rules are fired afterwards. With this approach, previously received Zones-of-Influence or recognized relations do not have to be considered for the insertion. In order to allow for temporal reasoning, namely to relate the time intervals of relations – e.g. in order to recognize that one relation holds during another one – as discussed in Section 3.2.5, it is necessary that equal relations at successive discrete points in time (e.g. three relations with the intervals [t,t], [t + 1,t + 2] and [t + 3,t + 3]) are merged to a single relation with a correspondingly extended time interval (i.e. [t,t + 3] in this example). It should be noted that this is based on the assumption that the respective relation also holds between those points in time where no information is available. A maintenance rule is used for merging such relations which belong together. It checks the repository for pairs of qualitative relations with the same name, value and identifications of the reference and primary Zone-of-Influence, whose intervals are not in an after or before relation to each other (cf. Figure 3.11), as just relations that either meet or overlap to a certain extent have to be merged. For each such pair, it replaces the interval of one relation with the union of the two intervals
5.2 Rule-Based Qualitative Spatial Reasoning
111
(i.e. its interval is extended) and removes the other relation from the repository. For the above example of a relation inserted at time t, this means that if there already exists an equal relation at the preceding point in time, its interval is extended to the end-point t and the other relation is removed. Another maintenance rule could be defined for limiting the length of the relations history for example, simply by removing relations whose end times range too far (i.e. greater than a defined number of discrete points in time) in the past. Listing 5.1: Maintenance rule for merging equal relations. rule " merge - equal - relations " when r1:Relation ( n:name , v:value , p Z:pZoI , rZ:rZoI , i1:interval ) r2:Relation ( name == n , value == v , pZoI == pZ , rZoI == rZ , i2:interval ) eval (! r1 . equals ( r2 )) // r1 and r2 are different relations eval (! i1 . before ( i2 ) && ! i1 . after ( i2 )) // i1 not after or before i2 then r1 . setInterval ( i1 . merge ( i2 )); up date ( r1 ); // extend interval of r1 retract ( r2 ); // remove relation r2 from the repository
Listing 5.1 shows the maintenance rule for merging equal relations, which is given in the syntax of the Drools rule engine [PNF+ 08]; however, it should be noted that other production rule systems could also be used. The rule’s condition (when clause) specifies a pattern on the stored relations; in this example, it is an implicit and combination of two qualitative relations as well as two eval statements containing Java code that returns a boolean value. Relation is a Java class, whose member variables (i.e. referred to as fields in [PNF+ 08]) – in this case the name and value of the relation, its primary and reference ZoI as well as the time interval in which the relation holds – can be used for constraints on field names of objects of that type. Two relation objects that match the given field constraints are bound to the variables r1 and r2, which are used in the first eval statement to ensure that different objects are compared. The fields of the first relation are bound to variables (e.g. v for the relation value) and used to constrain the value of the other relation (i.e. value==v in this example); thus, r1 and r2 must have the same name, value and ZoI identifications. The second eval statement limits a match to those relation pairs whose intervals are not in a temporal after or before relation. For each combination of Relation facts for which the condition – i.e. all four conditional elements – is fulfilled, the rule’s consequence (then clause) is executed. In the example of Listing 5.1, it extends the interval of relation r1 by setting it to the minimum start time and the maximum end time of the intervals i1 and i2 (method merge), and notifies the rule engine about this change with a subsequent update statement. Finally, r2 is retracted from the repository. Similar
112
5 Rule-Based Spatial Awareness
to the rule shown in Listing 5.1, other maintenance but also inference rules can be described. An important aspect is that the time required for pattern matching depends on how often facts change, as it is performed by the rule engine every time a new fact is inserted, modified or retracted. In the above example of a periodic recognition, all relations are recognized and inserted at each point in time. For the implementation of the Zones-of-Influence Framework discussed in Chapter 6 however, we used the more efficient event-based variant, where relationship recognition is performed just for changed and new Zones-of-Influence. Also the repository is updated with changed and new qualitative relations only, which together reduces the number of inserted and modified relation facts. The rule execution is performed immediately after the repository update, that is periodically at discrete points in time. For detecting those changes however, it is necessary to remember the Zonesof-Influence as well as relations previously recognized from them. With regard to time efficiency, it should be noted that the rule engine remembers past matches for facts that have not changed [PNF+ 08].
5.2.1 Inferring Relations and Application-Level Actions The rule-based approach described above is primarily used for inferring conclusions from spatial relations, which allows for defining the behavior of artifacts depending on their spatial environment. This comprises (i) the inference of relations based on known ones, which extends an artifact’s spatial awareness by making application-relevant knowledge available that is implicitly contained in the relations, as well as (ii) the inference of application-level actions, which are executed once the respective rule’s condition is fulfilled; a similar distinction can be found in [SGKK04]. We use so-called inference rules in both cases, which specify patterns on qualitative relations in their conditional parts. The basis for the first case – namely the inference of relations – is the representation of spatiotemporal relations presented in Section 3.2.5, where we discussed logical combinations of qualitative spatial and temporal relations as well as their interpretation as high-level spatiotemporal relations which exist in corresponding time intervals; in this regard, a distinct feature of our work is that such inferred qualitative relations can in turn be used like any other relation in the repository. We distinguish four general categories of inferred relations, depending on (i) if static or dynamic relations are combined and (ii) whether the represented situation relates to a single point in time or a time series of changing spatial relations. A classification of these four categories is given in Table 5.1 with two examples
5.2 Rule-Based Qualitative Spatial Reasoning
113
Table 5.1: Combinations of qualitative spatial and temporal relations. dynamic spatial relations
point in time
static spatial relations
oriented toward each other
approaching from the right
turning away to the right
moving around counterclockwise
passing by right
changing direction of movement
turning around for a while
time series
near and on the left hand side
at a time, showing situations of spatial relations between two Zones-of-Influence which are denoted as primary object p and reference object r in the following. First, if just spatial and no temporal relations are combined, the spatial situation at a certain point in time can be recognized. The combination in the respective inference rule’s condition could be e.g. front(p,r) ∧ opposite-dir(p,r) for the static spatial situation “oriented toward each other”, and leftward(p,r) ∧ toward(p,r) for the dynamic situation “approaching from the right”. As spatial relations are internally stored with time intervals, an additional combination of the relations’ intervals has to be considered, namely that they are not in a temporal after, before, meets or met-by relation to each other. Second, if a time series of changing relations is considered, both spatial as well as temporal relations between them have to be combined, where the respective combinations are for example back(p,r)[s1 ,e1 ] ∧ right(p,r)[s2 ,e2 ] ∧ front(p,r)[s3 ,e3 ] ∧ meets([s1 ,e1 ], [s2 ,e2 ]) ∧ meets([s2 ,e2 ], [s3 ,e3 ]) for the static situation “passing by right”, and away(p,r)[s1 ,e1 ] ∧ toward(p,r)[s2 ,e2 ] ∧ meets([s1 ,e1 ], [s2 ,e2 ]) for the dynamic situation “changing direction of movement”. In this regard, the subscripts of the primary object p in Table 5.1 indicate the discrete points in time with the respective spatial position or direction. Although these examples are fairly simple, more sophisticated combinations can of course be described. It would also be possible to define quantitative relations be-
114
5 Rule-Based Spatial Awareness
tween the start and end points s1 and e1 of intervals, for example to describe the situation “turning around for a while”, which however is not considered further as our focus is on qualitative temporal relations. In order to infer new from combinations of known relations as described above, the inference rule’s consequence part contains the insertion of the new relation in the repository, with a name and value corresponding to its semantics, an identification of the involved Zones-of-Influence as well as the time interval in which it holds. For the example in Table 5.1, this means that if ZoI p is both in the relations front and opposite-dir to r within a certain period of time, the resulting interval of the new relation oriented-toward-each-other(p,r) is the intersection of the intervals of the two relations due to the logical conjunction of the respective inference rule’s conditions (cf. Section 3.2.5); such new relations can eventually be used in the conditional parts of other rules. Listing 5.2 shows the rule for inferring the relation passing-by-right, which recognizes a sequence of static orientation relations between the same two Zonesof-Influence using temporal meets relations between the intervals in which the relations back, right and front hold. For reasons of simplicity, the relation names (i.e. their types) are disregarded in the following rule conditions, as we assume their values to be unique. A respective new relation is created in the consequence part of the rule and inserted in the repository. In the example of Listing 5.2, the name of the relation is InferredRelation, its value is passing-by-right and the resulting interval is a merge of the intervals i1, i2 and i3 as has been done in Listing 5.1 already. Listing 5.2: Inference rule for the relation passing-by-right. rule " passing - by - right " when Relation ( value == " back " , pZ:pZoI , rZ:rZoI , i1:interval ) Relation ( value == " right " , pZoI == pZ , rZoI == rZ , i2:interval ) Relation ( value == " front " , pZoI == pZ , rZoI == rZ , i3:interval ) eval ( i1 . meets ( i2 ) && i2 . meets ( i3 )) // temporal relations then insert ( new Q ua l it at i ve Re la t io n ( " InferredRelation " , " passing - by - right " , pZ , rZ , i1 . merge ( i2 . merge ( i3 ))));
An inferred can be used like any other relation, as for example in the condition of another rule. This is exemplarily shown with the rule in Listing 5.3, which combines a relation passing-by-right – i.e. generated by the previous rule – with a disjunction of the spatial distance relations near and medium-dist. A new relation passing-by-right-in-vicinity is inserted in the repository, whose interval equals the intersection of the two combined relations’ intervals.
5.3 Proof of Concept
115
In addition, such inference rules can also be used for triggering applicationlevel actions depending on observed patterns in the repository, which are encoded in the consequence part of the rule instead of or in addition to inserting a new relation. Examples for that have been published in [HHF08, HFH+ 08] and will be given in Chapter 7. Listing 5.3: Inference rule for the relation passing-by-right-in-vicinity. rule " passing - by - right - in - vicinity " when Relation ( value == " passing - by - right " , pZ:pZoI , rZ:rZoI , i1:interval ) Relation ( value == " near " || value == " medium - dist " , pZoI == pZ , rZoI == rZ , i2:interval ) eval (! i1 . before ( i2 ) && ! i1 . after ( i2 ) && ! i1 . meets ( i2 ) && ! i1 . metBy ( i2 )) then insert ( new Q u a l i t a t i v e R e l a t i o n ( " I n f e r r e d R e l a t i o n " , " passing - by - right - in - vicinity " , pZ , rZ , i1 . intersect ( i2 ))); end
5.3 Proof of Concept In order to evaluate the rule-based approach for spatial awareness presented in the previous sections in a real-world setting, we have implemented a simple prototype which uses physical sensors for tracking the position and direction of two objects. The aim of this evaluation was to show that (i) different types of static and dynamic qualitative spatial relations as discussed in Section 3.2.3 can reliably be recognized from sensor data, and (ii) that rules are a feasible technique for inferring new qualitative relations from them. The IS-900 motion tracking system from Intersense Inc. [Int08] mentioned in Section 3.1.2 has been used for it, which provides Cartesian coordinates (x,y,z) for position and Euler angles (yaw,pitch,roll) for the direction of tracked devices within a room, both given with respect to the local reference frame of the tracking system. Those trackers contain accelerometers and gyroscopes for calculating position and direction from linear and angular accelerations, and send them – together with signals received from ceiling-mounted ultrasonic strips – to an IS-900 base processor in the background via wireless communication, from which the position and direction of up to four trackers can eventually be accessed. We have chosen the IS-900 system for this evaluation because of the fact that it provides highly accurate spatial information in 6-DOF on the one hand (i.e. 3mm − 5mm for position and 0, 50°−1, 00° for direction), and that the tracked devices can be operated without cabling and thus allow for full freedom of movement on the other hand.
116
5 Rule-Based Spatial Awareness
Figure 5.2: Acquisition of position and direction from two IS-900 trackers.
For recognizing spatial relations over time, the position and direction of a primary object p and a reference object r are given by two such trackers, where each of them is connected to a wireless transmitter as shown in Figure 5.2. They are represented with point-shaped Zones-of-Influence, topological relations are not considered in this evaluation. While object r is placed stationary at the table, p is moved by hand as indicated with the dotted course of motion in the figure. The implemented prototype acquires position (given in meters) and direction (given in a range of ±180°) of the two trackers from the IS-900 processor periodically every 100ms, whereas the mean value of 10 such sensor readings per tracker is used to get smoother and less fluctuating data. For this reason, position and direction acquired from the two trackers are compared once every second, and quantitative static as well as dynamic positional and directional relations are processed and abstracted to qualitative relations. A 2-D projection of the tracked room is used, which means that the z-axis for position as well as pitch and roll for direction are disregarded in the following. It should be noted that the length of the interval between two relationship recognitions depends on the frequency of relation changes which is of relevance for the application. Figure 5.3 shows time series of the three types of quantitative static and dynamic spatial relations over 17 points in time, starting at t = 0 with a recognition interval of Δt = 1s. For this evaluation, dynamic relations are estimated by comparing the corresponding two successive static relation values, for which reason they are recognized first at t = 1 where the quantitative values of t = 0 and t = 1 are compared. First, it can be seen from the figure that the Euclidean static distance relations between p and r are decreasing and increasing again, and the dynamic distance relations – which represent distance changes at the respective points in
5.3 Proof of Concept
117
Figure 5.3: Trajectories of quantitative spatial relations.
time – are consequently negative in the former and positive in the latter case. Second, the static orientation of p with respect to the direction axis of r (i.e. intrinsic orientation) is first negative, which means that p starts its motion left behind r, moves to right behind r from t = 6 to t = 7 then (which results in a jump of the orientation angle) and finally passes r on its right hand side. A dynamic orientation relation represents the direction in which p is moving with respect to r, and it is calculated by determining the orientation of the position of p at the current relative to that at the preceding point in time, again with respect to the intrinsic direction of r. Third and last, static directional relations represent the differences between the direction axes of p and r, which results in positive and negative directional relations if p is directed away and toward r, respectively. The dynamic are simply the differences between two successive static directional relations, thus leading to a trajectory similar to that of the corresponding static relations.
118
5 Rule-Based Spatial Awareness
The recognition of quantitative relations described above is usually the first step followed by their qualitative abstraction, which is considered in the Zones-ofInfluence Framework by allowing qualitative relation recognizers to access results from quantitative ones (cf. Chapter 6). For this evaluation, the quantitative static and dynamic spatial relations are abstracted as shown in the Figures 3.6 and 3.8. The qualitative distances near, medium-dist and far are defined as distances less than 0, 30m, between 0, 30m and 0, 60m and greater than 0, 60m, respectively. In order to represent the absence of motion, the dynamic orientation, distance and directional relations stable-orient, stable-dist and stable-dir are used. The recognized qualitative spatial relations are finally – i.e. once per second in this experiment – inserted in the working memory of the Drools rule engine [PNF+ 08], where they can be queried and used for inferring high-level relations or triggering application-level actions; the time series of these qualitative relations can be seen in Figure 5.4. Due to the continuous movement with object p, no stable dynamic relations are recognized. An important aspect regarding the qualitative abstraction is the epsilon-neighborhood introduced in Section 3.2.3, which is necessary to avoid frequent relation changes at the transition from one to a neighboring qualitative static relation. They occur without actual movement but just because of the tracking system’s inaccuracies, and have to be taken into account depending on the semantics of the respective relation. In Figure 5.4 for example, there is no change from the qualitative relation medium-dist to near at t = 6 although the quantitative relation is below the threshold of 0, 30m, because the quantitative range of each distance relation is extended with epsilon = 0,05m. Similarly, qualitative dynamic relations do not change unless the quantitative difference between two successive points in time is sufficiently high. From the qualitative spatial relations in the working memory, new relations can be inferred using rule-based reasoning as described in Section 5.2. First, the relation passing-by-right is inferred with the rule of Listing 5.2. Every second, after recognizing the static and dynamic qualitative spatial relations, all rules are fired and the respective actions – in this case the insertion of the new passing-by-right relation – are executed. The relation is inferred first and inserted in the repository at t = 10, and at the points t = 11..16 its interval is extended by the maintenance rule of Listing 5.1. Due to using the minimum and maximum of the start- and endpoints of the three orientation relations back, front and right (cf. Section 3.2.5), the interval of the new relation is [0, 16] at the end (cf. Figure 5.4). Second, the inference rule passing-by-right-in-vicinity shown in Listing 5.3 uses the relation passing-by-right in its condition, and it results in a respective new relation with the interval [5, 13], in which the relations passing-by-right(p,r) and either near(p,r) or medium-dist(p,r) exist, in the working memory of the rule engine. Such inferred
5.4 Summary and Open Issues
119
Figure 5.4: Recognized and inferred qualitative relations.
relations constitute – together with those which are recognized by a comparison of Zones-of-Influence – the spatial relationship awareness of a digital artifact, and they can in turn be used in other rules.
5.4 Summary and Open Issues We have proposed a rule-based approach for reasoning about qualitative spatial and temporal relations, which constitutes a core component of the Zones-ofInfluence Framework presented in the next chapter. First, we argued for using rules to achieve spatial awareness, which is mainly due to their higher flexibility with regard to describing an artifact’s behavior, the more intuitive way of programming and the comparatively high speed and scalability by using a rule engine. In the field of pervasive computing, and especially with regard to context-aware applica-
120
5 Rule-Based Spatial Awareness
tions, the use of rules is widespread and can thus be considered as an established approach. We opted for production rules, and explained in detail the inference of conclusions from qualitative spatial relations therewith. We discussed examples of maintenance and in particular of inference rules, which are used for inferring both new relations and application-level actions, and presented a distinction of four categories of spatial situations that can be described with inference rules. These categories depend on whether static or dynamic relations are combined and if the represented situation relates to a single point in time or a time series of changing spatial relations. As a proof of concept, we have implemented a prototype which periodically recognizes and infers relations from position and direction information of a 6-DOF motion tracking system, and showed that our rule-based approach works reliably for recognizing and inferring different types of qualitative spatial and spatiotemporal relations. There are several open issues that have not been addressed yet. As will be seen in Chapter 7, the use of rules in connection with qualitatively abstracted spatial relations allows for developing spatially aware applications with less effort and in a more natural way. In this regard, it would be of interest to compare this declarative with an imperative approach for dealing with space, both with respect to the required storage and processing resources and concerning the application development time. A related open point is a user interface support for constructing rules, e.g. by using a graphical user interface which facilitates the process of selecting and combining relations to rules. The second issue is the use of rules with compound relations that comprise multiple alternatively possible base relations, like for example the combined orientation and distance relations we have used for compositional reasoning in Section 4.2.2. Although inference rules could be used to implement composition tables by inserting the respective relations in the repository for each combination of observed base relations, it is an open issue how to meaningfully use them for inferring high-level relations and application-level actions, as such compound relations are represented with multiple relations which may but need not hold. Another potentially problematic aspect in connection with inferred spatiotemporal relations is the fact that their intervals may range in the past and thus potentially cause actions to become invalid which have already been executed.
6 Zones-of-Influence Framework To me, the primary motivation behind the information appliance is clear: simplicity. Design the tool to fit the task so well that the tool becomes a part of the task, feeling like a natural extension of the person. This is the essence of the information appliance. Donald Norman, 1935-
In this chapter we present the Zones-of-Influence Framework, a software solution for enabling spatial awareness of autonomous embedded systems. It is based on the concepts presented in previous chapters, in particular the qualitative abstraction of spatial relations and the rule-based reasoning about them, and has been designed to facilitate the development of spatially aware applications for autonomous and spontaneously interacting systems. First, Section 6.1 presents design considerations for and an overview of the framework architecture, which has been developed with flexibility, modularity, efficiency and scalability in mind. It is split into components for the discovery of artifacts and the exchange of self-descriptions with them, the maintenance of a Zones-of-Influence model with the spatial properties of discovered artifacts, the application-dependent recognition and qualitative abstraction of spatial relations as well as the rule-based reasoning about qualitative relations over time in order to infer new relations or trigger according applicationlevel actions. An overview of these components – which provide spatial awareness to the application level – is given in Section 6.2, including a detailed description of their interfaces and relevant implementation-specific issues. The results of performance tests are presented in Section 6.3, which have been carried out in order to evaluate the runtime efficiency and scalability of the framework implementation; a qualitative evaluation by means of multiple application scenarios will be given afterwards in Chapter 7. Finally, Section 6.4 discusses features and properties of the framework and identifies open issues for future work. The development of the Zones-of-Influence Framework architecture and its implementation have been conducted in an industrial research project [FDE+ 08] with Siemens AG Germany.
122
6 Zones-of-Influence Framework
6.1 Architecture For the development of the framework architecture, several requirements had to be considered. They were derived from the general aim of this work, namely to make autonomous embedded systems – which are mobile, have to operate without a centralized instance, are potentially limited in their computational and communication capabilities and have to be able to incorporate physical sensors and actuators – aware about abstracted spatial relations over time, and will be discussed in the following subsection. Afterwards, the architecture of the Zones-of-Influence Framework is presented in Section 6.1.2, which has been developed according to these design considerations and builds on the general architecture for spatial awareness already presented in Section 2.5. Before going into implementation details in Section 6.2, the runtime platform – which has been used for it – is presented at the end of this section.
6.1.1 Design Considerations The Zones-of-Influence Framework should not only enable spatial awareness of spontaneously interacting autonomous embedded systems, which we have defined as a system’s awareness about its own spatial properties as well as about spatial relations to other systems, but it should also facilitate the development of spatially aware applications. We proposed a general architecture for achieving spatial awareness in Section 2.5, which consists of multiple layers through which lowlevel spatial context information is processed until it reaches the application. As we have mentioned in Section 2.1 already, layered architectures are used in many context-aware embedded systems – as for example in [GSB02, HBB02, HKG+ 05, SGKK04] – and can thus be considered as a common practice for building such systems. The layers of this previously discussed general architecture determine the functional requirements for the Zones-of-Influence Framework, which serves as a middleware between the spatial sensors and the communication module on the one hand and the application layer for running spatially aware applications on the other hand. These requirements are basically (i) the acquisition and representation of spatial contexts from one or multiple local sensors or by exchanging self-descriptions with other spatially aware artifacts, (ii) the application-dependent recognition and qualitative abstraction of spatial relations by comparing two Zones-of-Influence at a time, (iii) the maintenance of a spatial model of the environment which consists of recognized and inferred qualitative relations as well as (iv) the declarative rule-based approach for reasoning about them over time, as has been discussed in
6.1 Architecture
123
the Chapters 3 to 5. Although no general limitation, the functional scope within the present thesis is on processing the spatial properties position, direction and physical extension (cf. Section 3.1), and recognizing corresponding qualitative positional, directional and topological relations from them (cf. Section 3.2). The framework should enable the development of arbitrary spatially aware applications, and allow for dynamically adapting the incorporation of spatial sensors, the availability and semantics of spatial relations as well as the rules for reasoning about them according to the current application demands. Besides the functions the Zones-of-Influence Framework should provide, we identified several non-functional requirements for the design of its architecture, including but not limited to the following: • Flexibility with respect to the utilized spatial contexts, which means that it should be possible to exploit the spatial properties of arbitrary sensors with different reference systems, and to recognize, represent and reason about corresponding spatial relations • Modularity in order to allow for exchanging components by defined interfaces and facilitate the development of different spatially aware applications due to a clear separation of concerns • Efficiency of the implemented components concerning both processing and storage requirements, which contributes to their operation on resource-constrained embedded system platforms • Scalability with regard to the number of represented Zones-of-Influence, the maintained spatial or spatiotemporal relations and the rules for reasoning about them
6.1.2 Architecture Overview In this subsection, the architecture of the Zones-of-Influence Framework is presented, which is shown in Figure 6.1 and has been published in [HFH+ 08]. It consists of several components which cover different responsibilities, provide respective services to other components and use event-based communication for the exchange of data among them. In the following, their functional principles are explained in short, a detailed description of the components and their actual implementations will be given in Section 6.2. It should be noted that the current implementation is in a prototypical stage in that our focus was on the recognition and representation of qualitative relations from Zones-of-Influence contained in
124
6 Zones-of-Influence Framework
Figure 6.1: Overview of the Zones-of-Influence Framework architecture.
self-descriptions as well as means for reasoning about them, functionalities beyond that have been implemented only as far as required for evaluation purposes. The base component is the Digital Artifact Service, whose main functions are the discovery of artifacts and the exchange of self-descriptions with them, which contain – among others – information about their Zones-of-Influence and qualitative relations they are aware of (cf. Section 3.3); therefore, it also has to manage the local artifact’s self-description. The currently available implementation however just allows for a manual generation of self-descriptions, the discovery and communication functionalities have no been implemented yet. The Zones-of-Influence Service builds on the Digital Artifact Service and maintains an up-to-date model with the Zones-of-Influence of discovered artifacts. It registers as a listener with the Digital Artifact Service to get notified about new or changed self-descriptions, parses such self-descriptions for contained Zones-
6.1 Architecture
125
of-Influence upon notification and keeps them up-to-date (e.g. due to receiving a new self-description with changed spatial properties of an already known ZoI). For ZoIs which refer to dynamic sensor data, which means that the self-descriptions in which they are represented do not directly contain all of their spatial properties but sensor data references instead specify where they can be acquired from (cf. Section 3.3.1), the Zones-of-Influence Service continuously fetches the missing data with corresponding sensor data accessors and updates the ZoIs accordingly. An example of such a sensor is the IS-900 motion tracking system from Intersense Inc. [Int08], which has been used for evaluation purposes in this thesis. The Visualization Service visualizes currently maintained Zones-of-Influence, for which purpose it registers as a listener for new, changed or removed ZoIs with the Zonesof-Influence Service. An idea for future work that has not been addressed by now is to make the scenes – they determine which ZoIs are visualized and how – exchangeable, which is comparable to the runtime-adaptation of so-called themes in the peripheral display framework presented in [FEST06, Fer07a]. With regard to this thesis, the Relations Service is the heart of the Zones-ofInfluence Framework, as it is responsible for recognizing, representing and reasoning about qualitative relations. It allows for plugging in modules to recognize application-specific quantitative or qualitative spatial relations at runtime, which therefore register with the Zones-of-Influence Service as listeners for new, changed or removed ZoIs. Due to this dynamic adaptation, it is possible to recognize just those relations which are required by running applications and therewith save computing resources. In this regard, qualitative recognizers can register for changed relations with quantitative ones, which allows for separating qualitative abstractions from the quantitative relationship recognition and thus for using a single quantitative by multiple qualitative recognizers. Recognized qualitative relations are inserted in the relations repository, which is the working memory of the JBoss Drools 4.0 [PNF+ 08] rule engine in the current framework implementation (cf. Figure 5.1). For reasons of efficiency, relationship recognition is performed for changed or new Zones-of-Influence only, and qualitative relations are inserted or modified just in the case they are new or have changed (cf. Section 5.2). Applications can register with the Relations Service to get notified about relation changes or directly access the relations repository. The rule engine is used for inferring relations or application-level actions depending on the known relations stored in the repository. Additionally, relations can also be received with self-descriptions of artifacts in range, for which purpose the Relations Service registers as listener for new or changed self-descriptions with the Digital Artifact Service; however, this functionality has not been implemented yet.
126
6 Zones-of-Influence Framework
Two more components are shown in the architecture of Figure 6.1, but also not available at the time of this writing. The first one is the Rules Service, which encapsulates the production memory of the rule engine (cf. Figure 5.1) and provides means for managing rules – i.e. removing, modifying or adding them – at runtime. In the current implementation, rules are stored in a single file which is accessed from the Relations Service. For this reason, they cannot be modified at runtime, which however was not required for the evaluation. Secondly, the Query Service serves as an interface for applications to access spatial information; however, query techniques are beyond the scope of this thesis. In the case that the Drools rule engine is used, the Query Service could for example use both the Rules Service and the Relations Service in order to deploy so-called query rules and retrieve the respective results. Query rules are dedicated rules which have no consequence but just a conditional part, and their results – i.e. bound variables in the rule conditions – can be iterated by referring to the name of the query [PNF+ 08].
6.1.3 Runtime Platform For the implementation of the Zones-of-Influence Framework we decided to use the Java-based OSGi framework as runtime platform in order to meet the requirements discussed in Section 6.1.1. It implements a dynamic component model, in which the components (i.e. referred to as bundles) can be installed and managed at runtime and cooperate in a single Java Virtual Machine, and it has been used in a series of domains ranging from applications for mobile phones to the Eclipse IDE (cf. [All07]). The open-source Equinox [Ecl08] implementation of the OSGi R4 framework specification [All05] has been developed for the latter, and it also serves as a platform for our middleware framework. The OSGi framework is divided into several layers shown in Figure 6.2, which are in detail explained in [All05]. On top of the Java execution environment (e.g. J2SE or J2ME), the module layer defines a modularization model which provides means for sharing and hiding packages between bundles. The life-cycle layer provides a runtime model for managing the bundles in the module layer, and a corresponding API which allows for starting, stopping, installing, updating and uninstalling them at runtime. This is actually the most prominent feature, and it allows for changing the available services at runtime. The service layer provides a programming model for the development of bundles, which is based on a decoupling of their implementations and interfaces; bundles can dynamically register services with their interfaces, discover existing services or receive notifications when their registration state changes. The OSGi framework thus provides a service-oriented architecture that supports loosely-coupled application design, whereas an OSGi
6.2 Components
127
Figure 6.2: Layered architecture of the OSGi framework, after [All05, All07].
service is any Java object of a bundle which is made available to other bundles [All07]. The security layer extends the built-in security model of Java 2 with a strict separation of the bundles from each other. Using OSGi as runtime platform contributes to the ability of embedded systems to dynamically adapt their behavior, and it is thus valuable for their operation in changing environments. Moreover, such a service-oriented design is resource efficient as just services which are required by a running application have to be loaded and different applications can access the same service which therefore has to be loaded only once. From a software development viewpoint, using OSGi facilitates the integration of existing software and allows for developing small, loosely coupled components that can be composed to larger systems at runtime [All07]; however, it should be mentioned that OSGi is not mandatory for the Zones-ofInfluence Framework, but facilitates its implementation and deployment.
6.2 Components In this section, the interfaces and implementation details of the core framework components – i.e. the Digital Artifact Service, the Zones-of-Influence Service and the Relations Service – are presented, with the aim to clarify their internal structures and the interplay with other components.
6.2.1 Digital Artifact Service The Digital Artifact Service implements the communication component of the general architecture proposed in Section 2.5 and it is responsible for discovering artifacts in communication range and exchanging self-descriptions with them, which contain – among other things – information about their spatial contexts.
128
6 Zones-of-Influence Framework
It thus provides the basis for the recognition of spatial relations, which relies on the knowledge of the Zones-of-Influence of surrounding artifacts to which the relations should be calculated. The functionalities of the Digital Artifact Service are basically (i) the notification about discovered and no longer available artifacts as well as about changed self-descriptions of available artifacts, (ii) the selfdescription exchange with discovered artifacts as well as (iii) the management of the artifact’s local self-description at runtime (e.g. adding and removing Zones-ofInfluence). Figure 6.3 shows the main Java interfaces of the Digital Artifact Service component and their dependencies among each other; for the sake of a better understandability, not all methods contained in the respective interfaces are shown. The DigitalArtifactService interface provides methods for registering and unregistering DigitalArtifactAvailabilityListener objects, and notifies registered listeners about discovered or no longer available artifacts using the callback methods daAvailable or daUnavailable, respectively. Digital artifacts are represented with objects of type DigitalArtifact, whereas implementations of the DigitalArtifactService interface internally maintain a list of available artifacts that can be retrieved with the method getAvailableDigitalArtifacts. How artifacts are discovered depends on the actual DigitalArtifactService implementation, for example by broadcasting advertisements and taking advantage of timeouts as realized in [FHdSR+ 07] and [HKG+ 05]. The interface DigitalArtifact is used for retrieving an artifact’s self-description with the method getDADescription and for registering and unregistering as DADescriptionChangeListener in order to get notified whenever its self-description changes. The updateDADescription methods with and without parameter are called from the DigitalArtifactService to update the self-description of an artifact in the former or to notify about an already performed update in the latter case. The Zones-of-Influence Service as well as the Relations Service, which will be presented in the Sections 6.2.2 and 6.2.3, register both as DigitalArtifactAvailabilityListener with each Digital Artifact Service and as DADescriptionChangeListener with each available digital artifact to be notified about new and changed self-descriptions. As presented in Section 3.3, the XML-based self-description of an artifact may contain several elements of type DADescriptionElement, where each Zoneof-Influence or qualitative relation is represented by one such element. A selfdescription is encapsulated in a Java class DADescription which provides means for parsing its XML representation as well as for modifying and accessing the contents; every time the Digital Artifact Service discovers a new artifact, it adds a DigitalArtifact object with its self-description of type DADescription to the
6.2 Components
129
Figure 6.3: Interfaces of the Digital Artifact Service.
list of known artifacts. The self-description elements contained therein are represented with objects implementing the DADescriptionElement interface shown in Figure 6.3, which allows for retrieving an identification of the element type and the corresponding XML document. In order to be able to parse elements of different types, the DADescription class makes use of the factory design pattern. The DigitalArtifactService interface allows for registering and unregistering DADescriptionElementFactory objects, as for example by the Zonesof-Influence Service for parsing XML elements of type ZoI and SensorData (cf. Section 3.3.1) and creating DADescriptionElement objects which represent one Zone-of-Influence at a time; more details on that will be given in Section 6.2.2. By retrieving the corresponding factory with the DigitalArtifactService interface, the DADescription instances are able to parse all kinds of XML elements for which types respective factories have been registered, and maintain the generated DADescriptionElement objects in an internal data structure to be accessed by other services. As mentioned in the previous section already, just prototypical implementations for testing purposes which allow for a manual generation of self-descriptions are available at the time of this writing, but the discovery of artifacts and the communication between them have not been implemented yet. A potential system for that would be the Java-based Peer-it hardware/software platform proposed in [FHdSR+ 07], which could be integrated in our modular Zones-of-Influence Framework. Due to the loose coupling of components, multiple Digital Artifact Service instances at a time are possible, for example to support different communication technologies. Our first implementation of the DigitalArtifactService interface provides a GUI for loading self-descriptions from XML files, generates DigitalArtifact objects out of them and notifies registered listeners about the availability of new artifacts. The second implementation additionally serves as a Visualization Service, and thus both visualizes currently known Zones-of-
130
6 Zones-of-Influence Framework
Influence in 2-D and allows for interactively creating new and modifying existing ones; some screenshots will be seen later in Chapter 7. It therefore registers as a listener for new, changed and removed ZoIs with the Zones-of-Influence Service using the ZoIAvailabilityListener and ZoIChangeListener interface (cf. Section 6.2.2), generates DigitalArtifact instances corresponding to interactively created ZoIs and changes the self-descriptions of artifacts whose ZoIs have been modified. Registered DigitalArtifactAvailabilityListener and DADescriptionChangeListener objects are notified accordingly, in particular the Zones-of-Influence Service which thereupon updates its list of maintained Zones-of-Influence and in turn notifies registered listeners – including the interactive Visualization Service – about those that changed.
6.2.2 Zones-of-Influence Service The Zones-of-Influence Service builds on the Digital Artifact Service. It serves as an intermediary layer which collects the local (i.e. of the artifact which runs the framework) and remote (i.e. of artifacts in range) self-descriptions, extracts ZoI representations from them and provides an up-to-date model of all currently known Zones-of-Influence for the recognition of qualitative relations. In the case of dynamic Zones-of-Influence where one or more spatial properties are determined by sensor values, a retrieval of data from the respective sensors is required and will be explained later on. The Zones-of-Influence are internally maintained as a list of ZoI objects, which can be accessed via the ZoIService interface with several getter methods such as getZoIs for retrieving the whole list or getZoIsOfArtifact for retrieving all Zones-of-Influence belonging to a certain artifact; these two and related Java interfaces are shown in Figure 6.4. The ZoIService implementation also implements the interfaces DigitalArtifactAvailabilityListener and DADescriptionChangeListener, which contain callback methods for notifying about the availability of artifacts and changes of their self-descriptions (cf. Section 6.2.1), and is therewith able to constantly update the list of ZoIs accordingly. Whenever a digital artifact is notified to be available, the method daAvailable of the ZoIService implementation is called with a respective DigitalArtifact instance. It extracts all DADescriptionElement entries of type ZoI contained in the self-description, and adds the respective ZoI objects to its local list from which they are removed again in the method daUnavailable when the artifact becomes unavailable. As described in Section 6.2.1 already, the Zones-of-Influence Service registers a specialized factory of type DADescriptionElementFactory with the Digital Artifact Service for that, which is thus able to parse self-descriptions containing Zones-of-Influence, generate a ZoIDADescriptionElement object for
6.2 Components
131
Figure 6.4: Interfaces of the Zones-of-Influence Service.
each one and add them to the DADescription instance of the respective artifact. As can be seen in the ZoIDADescriptionElement interface of Figure 6.4, a respective ZoI object can be acquired with the method getZoI. In the case that a self-description has changed, the callback method daDescriptionChanged of the ZoIService implementation checks for changed Zones-of-Influence of the corresponding artifact and updates its list of ZoI objects accordingly. In order to notify other services like the Relations Service about available and no longer available Zones-of-Influence, the ZoIService interface provides methods for registering and unregistering ZoIAvailabilityListener objects which implement the callback methods zoiAvailable and zoiUnavailable. Each Zone-of-Influence is represented with an object that implements the ZoI interface shown in Figure 6.4. It contains methods for retrieving the name of the ZoI with getName, its full name (i.e. a colon-separated concatenation of the artifact’s name as well as the name of the ZoI which is unique per artifact) with getFullName as well as means for retrieving its position with getPosition, its direction with getDirection and a list containing one or more shapes with the method getShapes. In the current implementation of the framework, a Zone-ofInfluence comprises one position, one direction and one shape (cf. Section 3.3.1). For notifications about changes of a certain Zone-of-Influence, the ZoI interface provides the possibility to register and unregister as a ZoIChangeListener, which comprises the methods zoiChanged and zoiDescriptionChanged that are called for all registered listeners upon changed spatial properties (e.g. its dynamic position) or when its structure has changed (e.g. a second shape has been added),
132
6 Zones-of-Influence Framework
respectively. The ZoIChangeListener interface is again implemented by the Relations Service and the Visualization Service, which register as listeners once a new Zone-of-Influence is notified to be available. As presented in Section 3.3.1, different ZoI types – depending on (i) the reference systems for position and direction, (ii) the used shape types and (iii) if they rely on dynamic sensor data or not – are distinguished. These aspects are all handled by the DADescriptionElementFactory implementation the Zonesof-Influence Service registers with the Digital Artifact Service in order to obtain ZoIDADescriptionElement instances for all XML elements of type ZoI. Thus, the Zones-of-Influence Service can easily be extended with new reference systems, shapes and sensors, which meets the flexibility requirement with regard to incorporating different types of spatial context information. During parsing, the factory generates ZoI objects with each one containing a position ZoIPosition, a direction ZoIDirection as well as a list of ZoIShape shapes. In order to support different reference systems for position and direction, the ZoIService interface provides methods for registering and unregistering specialized ZoIPosition and ZoIDirection instances such as ZoIPositionXYZ for Cartesian coordinates and ZoIDirectionEuler for Euler angles, which are selected by their identification in the refSystem attribute of the respective XML element (cf. Section 3.3.1) and provide a method newInstance that generates a corresponding position or direction instance from its content. For the evaluation in Chapter 7, ZoIPositionXYZ and ZoIDirectionEuler have been used to represent position and direction information provided by the IS-900 motion tracking system; technical details about it can be found in Section 5.3. Similarly, a ZoIShapeFactory for different shapes such as Rectangle2D for generating rectangular two-dimensional Zones-of-Influence can be registered and unregistered with the ZoIService, and they are selected by their identification with the type attribute of the XML element Shape. As described in Section 3.3.1, the optional position and direction of a shape are given relative to that of the corresponding ZoI; in this case, the respective ZoIShape contains a ZoIPosition and ZoIDirection instance, which must have the same reference systems for position and direction. At the time of this writing, shapes for three-dimensional cubes and spheres as well as for two-dimensional ellipses, points, rectangles, triangles and freeform shapes are available. These Java interfaces and classes used for representing ZoIs are shown in Figure 6.5 with exemplary implementations for the position, direction and shape. In the case of dynamic Zones-of-Influence, namely those for which the spatial properties are determined by sensor values, the source from where the sensor readings can be acquired has to be included in the respective self-description.
6.2 Components
Figure 6.5: Interfaces and classes for representing Zones-of-Influence.
133
134
6 Zones-of-Influence Framework
This may require e.g. a direct hardware access in the case of local sensors, or a wireless connection to an artifact for retrieving a sensor data stream. In the remainder of this subsection, it is explained how ZoI objects maintained by the Zones-of-Influence Service are updated with dynamic data retrieved from so-called sensor data accessors. As can be seen in Figure 6.4, ZoI extends the interface SensorDataAccessorChangeListener containing a method valueChanged, which is invoked whenever a dynamic value of a Zone-of-Influence changes and notifies all registered ZoIChangeListener instances that the ZoI has changed. The spatial properties of a ZoI – which can be e.g. Cartesian coordinates (x,y,z), Euler angles (yaw,pitch,roll) or width and height of the two-dimensional rectangle – are therefore represented as objects of type DataValue. When creating an instance of a Cartesian position of type ZoIPositionXYZ for example, this class performs the actual parsing of the x, y and z components by creating DataValue objects for each of them and handing over the contents of the XML elements X, Y and Z of the artifact’s self-description (cf. Section 3.3.1), respectively. For each spatial property which references a sensor data accessor (e.g. with @PosDir.x as it is the case in Listing 3.3), the respective DataValue object checks the self-description for a suitable DADescriptionElement entry of type SensorData such as the one shown in Listing 6.1, and retrieves the corresponding SensorDataAccessor with which the ZoI instance is registered as change listener. The accessor component is defined by the XML attribute accessor; it is responsible for fetching the sensor value whenever it is requested (i.e. by calling the DataValue method getAsDouble), for what a respective factory must have been registered with the DigitalArtifactService instance. The elements SvcName, Param and UpdateInterval specify where the sensor data can be acquired from and how, which is encapsulated in a dedicated OSGi bundle. The element Values contains the actually provided sensor values, which are Cartesian coordinates and Euler angles with respect to the reference system of the IS-900 motion tracking system. If a sensor provides multiple values as a Comma Separated Values (CSV) string, they can be distinguished in the XML description by defining the separation tokens (e.g. semicolons) and assigning identifiers (e.g. x) to the values; parsing is done in the SensorDataAccessor implementation. Thus, every time the IS-900 sensor data accessor detects a change, the callback method valueChanged of registered ZoI instances is invoked; the new values of e.g. the position are acquired with the method getAsDouble from the DataValue interface. Listing 6.1: Specification of sensor data within a self-description. < DADescriptionElement type = " SensorData " name = " PosDir " accessor = " OSGiSens o r D a t a A c c e s s o r " > < SvcName > at . jku . pervasive . IS 9 00 Tr a ck er Se r vi ce
6.2 Components
135
< Param > COM1 < UpdateInterval > 10 < Values > <x type = " token " token = " ; " nr = " 0 " / > < yaw type = " token " token = " ; " nr = " 3 " / > < pitch type = " token " token = " ; " nr = " 4 " / > < roll type = " token " token = " ; " nr = " 5 " / >
6.2.3 Relations Service The Relations Service is the framework component which provides an awareness about qualitative spatial relations to spatially aware applications. It recognizes relations by a pairwise comparison of Zones-of-Influence, and reasons about them using the rule-based approach presented in Chapter 5. The realization of this component builds on the experience gained with the proof-of-concept implementation described in Section 5.3; thus, it integrates the Drools rule engine [PNF+ 08], and uses its working memory as repository for the history of recognized and inferred qualitative relations. In order to be able to recognize certain types of relations between Zones-of-Influence that are relevant for an application, the RelationsService interface allows for registering and unregistering RelationRecognizer objects at runtime which are explained below in more detail. For notifying applications about new or no longer existing relations, it provides means for registering and unregistering listeners which implement the RelationRecognizerListener interface to get notified either about all relations the Relations Service is aware of, or with an additional String value specifying the name of a certain recognizer implementation to get notified about relations of the corresponding type only. In the Sentient Computing project [ACH+ 01] which we have in short discussed in Section 2.6.1, a similar event-based approach for notifying applications about recognized containment relations has been used. An overview of important classes and interfaces of the Relations Service can be seen in Figure 6.6, again showing the most relevant public methods only. In Section 3.2, we discussed many different types of qualitative relations that can be recognized, depending on the application demands on the one hand and the availability of the required spatial data on the other hand. Each relation of a certain type – e.g. static orientation or dynamic distance – is represented with an object of type Relation and recognized with a dedicated recognizer which extends the class RelationRecognizer. It has a unique name representing the relation type it recognizes, and provides methods for registering and unregister-
136
6 Zones-of-Influence Framework
ing as RelationRecognizerListener in order to notify applications or other services about relations that start or end. Additionally, the method configure provides means for configuring the recognizer with specific key/value properties. Upon registration of a recognizer via the RelationsService interface, the Relations Service registers itself as RelationRecognizerListener with the recognizer. This in turn allows to notify other listeners, which are registered with the Relations Service as RelationRecognizerListener, about starting and ending relations. Once a recognizer has been registered with the RelationsService, it is first initialized with the initializeRelationRecognizer method. As mentioned in Section 5.2, relationship recognition is done event-based in that it is performed for changed and new Zones-of-Influence only, for which reason it first registers with the ZoIService as ZoIAvailabilityListener (cf. Section 6.2.2). When a new ZoI is notified to be available, the recognizer checks if (i) the required spatial properties – e.g. position and direction in the case of static intrinsic orientation relations – are contained and (ii) the reference system – e.g. Cartesian coordinates for position and Euler angles for direction – can be handled with this recognizer implementation. Additionally, a recognizer can also limit the recognition to certain identifications of ZoIs and artifacts, as for example to consider relations to the local artifact only (i.e. the one which performs the recognition). It then registers as ZoIChangeListener with this new Zone-of-Influence, iterates over all known ZoI objects to calculate the respective relations to the new one and notifies registered RelationRecognizerListener objects about started relations. Correspondingly, if a ZoI is no longer available, registered listeners are being notified about the ended relations. In the case a ZoI changes, all relations with this ZoI are reprocessed, and for each qualitative relation that changes registered listeners are notified that the previous one is no longer available (method relationEnds) and that the new qualitative relation is now available (method relationStarts). For detecting such changes, the current relations are stored locally in each recognizer. During the initialization of the recognizer, it fetches all available ZoI objects from the Zones-of-Influence Service and recognizes relations between them by explicitly calling its zoiAvailable callback method for each of them. As already mentioned in Section 5.3, often quantitative relations are recognized followed by a qualitative abstraction step. We thus distinguish between quantitative relation recognizers and qualitative relation recognizers, whereas qualitative recognizers typically build on quantitative ones. Relation recognizers such as the StaticOrientationRecognizer shown in Figure 6.6 can implement the RelationRecognizerListener interface and register with a corresponding quantitative recognizer such as the QuantitativeOrientationRecognizer to get
6.2 Components
137
Figure 6.6: Interfaces and classes of the Relations Service.
notified about quantitative relations. This provides a high degree of flexibility, as the actual recognition and the qualitative abstraction are separated from each other and can thus easily be exchanged, and because of the fact that multiple qualitative relation recognizers can access the same quantitative recognizer for perform-
138
6 Zones-of-Influence Framework
ing different qualitative abstractions. The RelationRecognizerListener interface hence contains an additional method relationChanged for notifying about changed numerical relation values. Quantitative relation recognizers that are notified about the change of a ZoI reprocess all relations containing this ZoI and notify registered listeners about the changed relation using the callback method relationChanged. A concept which is in some respects similar to our relations recognizers are the widgets of the Context Toolkit [SDA99] briefly mentioned in Section 2.6.1. Quantitative and qualitative recognizers produce QuantitativeRelation and QualitativeRelation objects; for the two relation recognizers which are shown in Figure 6.6, these are objects of type QuantitativeOrientationRelation and StaticOrientationRelation, respectively. They both extend the class Relation, which provides a series of getter and setter methods for accessing the primary and reference ZoI between which a relation is defined, the name and value of the relation as well as the time interval in which it holds. Besides these examples of quantitative and qualitative orientation recognizers, we have implemented a series of recognizers and corresponding Relation classes. In addition to those already used for the proof-of-concept implementation in Section 5.3, recognizers for static topological relations as well as for static distance relations (i) between the anchor point of the primary to the shapes of the reference ZoI and (ii) between the shapes of two ZoIs have been implemented; for the latter two, the minimum distances are taken. It should be noted that, at the time of writing this thesis, fragmented Zones-of-Influence (i.e. ZoIs comprising more than one shape) are not supported by any of the available relation recognizers. According to Section 5.3, dynamic qualitative relation recognizers register with corresponding quantitative ones and process dynamic relations by comparing two successive quantitative relations at a time. Qualitative recognizers have been implemented to be parameterizable with the configure method, which allows for passing key/value pairs specifying the qualitative relation values and their respective quantitative ranges. Time intervals are represented using the class Interval, which contains all necessary methods for recognizing the 13 qualitative temporal interval relations discussed in Section 3.2.5, as well as additional ones for intersecting and merging intervals, accessing the interval’s start and end time and checking if the relation ends with the current discrete point in time (method isCurrent), among others. Recognized qualitative relations are inserted in the relations repository for the purpose of being used by applications or other recognizers as well as for the inference of relations or application-level actions with rules. As relations are inserted with their intervals, the repository maintains a history of qualitative relations. The insertion and modification of relations takes place at discrete points in
6.2 Components
139
time which can be adjusted with setUpdateInterval in the RelationsService interface, and it is performed in a separate update thread. As mentioned in Section 5.2, the repository is updated for changed and new qualitative relations only, which is due to the fact that the Relations Service registers with each recognizer as RelationRecognizerListener. Once it is notified about a started or ended relation, the RelationsService instance in turn notifies registered listeners and additionally adds it to a queue for started and ended relations, respectively. Thereby, equal relations (i.e. with the same name, value and between the same ZoIs) that both started and ended within one period are disregarded, as it may otherwise lead to inconsistencies when multiple relations of the same type but with different values hold at the same point in time. According to the defined update interval, the repository is periodically updated with the started and ended qualitative relations. This means, that (i) all new relations – with their start- and end-times set to the current discrete time, which can be retrieved with getCurrentTime from the RelationsService interface – are inserted in the repository, (ii) the end-times of all ended relations are set to the preceding discrete point in time (cf. Section 3.2.5) and (iii) the end-times of all other relations in the repository, which have not ended yet, are set to the current time (i.e. their intervals are extended). Afterwards, all rules are executed, the queues are cleared for the next update step and the current discrete time is incremented. In Section 4.3, we discussed the inference of relations over multiple hops using composition tables. A precondition is the exchange of qualitative relations between artifacts with self-descriptions (cf. Section 3.3.2) which in principle could be supported by the Zones-of-Influence Framework, but has not been readily implemented at the time of this writing yet and is thus subject of future work. Similar to the Zones-of-Influence Service, the Relations Service would have to register as DAAvailabilityListener and DADescriptionChangeListener with each Digital Artifact Service and available artifact respectively, and register a factory for parsing specialized DADescriptionElement objects representing qualitative relations with the Digital Artifact Service. Relations contained in received selfdescriptions could then be inserted in the repository, where further relations are inferred by rule sets which implement certain composition tables (cf. Section 4.2.2). In this regard, one possibility would be to realize each entry in the table with a dedicated rule, which checks for the existence of the respective two relations and inserts one or more resulting relations in the repository. In order to share knowledge about qualitative relations with surrounding artifacts, the DigitalArtifactService implementation has to access the repository and include the current relations in its local self-description.
140
6 Zones-of-Influence Framework
6.3 Runtime Behavior We have carried out several basic performance tests for evaluating the runtime behavior of the Zones-of-Influence Framework. Although the implementation is still in a prototypical stage, the results should provide a first estimation of its runtime behavior with regard to efficiency and scalability. All tests have been carried out on a Windows XP notebook (Intel T2700 dual-core CPU with 2.33GHz and 2GB RAM). The first test shows how much processing time is needed for acquiring an upto-date model with the Zones-of-Influence of discovered artifacts. A Digital Artifact Service has been implemented for this purpose, which generates XML documents for self-descriptions with a varying number of Zones-of-Influence, creates DADescription instances out of it and notifies the Zones-of-Influence Service – which is registered as DigitalArtifactAvailabilityListener – about new artifacts containing such self-descriptions. The Zones-of-Influence Service registers as DADescriptionChangeListener with each new DigitalArtifact object in order to get notified about changed self-descriptions. The used 2-D Zonesof-Influence are generated with random values and comprise a Cartesian position (0 . . . 5m for x and y), an Euler direction (0 . . . 360° for yaw) as well as a single 2-D rectangular shape (0 . . . 2m for both width and height) at a time. Table 6.1 shows the measured times required by the Digital Artifact Service for (i) creating new DigitalArtifact instances from the XML-based self-descriptions and (ii) updating them with new random self-descriptions afterwards, which results in respective notifications of the Zones-of-Influence Service as described in Section 6.2. The times depend on the number of artifacts and whether they contain one or ten ZoIs in their self-descriptions; however, the measured times are similar for a small number of artifacts with a correspondingly huge number of ZoIs each. Up to 100000 ZoIs have been considered, whereas the time required for generating the random XML documents has not been taken into account. As can be seen, the measured time for building up a model of ZoI objects from self-descriptions grows with linear complexity with the total number of Zones-of-Influence. However, their update time grows polynomial, which is due to the fact that the Zones-ofInfluence Service has to check the ZoIs in each updated self-description against existing ones for new or missing ZoIs as well as for ZoIs with changed structure or values. The second performance test was designed to show how the framework scales with the number of relations and rules. We extended the setting of the first test by additionally starting the Relations Service, which registers with the Zones-ofInfluence Service as ZoIAvailabilityListener and as ZoIChangeListener
6.3 Runtime Behavior
141
Table 6.1: Time in [s] for adding new artifacts/updating their self-descriptions.
artifacts time (1 ZoI) time (10 ZoIs)
1
10
100
1000
10000
0,02/0,0 0,02/0,0
0,02/0,03 0,05/0,09
0,08/0,16 0,29/0,23
0,45/0,63 1,75/3,56
3,23/51,58 17,69/>60
with each new ZoI object. We then generated such random self-descriptions with just one Zone-of-Influence – comprising position, direction and a single rectangular shape – and for a varying number of artifacts, and measured the elapsed time until the Zones-of-Influence Service built up the model of ZoI objects and notified the Relations Service about the new Zones-of-Influence, which in turn recognized all possible relations between them. We registered the following recognizers for qualitative static spatial relations as defined in Section 3.2.3: (i) a quantitative and qualitative static orientation recognizer for the qualitative orientation relations front, right, back and left, (ii) a quantitative and qualitative static distance recognizer for the qualitative distance relations near, medium-dist and far between the anchor points of the related Zones-of-Influence as well as (iii) a topology recognizer for the qualitative topological relations disjoint, overlaps, inside, contains and equal between their rectangular shapes. For each relation type, these are 20, 2450 and 39800 recognized relations in the case of 5, 50 and 200 Zones-of-Influence, respectively. The upper diagram of Figure 6.7 shows the required time from parsing to relationship recognition and the lower one the time for updating the relations repository therewith, whereas no rules were used in this first run. Approximately half of the update time is used for the actual insertion of new relations in the repository, and the other half for extending the intervals of existing relations. As can be seen from the diagrams, the time for recognizing about 30000 relations (i.e. relations of all three types among 100 artifacts) and updating the repository for about 5000 relations (i.e. relations of two types among 50 artifacts) takes approximately one second. Finally, we tested the repository update time for varying numbers of artifacts and rules. The recognizers for qualitative orientation, distance and topological relations as well as the random self-descriptions from above have been used again; thus, the recognition times are equal to those of Figure 6.7. Figure 6.8 shows the required time for updating the repository with all relations that can exist between a certain number of artifacts with one Zone-of-Influence each and executing all rules which infer further relations; for the case of 3 relation types, 50 artifacts and 100 rules, 3 ∗ 50 ∗ 49 = 7350 relations have been recognized and 33310 have been inferred during our test run, leading to a total number of 40660 relations as can be
142
6 Zones-of-Influence Framework
Figure 6.7: Time for relationship recognition (top) and repository update.
seen in the lower diagram of Figure 6.8. We evaluated the framework with 2, 5, 20, 50 and 100 inference rules. The first five rules contain different combinations of all three types of relations and insertions of the inferred relations in the repository, the next five rules use these inferred relations in their conditional parts and again infer new relations, and the remaining 90 relations are copied from the first 10 where just the names of the inferred relations have been changed such that no two equal rules insert or use the same inferred relation. In the case of conjunctions of two or more relations, their intervals are compared to check if they both hold at the same time, and the intersection of these intervals is used for the inferred relation’s interval. Within the investigated ranges, the required time for updating the relations repository and given a certain number of rules grows polynomial with the number of artifacts, which is because of the quadratically increasing number of recognized relations between artifacts in the repository.
6.4 Discussion
143
Figure 6.8: Repository update time (top) and involved number of relations.
6.4 Discussion The presented Zones-of-Influence Framework uses quantitative and qualitative abstractions of space in order to support the development of spatially aware applications for autonomous and spontaneously interacting systems. Thereby, Zonesof-Influence represent spatial regions that are associated with artifacts and qualitatively abstracted relations between two ZoIs at a time represent an artifact’s spatial situation with respect to its environment, both as far as relevant for their application-level behavior. We thoroughly described a first prototypical implementation of the framework, which implements the concepts for spatial awareness presented in the Chapters 3 to 5, and thus provides means for developing and experimenting with applications that are based on the use of qualitatively abstracted spatial or spatiotemporal relations. In this regard, our framework facilitates the application development due to the flexible use of qualitative relationship abstractions as well as their declarative combination for describing the application-level
144
6 Zones-of-Influence Framework
behavior. As will be discussed in the next chapter, we have realized different spatially aware applications with the Zones-of-Influence Framework, by means of which the proposed concepts and the framework itself have been evaluated. In the following, a series of open issues for future work are presented, which – in some cases – require a modification or extension of the proposed concepts. The first one concerns the sensing and representation of spatial properties, which usually requires to take into account sensor inaccuracies that lead to fluctuating values as we have mentioned in Section 3.2.3. This has to be considered by the relation recognizer, which has to know a sensor’s accuracy in advance at the moment; however, a more flexible solution would be to include this information in the sensor accessor specification of an artifact’s self-description, and calculate a respective epsilon value at runtime. A related issue concerns the reference systems of sensor values, which must be known by the relation recognizers in order to interpret the spatial properties of the Zones-of-Influence to be related. However, it would be possible to use logical sensor components which transform sensor values of a certain spatial property between different reference systems, as for example between Cartesian coordinates and spherical polar coordinates for the property position. Another aspect is the use of three-dimensional Zones-of-Influence and the recognition of relations from them, which is supported by the framework in that it is extensible with arbitrary XML-representations of ZoIs and allows for recognizing relations between them by using appropriate recognizers; however, it has not been evaluated yet with respect to the framework performance compared to two-dimensional representations, and it seems to be challenging especially with regard to representing and recognizing relations from three-dimensional freeform ZoIs. Similarly, also dynamic spatial properties such as linear or angular acceleration have not been considered by now, but they could in principle be used by the framework. There are also open issues regarding the recognition of spatial relations. First, it is not clear yet how to represent compound qualitative relations, which are disjunctions of alternatively possible base relations (cf. Chapter 4). One possibility would be to store multiple base relations between two Zones-of-Influence in the repository and interpret them as a compound relation; however, this complicates rule-based reasoning as the relations in the repository need not hold but are in fact simply possible alternatives. Second, the qualification of relations – i.e. which recognizer is used – can be configured by the spatially aware applications at runtime. This means, that an application which detects a context-change – e.g. a changing location, which may require to change the used sensors or abstract certain qualitative relations differently – has to dynamically load the appropriate relation recognizers. In this regard, it would be worth to investigate how the provision of
6.4 Discussion
145
appropriate recognizers could automatically be adapted to the current context of use, and thus unburden applications from having to deal therewith. Another issue is the extension of relation intervals with every update step, which keeps the time stamps up-to-date but consumes a lot of processing power (cf. Section 6.3). In future work, this could be improved by updating the end-times of intervals not until the relation is notified to have ended, which reduces the repository update time as (i) no relations have to be extended and thus (ii) the number of relationship changes and corresponding rule executions are reduced. For some applications it might be useful if multiple relations of the same type could hold between two Zones-of-Influence at the same time, as for example in the case of a curved table which may be both in the relations front and left to a certain person. Apart from the fact that we are not aware of how relevant this is from a practical viewpoint, it is not clear how to represent and reason about such relations, particularly in combination with compound relations. Another issue of future work is the consideration of relations between more than two Zones-ofInfluence at a time (e.g. the ternary relation between), which results in relations with more than one primary ZoI. From our viewpoint, especially the use of relations between different numbers of Zones-of-Influence in the inference process is challenging, but worth to investigate as the incorporation of non-binary relations would allow for dealing with more complex spatial situations. Besides relations among single Zones-of-Influence, the recognition and representation of relations between ZoI groups – e.g. crowds of people or lines of cars which are in certain spatial relations to each other – could be practically useful, but has not been investigated yet at the time of writing this thesis. The final block of open issues concerns the state of the framework implementation, in particular improvements regarding its efficiency and extensions with further components that are not available yet (cf. Section 6.1.2). One of them is the Digital Artifact Service, which has just been implemented for testing purposes yet, but without the capabilities of discovering artifacts and exchanging self-descriptions with them. Due to the modular design of the Zones-of-Influence Framework, it should be quite easy to port it to an existing hardware/software platform which implements this functionalities, as for example the Peer-it system mentioned in Section 2.6.1. The implementations of the Rules Service, which allows to dynamically change rules at runtime, as well as the Query Service, which serves as an interface for accessing spatial information by applications, are also subjects of future work.
7 Framework Evaluation What we have to learn to do, we learn by doing. Aristotle, 384 BC - 322 BC
After having described the Zones-of-Influence Framework in its current prototypical stage, and shown with performance tests that it runs stable and how it behaves under extreme conditions, this chapter presents a qualitative evaluation of the overall framework by means of several application scenarios. They have been selected to investigate different aspects of the current implementation, and therewith prove the feasibility and quality of the framework and its underlying concepts for spatial awareness. The evaluation of prototypes with different scenarios is a common methodology in the field of pervasive computing [Sch02], and it has been used for many of the projects presented in Section 2.6.1; examples are the projects Mediacup [BGS01], Easy Living [BMK+ 00], Cooperative Artefacts [SGKK04] and the ParcTab system [WSA+ 95]. First, in Section 7.1, a design process for the development of spatially aware applications is proposed, which is based on quantitative and qualitative abstractions of spatial context information. Afterwards, Section 7.2 presents application scenarios which have been developed therewith, provides implementation details and discusses evaluation criteria as well as our subjective experiences with regard to using the Zones-of-Influence Framework for each of them. Section 7.3 finally compares the application scenarios with respect to several aspects including the utilized Zones-of-Influence and qualitative spatial relations, the incorporated physical sensors and actuators as well as the applied rule-based inference techniques for achieving spatial awareness, and concludes with a comprehensive discussion on the realized implementations and the feasibility and quality of the awareness concepts developed within the present thesis.
7.1 Development of Spatially Aware Applications The proposed concepts and solutions for spatial awareness are based on quantitative and qualitative abstractions of the spatial contexts of spontaneously interacting
148
7 Framework Evaluation
digital artifacts, and we consider spatial relations among them – e.g. their positions with respect to each other – particularly useful for developing spatially aware applications. With the Zones-of-Influence Framework presented in Chapter 6, a first prototype for using qualitative spatial relations in real-world application scenarios has been implemented, which allows for adapting the behavior of digital artifacts depending on their spatial environment. In this section, we present a design process for the development of spatially aware applications with it, which we have used for implementing the application scenarios in Section 7.2. It corresponds to the process for building context-aware applications presented in [Dey00], which basically covers (i) the specification of the problem and a high-level solution, (ii) the acquisition of context information from sensors, (iii) its delivery to one or more applications, (iv) a conversion to a usable form through interpretation and finally (v) its use for achieving context-aware behavior; the steps (iii) and (iv) can be removed by providing respective abstractions with a supporting framework [Dey00]. From our practical experience, using the design process – together with the Zonesof-Influence Framework – makes it easier for application programmers to design and build spatially aware applications. Our process is focused on spatial contexts on the one hand, and it is – to a certain extent – specific to the Zones-of-Influence Framework on the other hand. It has been published in a shortened version in [HHF08], and consists of the following five steps: 1. Identification of spatial relations between objects which are relevant for the planned application, including the decision (i) for a quantitative or qualitative representation at a time and (ii) about which relations have to be inferred from recognized or other inferred relations. As described in Section 3.2.2, a spatial relation is characterized by its type (e.g. qualitative static orientation), the possible values (e.g. left, front, right and back), their semantics – i.e. how they are interpreted, which depends on the respective application (e.g. the meaning of front may be different in automotive scenarios and human-computer interaction) – and the dimensionality, among others. 2. Definition of Zones-of-Influence needed for the recognition of the relations identified in step 1. Therefore, the artifacts which are to be related, as well as their corresponding Zones-of-Influence (i.e. represented by one or more possibly different spatial regions), have to be identified. In this regard, it has to be considered which types of ZoIs (e.g. with freeform shapes, which are polygons defined by a sequence of points, or using spherical polar coordinates for representing the anchor point’s position) are either already supported by the available framework implementation (i.e. they can be parsed
7.2 Application Scenarios
149
by the Zones-of-Influence Service described in Section 6.2.2) or have to be integrated first. 3. Selection of spatial sensors which are used for determining the properties position, direction as well as the spatial extension of dynamic Zones-ofInfluence. For example, the ZoI of a moving car’s breaking distance is modelled by its changing position and direction, as well as its shape which is defined by the current velocity of the car (cf. Section 3.1.1). Such dynamic properties can be specified in the self-descriptions of artifacts by including references to the respective sensors, for which sensor data accessors are either available or have to be implemented. 4. Selection of relation recognizers according to the relations and Zones-ofInfluence identified in the steps 1 and 2. Similar to the use of spatial sensors, suitable recognizers are either already available or have to be implemented otherwise. Relation recognizers generate spatial relations by comparing two Zones-of-Influence at a time, for which reason they must be compatible with the spatial properties contained in the Zones-of-Influence that are to be related and their reference systems. As described in Section 6.2.3, they can (i) directly provide qualitative spatial relations or (ii) generate quantitative relations from Zones-of-Influence, whereas in the latter case the quantitative results are used by one or more qualitative recognizers which abstract them to qualitative spatial relations. 5. Modelling of the application behavior by using the recognized qualitative spatial relations. Basically two – not mutually exclusive – ways are distinguished: (i) the use of rules for inferring high-level relations or mapping combinations of recognized and inferred relations to application-level actions (cf. Chapter 5), and (ii) a registration of the application with the Relations Service for getting notified about recognized qualitative relations. While in the latter case the application logic for combining relations has to be implemented traditionally, this is done by declarative means in the former case.
7.2 Application Scenarios The goal of the evaluation discussed in the following was mainly to prove the feasibility of the Zones-of-Influence Framework and its underlying concepts for spatial awareness. We decided for a scenario-based evaluation by means of spatially aware applications, which are – according to the focus of this work – built
150
7 Framework Evaluation Table 7.1: Relevancy of spatial abstractions for application scenarios.
Vibro-tactile space awareness Focus/nimbus awareness Spatiotemporal awareness
quant. spatial properties
qual. spatial relations
relevant relevant less relevant
less relevant relevant relevant
upon qualitative spatial relations. Thus, the co-location of multiple possibly moving objects has to be of relevance on the one hand and qualitative abstractions must be appropriate for them on the other hand, which is for example due to the fact that quantitative representations are not needed or simply not possible for reasons of computational or sensory limitations. In order to cover a wide spectrum of framework features and spatial awareness concepts, multiple applications requiring different kinds of spatial contexts and their abstractions have been developed by using the design process presented in the previous section. However, the evaluation was not intended just to provide a proof of concept, but also to show the flexibility of the Zones-of-Influence Framework with respect to the utilization of different application-specific spatial contexts and abstractions, the reusability of components for quantitative and qualitative spatial abstractions, the adequacy of qualitative relations and rule-based reasoning for building spatially aware applications as well as the correctness and efficiency of the prototypical framework implementation. According to the spatial abstractions discussed in Chapter 3, spatially aware applications can be distinguished depending on the relevancy of (i) quantitative abstractions of spatial properties with Zones-of-Influence and (ii) qualitative abstractions of spatial relations for their development. As can be seen in Table 7.1, we have chosen the following applications – which have a different focus on these abstractions at a time and have been published in shortened and slightly modified versions in [FER+ 08, HFH+ 08, HHF08] – for the evaluation: (i) vibro-tactile space awareness (cf. Section 7.2.1), where recognized relations are displayed with physical actuators for providing enhanced spatial awareness to humans, (ii) focus/nimbus awareness (cf. Section 7.2.2), where high-level spatial relations are inferred in order to control the spontaneous interaction between autonomous artifacts, and (iii) spatiotemporal awareness (cf. Section 7.2.3), where combinations of spatial and temporal relations provide the basis for an artifact’s adaptation to spatial changes in its environment. For the sake of simplicity, just two-dimensional relations are used, and they turned out to be sufficient for the selected scenarios. To the best of our knowledge,
7.2 Application Scenarios
151
all presented applications are novel in that qualitative spatial relations and their combinations are used for designing and building them. It should be noted that the discovery of artifacts and the exchange of spatial context information are just emulated by manually loading self-descriptions from XML files or interactively creating them with our Visualization Service implementation, as these issues have already been addressed within the Peer-it project [FHdSR+ 07, FHR+ 06a, FHR+ 08].
7.2.1 Vibro-Tactile Space Awareness With the first scenario, the Zones-of-Influence Framework is evaluated with a realworld application that incorporates physical sensors for acquiring position and direction information as well as physical actuators for displaying respective spatial relations. The aim was to prove the feasibility and quality of the framework by means of a working hardware prototype, with a focus on quantitative abstractions of spatial properties. As physical sensors and actuators should be incorporated, we were searching for an application scenario in which recognized spatial relations lead to respective actuator actions. We decided for directly adapting the artifact’s actuators to relations it is currently aware of, and used a belt with eight vibro-tactile actuators as one representative example for that, similar to the one presented in [TY04]. It has the advantage that spatial relations can be perceived by a moving person and therewith allows for evaluating the spatial awareness of the running system in a living lab environment. This vibro-tactile space awareness application, which has been developed within an industrial research project with Siemens AG Germany [FDE+ 08] and published in [FER+ 08, HHF08], should display orientation and distance relations to surrounding objects – which are inside a certain spatial region around the belt – by correspondingly adapting the actuators’ vibration intensities. This application has been developed according to the five steps of our design process as described in the following. First, we identify the need for two-dimensional static topological, orientation and distance relations. We define a region around the belt in which it is responsive to objects around, requiring the qualitative topological relations overlaps and inside. According to the eight vibrator elements of the tactile belt, we decide for the eight qualitative orientation relations front, front-right, right, backright, back, back-left, left and front-left, which cover equally sized sectors and are aligned with respect to the belt’s front side. We additionally distinguish ten qualitative distance relations distance_0 to distance_9, which partition the surrounding space in circular regions of the same size and are to be represented by different vibration intensities. Each one has a range of 0.25m, which means that distance_9 covers quantitative Euclidean distances greater that 2.25m; the number of distin-
152
7 Framework Evaluation
guished distances is estimated with regard to the haptic perception of humans. As the spatial extension of the belt can be disregarded due to its small diameter, we are interested in the distance from the belt’s center to the spatial regions of surrounding objects. These relations can all be recognized from a pairwise comparison of Zones-of-Influence, and no relationship inference is required therefore. The use of such qualitative spatial relations turns out to be an adequate abstraction in the design phase of this application, as the involved discretization of space naturally maps to the number of available vibrator elements with regard to orientation relations, and to the number of vibration intensities that can be reasonably distinguished by humans with regard to distance relations. The same holds for the topological relations, as we are only interested in whether a surrounding object is within a certain responsive area around the belt or not. From these three examples, it can be seen that the qualitative relationship abstractions – including their dimensionality and granularity, the spatial primitives and the used frames of reference – are application-specific, and the distinction of four orientation and three distance relations presented in Section 3.2.3 cannot be applied for example. Qualitative relations thus impose a layer of abstraction in the design process, which allows for building spatially aware applications without having to deal with quantitative and sensor-specific representations. However, although our focus is on qualitative relations, it should be noted that in the case of distance relations a quantitative representation could be the preferred choice if a fine-grained mapping of distances to vibration intensities is required. In the second step of the design process, we define two types of Zones-ofInfluence which are required for recognizing the three relations identified above. They are associated with the vibro-tactile belt on the one hand and physical objects it should display on the other hand, and thus enable the computational processing of spatial relations among the involved artifacts (cf. Chapter 3). The first Zone-of-Influence is the awareness shape of the belt (and thus also that of the person wearing it), which represents the spatial region in which it is responsive. It is modelled as an ellipse-shaped ZoI with the belt in its center, which can be dynamically adjusted in width and height, and whose position and direction are given by respective spatial sensors; the XML description of this Zone-of-Influence is shown in Listing 7.1. All other physical objects an application is interested in are represented with static ellipsoid- and freeform-shaped Zones-of-Influence corresponding to their positions, directions and physical extensions in real space. Thus, similar to qualitative abstractions of spatial relations, Zones-of-Influence are a means for quantitatively abstracting those spatial properties from the multitude of physical objects that are actually relevant for the application, while others are not represented. This contributes to the efficiency of the framework, as just rela-
7.2 Application Scenarios
153
tions between two such Zones-of-Influence are calculated at a time. The position and direction reference system as well as the shape type of each used ZoI must be known by the Zones-of-Influence Service, but can easily be extended with new ones as has been described previously in Section 6.2.2. Listing 7.1: Zone-of-Influence for the vibro-tactile belt’s awareness shape. < D A D e s c r i p t i o n E l e m e n t type = " ZoI " > < Name > AwarenessShape < Position refSystem = " IS900AtJKU " > <X > @PosDir . x @PosDir . y 0 < Direction refSystem = " IS900AtJKU " > < Yaw > @PosDir . yaw < Pitch >0 < Roll >0 < Shape type = " Ellipse2D " > < Width > @ShapeSize . width < Height > @ShapeSize . height
The underlying concept of our research is that each one of these ZoIs is associated with a computing system embedded in the respective physical object, which shares this spatial information with the belt – and maybe also with other objects (cf. Section 2.2). Thus, each such digital artifact which contains a representation of its physical extension in the self-description can be displayed by the vibro-tactile belt – autonomously upon coming into communication range and without any explicit user interaction. However, as the ad-hoc exchange of self-descriptions has not been implemented yet, just one digital artifact – namely the belt which is connected to an external notebook computer running the framework – is used for the evaluation, and the Zones-of-Influence of surrounding objects like chairs and tables are static (i.e. they do not move) and represented with self-descriptions that are manually loaded at runtime. Step three of the design process is about the selection of spatial sensors. In order to determine the position and direction of the vibro-tactile belt’s dynamic Zone-of-Influence, the IS-900 motion tracking system [Int08] – which is specified with the SensorData element previously shown in Listing 6.1 – is used. This physical sensor must be running and the respective sensor data accessor must be registered with the Zones-of-Influence Service when starting the application. For the size of the dynamically adjustable awareness shape, a software sensor has been implemented that consists of a GUI with two sliders for changing the ellipse’s width and height. As with the physical sensor, it also has to be loaded at startup and is included in the belt’s self-description by specifying another SensorData element with the name ShapeSize. The use of these two fundamentally different sensors demonstrates the flexibility of the framework, which can be extended with
154
7 Framework Evaluation
sensors for acquiring further spatial contexts by simply implementing a new sensor data accessor component and including the respective SensorData specification in the self-description (cf. Section 6.2.2). Afterwards, a relation recognizer is implemented for each required qualitative spatial relation identified in the first step of the design process, whereas the static orientation and distance recognizers are split in a quantitative recognizer which provides numeric values for the relations (e.g. 11, 4° for orientation) and a qualitative recognizer which abstracts these numeric values to qualitative symbols (e.g. front) at a time. The qualitative orientation and distance recognizer are parameterized to provide the eight orientation and ten distance relations, respectively. In this regard, orientation relations are calculated by relating the anchor points of two Zones-of-Influence. Distance relations are computed from the belt’s center (i.e. the primary ZoI in this case) to the shape of a surrounding ZoI (i.e. the reference object), independently of the belt’s awareness shape. For topological relations, a single qualitative recognizer which provides the five relations according to the RCC-5 calculus mentioned in Section 3.2.3 is used. All implemented recognizers are able to recognize relations between arbitrary Zones-of-Influence with a position given in Cartesian coordinates, a direction given in Euler angles and any shape supported by the Zones-of-Influence Service (i.e. an ellipse-shaped ZoI for the awareness shape of the belt and a freeform-shaped ZoI for the objects around the belt in this case), and they are eventually registered with the Relations Service by the vibro-tactile space awareness application at runtime. Relation recognizers acquire relationships and make them available to applications, comparable to the context widgets mentioned in Section 2.6.1 which have been used as abstractions of general contexts in [SDA99]. An advantage of using relation recognizers is that they abstract from sensors and low-level spatial representations conforming to the application needs, which eases the development of spatially aware applications. It can be seen from the above example that recognizers can be implemented according to application needs, which proves the flexibility of the Zones-of-Influence Framework with regard to utilizing different spatial contexts and abstractions. Moreover, they are reusable building blocks that can easily be exchanged and used by different applications, which is facilitated by the possibilities to use different qualitative for abstracting the relations of a single quantitative recognizer as well as to parameterize qualitative recognizers with relation names and corresponding quantitative ranges. Recognizers can thus be tailored to the requirements of different applications, as for example to build another vibro-tactile belt containing six vibrator elements only. Due to the fact that relation recognizers can be dynamically registered and unregistered with the Relations Service, just relations that are required by running applications and no
7.2 Application Scenarios
155
unnecessary relations are calculated, which again contributes to the efficiency of the Zones-of-Influence Framework. This not only allows to change the recognized relation types at runtime, but also their semantics by e.g. dynamically replacing a distance recognizer which calculates distances between two points with another one that takes into account the shape of the reference object. A further aspect is that relation recognizers detect changes of relations, which causes calculations in registered listeners only if relations have changed and thus improves the overall efficiency. Listing 7.2: Rule for the vibro-tactile space awareness application. rule " vibro - tactile - space - awareness " when Relation ( name == " StaticTopology " , v alue == " overlaps " || value == " inside " , pZ:pZoI , rZ:rZoI , i1:interval ) Relation ( name == " StaticOrientation " , o:value , pZoI == pZ , rZoI == rZ , i2:interval ) Relation ( name == " StaticDistance " , d:value , pZoI == rZ , rZoI == pZ , i3:interval ) eval ( i1 . isCurrent () && i2 . isCurrent () && i3 . isCurrent ()) eval ( rZ . equals ( " P e r s o n : A w a r e n e s s S h a p e " )) then TactileBelt . vibrate ( pZ , o , d ); // start vibration of the belt end
In the last step of the process, the application behavior is modelled with a single rule shown in Listing 7.2, which infers the application-level actions by using the recognized relations. It checks for the simultaneous existence of a topological overlaps or inside relation as well as an orientation and distance relation between the primary and reference ZoIs pZ and rZ, which represent a certain object’s physical shape and the belt’s awareness shape at the current point in time (method isCurrent that is invoked for the relation intervals). With the second eval statement, the reference ZoI is limited to the specified name. Thus, for each combination of relations in the repository, for which the rule condition is fulfilled, the application-level method TactileBelt.vibrate is invoked with the primary ZoI pZ and its orientation o and distance d to the reference ZoI rZ as parameters. This method simply activates the vibration element as defined by the orientation relation (e.g. the vibrator in the middle front of the belt for the orientation front) and with the vibration intensity given by the distance relation (e.g. an intensity of 80% for the relation distance_2), and it consists of a few lines of code only. In the case this rule is fulfilled for multiple objects, it is executed for each of them and multiple vibrator elements are activated. Once the rule’s condition – which is periodically evaluated whenever the repository is updated – is no longer fulfilled, the belt stops vibrating.
156
7 Framework Evaluation
From this example, it can be seen that the rule-based implementation is pretty simple, intuitive and would also allow for easy changes of the application behavior; we thus consider it adequate for applications where actions have to be performed depending on the existence of certain combinations of relations. We have additionally implemented the vibro-tactile space awareness application without using rules by registering with the Relations Service to get notified about recognized relations; the information which vibrator element is selected (qualitative orientation relation), if it should vibrate (qualitative topological relation) and with which intensity (quantitative distance relation) is updated in the respective callback methods (cf. Section 6.2.3). In contrast to the rule-based variant, in which rules are executed periodically in the update thread of the Relations Service, it has the advantages that the application can react immediately once a relation has changed, and that a higher resolution is achieved by mapping quantitative relation values to vibration intensities (which however can also be achieved by increasing the number of qualitative relations in the rule-based variant). On the other hand, it has the disadvantages that the application behavior is hard-coded and cannot be changed at runtime, and that the evaluation of relationship combinations has to be implemented, which requires a multiple of lines of code. The hardware of the vibro-tactile belt, which has also been developed within the Siemens research cooperation [FDE+ 08], can be seen in the Figures 7.1 to 7.3. Figure 7.1 shows the belt with eight vibrator elements enclosed in plastic housings. They are connected to the belt controller shown in Figure 7.2, which consists of an AVR Mega 32 microcontroller board from RN-Control [Rnc08] for controlling the vibrator switches (i.e. a transistor array) according to commands received via Bluetooth from a notebook computer running the Zones-of-Influence Framework, for which a connectBlue serial port adapter [Con03] is used. Figure 7.3 shows the waist of a person wearing the vibro-tactile belt, together with a battery pack and the IS-900 system for acquiring the belt’s position and direction. It comprises a tracker mounted on the belt controller, as well as a wireless transmitter that sends measurements to the IS-900 processor in the background where they can be accessed by the computer. The belt is controlled from a dedicated framework component, which is started by the application and sends vibration commands to the microcontroller whose firmware activates and deactivates the vibrator elements accordingly. In order to prove the feasibility and the correctness of the framework implementation, the application has been evaluated with a simple scenario described in the remainder of this subsection. First, the Zones-of-Influence Framework with all the components described in the previous chapter as well as the vibro-tactile space awareness application have been started. After loading the static Zones-of-Influence specifications of four
7.2 Application Scenarios
157
Figure 7.1: Vibro-tactile belt with eight vibrator elements.
Figure 7.2: Controller of the belt.
physical objects from XML files, they are shown in the 2-D visualization of the interactive Visualization Service implementation (cf. Section 6.2.1) with corresponding polygonal shapes as can be seen in the screenshot of Figure 7.4. This GUI also has controls for adding new ZoIs and changing spatial properties of displayed ones, which are however not relevant for the explanation and thus hidden in the figure. Additionally, the self-description for the person wearing the vibro-tactile belt has been loaded, which contains the dynamic ellipse-shaped ZoI representing the
Figure 7.3: Person wearing the belt.
158
7 Framework Evaluation
Figure 7.4: Visualization of the 2-D scene and the belt’s vibrator elements.
awareness shape around the belt. It is visualized with an arrow indicating its spatial direction in the figure, and changes with the position and direction of the moving person as well as the adjustment of the software sensor defining its shape. As soon as it overlaps with Zones-of-Influence around, the respective vibrator elements are activated. In Figure 7.4, two of them are in a topological overlaps relation with the awareness shape, resulting in a front-right relation to the table (with respect to its anchor point that is shown with a darker dot in the figure) and a back relation to an obstacle behind the user. This leads to the activation of the corresponding vibrator elements, which are visualized in the smaller window of the figure. As described at the end of Section 3.2.2 however, the recognition of orientation relations from anchor points may result in cognitively incorrect relations; this appeared to be the case for some spatial configurations of the belt and the table also in this scenario, mainly due to the fact that the table’s anchor point lies at the border of its physical shape. Both implementations – namely the rule-based variant and that without using rules – run stable and exhibit the same behavior, except the difference that the application which does not use rules reacts immediately and not periodically on changed relations.
7.2.2 Focus/Nimbus Awareness The aim of the second spatially aware application scenario was to evaluate the framework with regard to the rule-based inference of high-level spatial relations. Thus, instead of just displaying recognized spatial relations to surrounding artifacts like in the first scenario, the issues of inferring new relations between artifacts from combinations of recognized ones and using these relations at the application level for controlling their spontaneous interaction are addressed in the following. As discussed in Section 2.3, this provides a basis for achieving self-organizing behavior without explicit input from outside the system. A well understood model for
7.2 Application Scenarios
159
controlling the interaction among objects in virtual environments based on their spatial awareness of each other is proposed in [BF93]; as it can be translated to physical objects and their interaction in real space, we use this model for the second scenario. It introduces three spatial abstractions: aura, focus and nimbus. Aura is defined as a spatial (e.g. circular) area associated with each object, which limits the interaction to those objects whose auras collide. Furthermore, an object p is said to be in the focus of another object r if r is aware of p, in which case r is said to be in the nimbus of p; focus and nimbus can be modelled e.g. as segments of a circle [BF93]. This model has been translated to ad-hoc networked mobile computing devices which start to interact once their auras collide [FHM+ 04], as well as to wall-sized displays where the presence of people in defined regions in front of a display determines which information is shown [Vog02]. For the scenario presented in this subsection, we extended the example of [Vog02] by inferring relations and triggering actions if and only if a person is within a certain point of interest’s view area (i.e. from where it can be seen) and additionally at the same time the point of interest is in his field of view (i.e. the person looks at it); thus, it is considered to be in the focus of the person and consequently the person is in the nimbus of that point of interest. This is exemplarily shown in Figure 7.5 with a moving person (i.e. the reference artifact r) and his field of view by means of different positions and directions at three points in time, whereas only at the time t3 the point of interest (i.e. the primary object p) is in the person’s focus. As for the previous scenario, we developed the focus/nimbus awareness application along the five steps of the design process presented in Section 7.1. From the application description above, two types of qualitative spatial relations are needed, whereas the first one are static orientation relations for the person’s field of view (FoV). We are just interested in the two orientation relations within and outof, which are defined as the ±45° range with respect to the person’s axis of direction equal to the relation front (i.e. the field of view) and the remaining 270° corresponding to the relations right, back and left defined in Section 3.2.3 respectively, and we introduce the new application-specific relation type FoV for it. The second type are static topological relations for detecting whether the user is within the polygonal view area of a point of interest, for what the RCC-5 relation contains introduced in Section 3.2.3 is used. The person’s spatial extension can be disregarded, as it is sufficient for the application to know if his center is in a point of interest’s view area. We just consider discrete focus and nimbus relations (i.e. different levels of awareness are not supported), and they are – together with an application action notifying about the current focus/nimbus awareness – inferred from the orientation and topological relations with a respective rule. As in
160
7 Framework Evaluation
Figure 7.5: Exemplary application scenario for focus/nimbus awareness.
Section 7.2.1, qualitative relations have shown to be an adequate abstraction also for this scenario, as they directly match the requirements of the application which distinguishes a few different spatial situations only (i.e. the detection of presence in a person’s field of view and the containment in a point of interest’s view area), and thus allow for an application development at a higher level of abstraction. For recognizing relations of these two types, we observed the following Zonesof-Influence to be relevant: (i) the view area of a certain point of interest which is modelled as a static freeform-shaped ZoI with the point of interest as its anchor point, and (ii) a dynamic point-shaped ZoI representing the person which changes its position and direction in space. In both cases, position and direction are again represented with Cartesian coordinates and Euler angles from the IS-900 tracking system, respectively. The possibility of representing such application-specific Zones-of-Influence with the same sensor used in the first scenario already not only shows the flexibility of the framework with respect to different quantitative spatial abstractions, but also the reusability of already implemented sensor data accessors. The XML representation of the point-shaped ZoI equals that of Listing 7.1 except for the shape-description which is of type Point, and the freeform ZoI is given by a fixed position and direction as well as a shape of type Freeform2D containing a series of points with Cartesian coordinates. An issue we have not considered in the present thesis is that the view area may be influenced by other ZoIs, in particular by those occluding the point of interest. According to the two types of relations identified above, two relation recognizers are required. First, the application-specific FoV recognizer builds on the same quantitative orientation recognizer also used in the first scenario, which provides a numeric angle for the orientation of a primary ZoI’s anchor point to that of the ref-
7.2 Application Scenarios
161
erence ZoI. It is implemented as a parameterized qualitative orientation recognizer, and results in a qualitative within relation for orientations in the range of ±45° with respect to the person’s axis of direction as well as an outof relation otherwise. Second, the topological recognizer is the same used in the first scenario also, which provides one of five topological relations between a point (i.e. the moving person) and any other Zone-of-Influence (i.e. the view areas of points of interest) at a time in this scenario. Thus, three recognizers – two qualitative and one quantitative recognizer – are required, and they are registered with the Relations Service by the focus/nimbus awareness application at runtime. It can be seen that all of them are reused from the first scenario, which is a crucial advantage of using a dedicated relation recognizer for each spatial relation as mentioned in Section 7.2.1 already. The application behavior is defined with a single rule depicted in Listing 7.3. Its condition checks for the simultaneous existence of a FoV relation within of a primary ZoI pZ from a point of interest to the reference ZoI rZ of the person, as well as a topological contains relation from pZ to rZ. As a consequence of a fulfilled rule condition, two relations are inserted in the repository: (i) a relation within of type Focus from pZ to rZ and (ii) a relation within of type Nimbus from rZ to pZ (i.e. pZ and rZ are switched). Applications building on top of this focus/nimbus awareness can register with the Relations Service to get notified about relationship changes or query the repository for relations of these two types, and conduct according behavior. In addition, the rule also infers an application-level action by invoking the method FocusNimbusAwareness.notify, which notifies about the focus/nimbus relations between pZ and rZ with a simple audio and visual sign. Due to the periodic rule execution, this happens once per second in the current implementation. Listing 7.3: Rule for the focus/nimbus awareness application. rule " focus - nimbus - awareness " when Relation ( name == " FoV " , value == " within " , pZ:pZoI , rZ:rZoI , i1:interval ) Relation ( name == " StaticTopology " , v alue == " contains " , pZoI == pZ , rZoI == rZ , i2:interval ) eval ( i1 . isCurrent () && i2 . isCurrent () && rZ . equals ( " Person:FoV " )) then insert ( new Q u a l i t a t i v e R e l a t i o n ( " Focus " , " within " , pZ , rZ , i1 . intersect ( i2 )); insert ( new Q u al it a ti ve R el at i on ( " Nimbus " , " within " , rZ , pZ , i1 . intersect ( i2 )); F o c u s N i m b u s A w a r e n e s s . no tify ( focus , nimbus ); end
There are three important things to consider. First, the rule for merging relations shown previously in Listing 5.1 must additionally be contained in the rule base, as the two inferred relations are inserted in the repository every time the rule is
162
7 Framework Evaluation
Figure 7.6: Visualization and relations of the focus/nimbus application.
fired, and consequently two additional relations with overlapping intervals would be added to the repository at a time. Second, as in the first scenario also, it is necessary to check with the method isCurrent that both relations hold at the same time, as otherwise relations with intervals that do not overlap may cause the rule to execute and thus lead to wrong results. However, as rules are fired for changed relations only (cf. Section 5.2), it would instead be possible to ensure that the two intervals are not disjoint (i.e. they are not in a temporal before, after, meets or metby relation to each other). When comparing the two rules of Listing 7.2 and 7.3, it can be seen that they are quite similar, although effecting completely different behavior; from an implementation viewpoint, the main difference between the two application scenarios is the actual implementation of the invoked methods and the inference of relations in the second scenario. Third and last, the reference ZoI is limited to a certain name; in order to distinguish points of interest from other Zones-of-Influence in the environment, further conditions on the name of the primary ZoI could be added. Like in Section 7.2.1, the feasibility and correctness of the framework have been evaluated with a simple scenario visualized in the 2-D visualization of Figure 7.6.
7.2 Application Scenarios
163
It shows a certain spatial situation involving a person and a single point of interest, together with the corresponding relations of the reference artifact (i.e. the person) in the repository. The polygonal freeform shape, which has been constructed with this GUI, represents the view area from where the point of interest can be seen. As in the first scenario, the dynamic ZoI of the person – who is again equipped with a wireless IS-900 tracker – has been loaded from a file, and it is shown in Figure 7.6 with lines visualizing the 90° viewing angle. The recognized and inferred relations are stored with their intervals in the relations repository, according to the current and past spatial situations. It can be seen that the FoV relation within and the StaticTopology relation contains simultaneously hold between the primary ZoI PointOfInterest:ViewArea and the reference ZoI Person:FoV within the discrete time interval [36, 45], for which reason the two relations within of type Focus and Nimbus are inferred for this interval. Although the current implementation uses two-dimensional Zones-of-Influence and relation recognizers only, transferring it to 3-D – e.g. by representing the field of view with a cone and the view area of the point of interest with a volumetric freeform shape – would be possible due to the flexible framework architecture, simply by extending it with components for parsing respective ZoI definitions and recognizing relations between them.
7.2.3 Spatiotemporal Awareness The previous scenarios evaluated the Zones-of-Influence Framework with two applications that have been implemented with the design process presented in Section 7.1. They have shown its feasibility and quality concerning a series of aspects, including the application development with and without rules, the use of physical sensors and actuators, Zones-of-Influence with dynamic position, direction and shape, the recognition of qualitative static spatial relations with both quantitative and qualitative recognizers, as well as the rule-based inference of new relations and application-level actions from the history of qualitative spatial relations. However, two aspects have not been covered by now: (i) the recognition of qualitative dynamic spatial relations and (ii) the inference of qualitative spatiotemporal relations. While in the former case two or more successive quantitative spatial relations of a certain type are taken into account in order to estimate the spatial change at a discrete point in time (cf. Section 3.2.3), two or more qualitative relations are combined logically and temporally in order to represent a time series of spatial relations in the latter case (cf. Section 5.2.1). Taking into account time allows for representing spatial changes of an artifact’s situation with respect to its environment, and thus provides a basis for adapting its behavior accordingly; in [GGH06] for example, the inference of spatiotemporal relations has been identified to be use-
164
7 Framework Evaluation
Figure 7.7: Qualitative spatial relations at a point in time.
ful for smart home environments in order to recognize activities with pre-defined sequences of relations. For evaluating the Zones-of-Influence Framework regarding the achievement of such spatiotemporal awareness, we will – again by using our design process – describe the implementation of two exemplary application scenarios. In contrast to those presented in the previous sections, no physical sensors but just the interactive Visualization Service described in Section 6.2.1 has been used. The first scenario, which is about recognizing the spatial relation that an artifact p is moving from behind toward an artifact r at a point in time, is shown in Figure 7.7. The following qualitative relations are required for that, which are again twodimensional and defined like in Section 3.2.3: (i) static orientation relations to detect from where artifact p is approaching (i.e. the four relations front, right, back and left with respect to the direction of r), (ii) dynamic distance relations for distinguishing if artifact r is moving toward or away from artifact p, or if it is not moving at all which is represented with the relation stable-dist, and (iii) static distance relations for evaluating whether p is in proximity (i.e. either near or medium-dist) of r or not, which has a different meaning with respect to an approaching artifact p in this scenario. Each distance relation has a range of 2m; thus, the relation far covers distances greater than 4m. In order to allow spatially aware applications to react accordingly, two relations – one representing the time interval in which p is moving toward r from behind and the other one representing the interval in which they are additionally in spatial proximity – are inferred with rules. This selection of qualitative relations and their declarative combinations shows the high flexibility of the Zones-of-Influence Framework concerning the provision of application-specific spatial abstractions. The spatial extensions of
7.2 Application Scenarios
165
Figure 7.8: Time series of qualitative spatial relations.
both artifacts can be disregarded, as we assume that they are too small and distant to be of interest; thus, in contrast to the scenario of Section 7.2.1, the quantitative spatial abstractions with Zones-of-Influence are less relevant in this case. The second scenario, which is shown in Figure 7.8, is about inferring a qualitative spatiotemporal relation that describes a time series of qualitative spatial relations (i.e. shown with the four points in time t1 , . . .t4 in the figure). It should represent the time interval in which artifact p1 moves – starting from a back relation to the reference artifact r – around r in clockwise direction, and during this time another artifact p2 is moved back and forth. From this scenario description, we identify the need for (i) static orientation relations with front and back covering 120° and left and right covering 60° at a time, as well as for (ii) dynamic distance relations that are equally defined like in the previous scenario. Spatial extensions are again disregarded, as neither topological relations nor distance relations to the boundaries of artifacts are required. As discussed in Section 3.2.1, qualitative relations are often the preferred choice when quantitative representations are not required or unavailable, and they appeared to be adequate abstractions in these two scenarios where the relations can be extracted more or less directly from their natural language specifications. The next step in the design process is the identification of Zones-of-Influence between which the qualitative relations are actually recognized. In the scenario of Figure 7.7, these are the ZoIs of the involved artifacts’ physical extensions, whereas just the position of artifact p is dynamic. In the other scenario shown in Figure 7.8, all artifacts are also represented with their physical ZoIs, which are static as well as rectangular- and triangular-shaped for the artifacts r and p2 respectively, except for artifact p1 which is represented as a point-shaped ZoI with
166
7 Framework Evaluation
a dynamic position. However, just the anchor points of the respective Zones-ofInfluence are considered in both scenarios, which is due to the fact that no relations that take into account their shapes have to be recognized. As we use simulation only for evaluating the spatiotemporal awareness of the Zones-of-Influence Framework, no sensors have to be integrated, but the interactive Visualization Service implementation is used instead; thus, step three of the design process is skipped. However, given that the Zones-of-Influence Framework runs on a platform which discovers artifacts and exchanges self-descriptions with them upon coming into range, artifact r would be able to autonomously recognize and infer the relations defined in the first step to its surroundings. In step four, the relation recognizers are selected. For static orientation, the quantitative and qualitative recognizer implemented for the vibro-tactile space awareness application of Section 7.2.1 are used, and a new one for quantitative static distance relations between the anchor points of two ZoIs is implemented; the qualitative distance recognizer, which abstracts the distance relations of the quantitative one, can also be reused. The qualitative static orientation and distance recognizer are parameterized to provide four orientation and three distance relations as identified in the first step above. For dynamic distance relations, a qualitative recognizer is implemented which uses the same quantitative static distance recognizer by registering for notifications about started, changed or ended quantitative relations with it, and compares two successive quantitative values for detecting a significant change in distance. This shows the reusability of already available recognizers, in particular the possibility to use a single quantitative by multiple qualitative recognizers, and again demonstrates the flexibility of the framework with regard to providing those relationship abstractions that are actually required by a particular application. Because of the strategy for queueing started and ended relations in the Relations Service (cf. Section 6.2.3), namely that equal relations which both started and ended within one update period are disregarded, just the most recent qualitative relation is stored in the repository at a time. For this reason, the repository update rate must be adapted to the frequency of relation changes which should be recognized. For the inference of qualitative relations, which provide the basis for the application-level adaptation to spatial situations, multiple rules are used. This not only contributes to the reusability of spatial abstractions, which is due to the fact that relations inferred by one rule can be used in multiple other rules, but also allows for splitting complex conditions in two or more smaller ones and thus ease the application development. For the first scenario, the rule approaching-from-behind shown in Listing 7.4 infers the relation behind of type Approaching from pZ to rZ (i.e. the artifacts p and r in Figure 7.7), and the second rule infers the relation
7.2 Application Scenarios
167
behind-in-proximity of the same type if and only if the inferred relation behind holds and at the same time the two artifacts are in a distance relation of either near or medium-dist. It should be noted that the names and values of inferred relations are user-defined in order to meet the respective application needs. We have simulated the scenario by creating the Zones-of-Influence P:PhysicalShape and R:PhysicalShape representing the primary and reference artifact’s physical extension respectively, an moved the former toward the latter in the GUI of the Visualization Service shown in Figure 7.9; the corresponding correctly recognized and inferred relations of artifact r can also be seen in the figure. Listing 7.4: Rules for the scenario shown in Figure 7.7. rule " approaching - from - behind " when Relation ( name == " StaticOrientation " , value == " back " , pZ:pZoI , rZ:rZoI , i1:interval ) Relation ( name == " DynDistance " , value == " toward " , pZoI == pZ , rZoI == rZ , i2:interval ) eval ( i1 . isCurrent () && i2 . isCurrent ()) then insert ( new Q ua l it at i ve Re l a t io n ( " Approaching " , " behind " , pZ , rZ , i1 . intersect ( i2 ))); end rule " approaching - from - behind - in - proximity " when Relation ( name == " Approaching " , value == " behind " , pZ:pZoI , rZ:rZoI , i1:interval ) Relation ( name == " StaticDistance " , v alue == " near " || value == " medium - dist " , pZoI == pZ , rZoI == rZ , i2:interval ) eval ( i1 . isCurrent () && i2 . isCurrent ()) then insert ( new QualitativeRelation ( " Approaching " , " behind - in - proximity " , pZ , rZ , i1 . intersect ( i2 ))); end
Three rules are required for the second scenario shown in Figure 7.8. The first one infers – similar to the rule of Listing 5.2 – a new relation around-cw of type Moving for a sequence of the static orientation relations left, front and right that are in a temporal meets relation, thus representing the time interval in which the primary artifact pZ is moving around rZ (i.e. p1 and r in Figure 7.8). A second rule, which is shown in Listing 7.5, checks the repository for an away and a toward relation between pZ and rZ, whereas the latter holds after the former relation (i.e. at least one discrete point in time must be between the two intervals). It infers a relation to-and-fro between pZ and rZ (i.e. p2 and r in Figure 7.8), which is again of type Moving. For both inferred relations, the resulting time interval is the minimum start- and the maximum end-point in time of the involved relations’ intervals. These two inferred relations are combined with a third rule also shown in
168
7 Framework Evaluation
Figure 7.9: Visualization and relations of the scenario shown in Figure 7.7.
Listing 7.5, which is fired every time (i) the to-and-fro holds during the around-cw relation, (ii) the rules are fired by the Relations Repository and (iii) the intervals of the relations changed. Its consequence, which can be either the inference of a new relation or the triggering of an application-level action and is not specified in this example, is thus not executed unless all three conditions are fulfilled. As in Section 7.2.2, the maintenance rule for merging equal relations – which is shown in Listing 5.1 – must additionally be contained in the rule base, as temporal interval relations could not be recognized otherwise (cf. Section 5.2). Listing 7.5: Rules for the scenario shown in Figure 7.8. rule " moving - to - and - fro " when Relation ( name == " DynDistance " , value == " away " , pZ:pZoI , rZ:rZoI , i1:interval ) Relation ( name == " DynDistance " , value == " toward " , pZoI == pZ , rZoI == rZ , i2:interval ) eval ( i2 . after ( i1 )) then insert ( new Q ua l it at i ve Re la t io n ( " Moving " , " to - and - fro " , pZ , rZ , i1 . merge ( i2 ))); end
7.3 Comparison and Discussion
169
rule " spatiotemporal - awareness " when Relation ( name == " Moving " , value == " around - cw " , pZ1:pZoI , rZ:rZoI , i1:interval ) Relation ( name == " Moving " , value == " to - and - fro " , pZ2:pZoI , rZoI == rZ , i2:interval ) eval ( i2 . during ( i1 ) && rZ . equals ( " R:PhysicalShape " )) then // any application - specific consequence end
Figure 7.10 shows a screenshot of the 2-D visualization with the Zones-ofInfluence P1:Centroid, P2:PhysicalShape and R:PhysicalShape representing the artifacts of Figure 7.8, as well as the relations – which have been recognized and inferred by moving the former two as described in the scenario above – demonstrating the correct functionality of the framework. In the figure, just those relations which are relevant for the scenario are visualized, although many more are contained in the repository like for example relations between P1:Centroid and P2:PhysicalShape. It should be noted that recognizers can be implemented so as to recognize relations between certain Zones-of-Influence only, which can significantly improve the efficiency if just a few of the possible relations among artifacts are required by the application. In the given example, the relation to-and-fro with the interval [82, 93] holds during around-cw with the interval [59, 108], for which reason the rule spatiotemporal-awareness is being executed. These two examples showed the feasibility of (i) recognizing and using qualitative dynamic relations for changing spatial situations, which are characterized by changing self-descriptions of artifacts, and (ii) inferring qualitative spatiotemporal relations with the Zones-of-Influence Framework, which allows for considering time-series of qualitative relations at the application level. When using temporal interval relations however, two more things have to be considered. First, if a temporal after and before relation should take into account successive qualitative relations only, a further maintenance rule is required which removes all relations that are equal (i.e. having the same name, value, primary and reference ZoI), except for the most recent one. Second, as already mentioned in Section 3.2.5, the interval of an inferred relation has to be explicitly defined in the rule’s consequence part, and it thus depends on the application demands and can be defined differently in other scenarios.
7.3 Comparison and Discussion In this chapter, the Zones-of-Influence Framework and its underlying concepts have been evaluated with multiple spatially aware applications for spontaneously
170
7 Framework Evaluation
Figure 7.10: Visualization and relations of the scenario shown in Figure 7.8.
interacting digital artifacts, which are situated in physical space and whose spatial properties – and thus their spatial relations with respect to other artifacts – change over time. Our approach for spatial awareness proposed in the present thesis is based on the quantitative abstraction of spatial properties with Zones-of-Influence, their ad-hoc exchange with artifacts in communication range via so-called selfdescriptions as well as the recognition and inference of qualitative relations from them, which enables artifacts to autonomously adapt to changing spatial situations at runtime. In order to allow for a comprehensive evaluation of the framework, the application scenarios presented in Section 7.2 have been chosen such that they require different spatial contexts and abstractions on the one hand, and that the relevancy of quantitative abstractions of spatial properties and qualitative abstractions of spatial relations for their development differs on the other hand. They have all been developed with the design process presented in Section 7.1, which is based on these two types of spatial abstractions and should facilitate the development of spatially aware applications with the Zones-of-Influence Framework. A comparison of the applications with respect to (i) how they have been imple-
7.3 Comparison and Discussion
171
Application implementation additional implementation without rules hardware prototype simulation only (i.e. no spatial sensors) Zones-of-Influence use of static and dynamic ZoIs dynamic [po]sition, [di]rection and [sh]ape used spatial sensors ([ph]ysical/[so]ftware) Relationship recognition recognition of static relations recognition of dynamic relations quantitative and qualitative recognizers Rule-based inference inference of qualitative relations inference of application-level actions use of multiple rules
Spatiotemporal Awareness
Focus/Nimbus Awareness
Vibro-Tactile Space Awareness
Table 7.2: Framework aspects covered by implemented spatially aware applications.
x xa x x po, di, sh ph, so
x po, di ph
x po
x
x
xb
x
x x xc
x
x x
xd x
a embeds
sensors and actuators only, the framework runs on an external computer use of quantitative relations c a single quantitative is used by multiple qualitative relation recognizers d consideration of qualitative temporal interval relations b application-level
mented, (ii) which types of Zones-of-Influence are used, (iii) which relations are recognized and how, and (iv) whether application-level actions or qualitative relations are inferred with rules, which are key aspects of the general architecture for spatial awareness presented in Section 2.5, is given in Table 7.2. The first one is the vibro-tactile space awareness application (cf. Section 7.2.1), with which the Zones-of-Influence Framework has been evaluated using a hard-
172
7 Framework Evaluation
ware prototype of a vibro-tactile belt that embeds physical sensors and actuators. The vibrator elements are activated depending on recognized static orientation, distance and topological relations, whereas for orientation and distance relations – as in the other scenarios also – both a quantitative and a qualitative recognizer are used at a time. For the recognition of relations, static ZoIs as well as a dynamic ZoI whose position and direction depend on a physical and whose shape depends on a software sensor are used. The mapping of logical combinations of relations to actuator actions has been implemented both with and without using rules. For the focus/nimbus awareness application (cf. Section 7.2.2), a static freeform-shaped and a dynamic point-shaped ZoI with position and direction given by the physical sensor of the first application have been used. Static relations are also acquired with the same recognizers used above already, but with different parameterizations. In contrast to the first application scenario, a rule is used for both inferring relations that can be queried from applications as well as for triggering application-level actions. Finally, two scenarios for spatiotemporal awareness (cf. Section 7.2.3) have been implemented in order to evaluate the recognition of qualitative dynamic relations as well as the inference of qualitative spatiotemporal relations with the framework, which allows for representing spatial changes and is not covered by the previous two applications. For the evaluation, just simulation and no spatial sensors have been used, whereas the positions of the primary artifacts are considered dynamic (i.e. manipulated in the 2-D visualization, which leads to changes of the respective self-descriptions as described in Section 6.2.1) and all other ZoIs are static. The quantitative static distance recognizer is utilized by both a qualitative static and dynamic distance recognizer, and multiple rules are used for inferring qualitative spatial and spatiotemporal relations. With these implemented spatially aware applications, we have shown the feasibility and quality of the Zones-of-Influence Framework and the awareness concepts described in previous chapters of this thesis, in that the recognition of relations from Zones-of-Influence as well as the inference of new relations and application-level actions have been performed correctly for all test scenarios. As can be seen from the table, the focus of the first application is on quantitative abstractions of spatial properties with different types of ZoIs, whereas just a few simple static relations are recognized and none are inferred. In contrast, many different types of relations – including dynamic and spatiotemporal relations – are recognized and inferred with rules in the spatiotemporal awareness application, which on the other hand requires just very simple quantitative spatial abstractions without shapes and spatial sensors. The design process we used for it appeared to be helpful for a more structured application development in our scenarios, and we expect the same for applications involving more and different ZoIs, relations
7.3 Comparison and Discussion
173
and rules. However, its steps are not considered to be strictly separated from each other, as it may be necessary to make assumptions about the Zones-of-Influence in the first step for example, which in turn may be affected by the available sensors. In the remainder of this chapter, findings of the evaluation along the awareness aspects distinguished in Table 7.2 are discussed. First, the quantitative representation of spatial properties with Zones-of Influence has shown to be an adequate abstraction from the multitude of physical objects to those which are actually relevant for the application in the presented scenarios. They allow for developing applications at a higher level of abstraction by considering just the defined Zonesof-Influence, and contribute to the efficiency of the framework as relations are recognized between two such ZoIs at a time only. The implementation of application scenarios requiring different spatial properties demonstrated the flexibility of the framework in that arbitrary spatial regions – be it the physical extensions of objects or any other region which is relevant for the application – can be represented with Zones-of-Influence and computationally treated the same way. Moreover, with the proposed self-descriptions and the current implementation of the framework, it is easily possible to make spatial properties of Zones-of-Influence dependent on sensor data. In this regard, it has been shown that fundamentally different spatial sensors can be used for acquiring their dynamic positions, directions and shapes on the one hand, and that they are reusable building blocks by using the same sensor data accessor component in the first two applications on the other hand. The second aspect are qualitative spatial relations, which constitute the basic building blocks for developing spatially aware applications in this thesis. From the presented scenarios, it can be seen that they are adequate abstractions for applications running on autonomously operating embedded systems which do not require or have no access to quantitative relationship representations, but distinguish a finite and rather small number of spatial situations with respect to the artifact’s environment instead. The used relations are thus specific to the application and provide a layer of abstraction which has shown to ease the application development by separating from low-level spatial representations. In the first scenario for example, the eight qualitative orientation relations are determined by the available vibrator elements, whereas just two orientation values are sufficient in the second scenario to distinguish whether a point of interest is within or out of a person’s field of view. A further example is the dynamic distance relation for representing that an artifact is approaching another artifact in the third scenario, which is however irrelevant for the first two scenarios as the application behavior is defined by static situations only. From the developed applications as well as from further scenarios we considered for realizing with the Zones-of-Influence Framework, it
174
7 Framework Evaluation
has been observed that static seem to be more relevant than dynamic relations, and that directional relations are of minor interest compared to positional and topological relations. We also observed that point-shaped abstractions of objects as spatial primitives for recognizing both orientation and distance relations are often sufficient, which means that no shape information and thus less computational resources are required. With regard to orientation relations however, it has to be considered that the recognition from the anchor points of two Zones-of-Influence without taking into account their shapes may lead to cognitively wrong results in the case they are nearby, as has been seen in the first application scenario. For the recognition of qualitative spatial relations, so-called relation recognizers are used. As they are loaded dynamically by the application, just those relations which are actually required by running applications are recognized, which enables runtime adaptivity on the one hand and improves the efficiency of the Zones-ofInfluence Framework on the other hand. Additionally, relation recognizers notify just about changed relations and can be implemented to acquire relations between particular Zones-of-Influence only, which both reduce the calculations for registered listeners and thus contribute to the overall efficiency, too. By developing multiple applications which use very different qualitative spatial relations, we have demonstrated the flexibility of the framework with regard to the provision of application-specific spatial abstractions. An important issue is the reuse of recognizers in different scenarios, in particular (i) the combination of quantitative and qualitative recognizers which allows for providing different qualitative abstractions based on the same quantitative relations, as well as (ii) the parametrization of qualitative recognizers in order to provide relations of a certain type whose granularity and semantics are defined by the application. This has proven to significantly ease the application development, as for example all recognizers of the second scenario could be reused from the first one, and those for both qualitative static and dynamic distance relations in the third scenario use the same quantitative static distance recognizer. A further important aspect is the dimensionality of represented relations; although 3-D relations could be used with the framework by implementing respective recognizers, just 2-D relations are considered in the presented evaluation as they have shown to be sufficient and can be recognized with less computational resources. In each of the evaluated application scenarios, one or more production rules have been used for (i) inferring high-level relations or (ii) triggering applicationlevel actions. The use of rules allows for describing the behavior of spatially aware applications in a declarative and more intuitive way, which has been demonstrated in the evaluation scenarios where actions had to be performed once certain combinations of spatial and temporal relations occurred. From the different implemented
7.3 Comparison and Discussion
175
applications, it can also be seen that the declarative combination of qualitative relations provides a high degree of flexibility with respect to the application-specific inference of conclusions. In the first scenario, two implementation variants have been evaluated, namely with and without using rules. While the former has shown the advantages that the application-level behavior is not hard-coded (and can thus be changed by simply rewriting the rules) and the evaluation of relationship combinations does not have to be implemented, it has – compared to the latter – the disadvantages that it cannot react immediately on relationship changes but adapt to spatial changes at discrete points in time only, which is due to the fact that the rules are periodically fired by the Relations Service, and that the spatial resolution is lower because of the sole use of qualitative abstractions and the involved discretization of space. The mentioned periodicity has to be considered with regard to the inference of conclusions, as just the most recent qualitative relations recognized within an update period – i.e. the period between two points in time where recognized relations are stored and rules are fired – are inserted in the repository. With the spatiotemporal awareness application, we have finally evaluated the inference of spatial and spatiotemporal relations with multiple rules, which has shown to facilitate the application development by allowing to split complex spatial conditions in two or more simpler rules and reuse inferred spatial relations in other rules. An inherent problem of the rule-based approach for inferring spatiotemporal relations however is the fixed nature of spatial and temporal conditions, which means that spatiotemporal patterns must occur exactly as defined or they are not recognized otherwise; hence, from our experience gained during the evaluation, we consider the use of rules adequate whenever the patterns are precisely known and do have a reasonable complexity. Finally, it should be noted that using rules for inferring conclusions is complementary but independent from the other concepts presented in this thesis, in particular the quantitative and qualitative abstractions of space.
8 Conclusion What we call the beginning is often the end. And to make an end is to make a beginning. The end is where we start from. Thomas Stearns Eliot, 1888 - 1965
The pervasion of our environment with computing technologies has already become reality, and the trend for more and more embedded systems which are invisibly integrated in everyday objects, equipped with sensors and actuators, ad-hoc networked among each other and available anytime and everywhere is supposed to continue in the future. With an increasing number, complexity and heterogeneity of such digital artifacts, it becomes important that they autonomously – this is without centralized control – manage their individual behavior by adapting to changing contexts, and self-organize spontaneous configurations of multiple artifacts in order to achieve collective behavior, so as to unburden humans from their configuration and management and allow for a more implicit interaction with such computationally enriched environments. A precondition for this is that artifacts are able to assess and adapt to their current situation or context in a semantically meaningful way; thus, they must be able to sense their context, process and reason about it, and share it with other artifacts in range. This allows for developing applications that not only depend on the individual context of a digital artifact, but rather on the co-existence of multiple such artifacts and the contextual relations among them. Important context information are an artifact’s spatial properties such as position and shape on the one hand and spatial relations between artifacts such as distance and orientation on the other hand, which is because computationally augmented real-world objects are situated in physical space and may change their spatial contexts over time. The present thesis is thus concerned with the spatial awareness – i.e. an awareness about spatial context information – of autonomous embedded systems, which we consider to play a vital role in future pervasive computing systems, and focuses on both static relations between the position, direction and spatial extension of two artifacts at a time as well as their changes over time. With digital artifacts that are
178
8 Conclusion
able to autonomously adapt their behavior to the spatial contexts of surrounding ones, the development of spatially aware applications for spontaneously interacting artifacts becomes possible. A key issue in this regard are spatial abstractions, which separate from details of low-level sensor information that are not relevant for a certain application. Hence, the focus of this work is on recognizing, representing and reasoning about spatial relations between autonomous digital artifacts using qualitative abstractions that are closer to human concepts of space, as well as their provision and utilization at the application level. Summing up, its main contributions can be stated as follows: • The development of an architecture for providing spatial awareness to autonomous digital artifacts, which is based on an ad-hoc exchange of spatial context information among them • The investigation and development of concepts for representing spatial properties of artifacts and recognizing relations between two of them at a time by means of quantitative and qualitative abstractions of space • The development of a spatial calculus for compositional reasoning about qualitative orientation and distance relations • An algorithm for inferring and distributing spatial relations among spontaneously interacting artifacts over multiple hops, which makes use of composition tables or the properties of binary relations • A rule-based approach which allows to infer conclusions – i.e. new relations or application-level actions – from a history of qualitative spatial relations and their changes over time • The prototypical implementation of the spatial awareness architecture with a modular and flexible software framework, as well as its evaluation by means of multiple spatially aware applications The scope of this thesis is on software concepts for spatial awareness, for which reason neither cognitive and linguistic aspects of spatial abstractions nor embedded systems and communication technologies have been addressed in detail. The remainder of this chapter is structured as follows. First, in Section 8.1, the thesis and in particular its contributions and limitations are summarized and discussed. Section 8.2 finally provides a critical discussion of the whole work and gives an outlook on future research issues.
8.1 Summary
179
8.1 Summary First, in Chapter 1, the thesis is motivated and a problem statement is presented, which was basically the investigation and development of concepts and architectures for making autonomous embedded systems aware of qualitatively abstracted spatial relations and their changes over time. In this regard, a particular focus was on the development and implementation of respective software solutions as well as their use in spatially aware real-world applications. In order to achieve such spatial awareness of an autonomous embedded system, which is the ability to determine and use its spatial context, several aspects which are elaborated in Chapter 2 can be identified. Central issues are the acquisition, abstraction and use of context information, which contribute to the autonomous operation of embedded systems in the background. First, the foundations of context-awareness and architectures for context-aware systems are discussed, and the importance of spatial context information in this regard is emphasized. The fundament of our research are spatially distributed real-world objects that are augmented with autonomous embedded systems, which comprise sensing, computing, actuating and wireless communication capabilities in order to autonomously provide contextualized services by sharing information among each other. We adopt the ad-hoc exchange of structured self-descriptions between artifacts upon coming into communication range, which contain – among other things – their current spatial context, the pairwise comparison of an artifact’s own with received self-descriptions and the triggering of according actions. The exchange of acquired contexts among spontaneously interacting artifacts contributes to their self-organization, which is covered in the present thesis insofar as the acquisition and sharing of spatial contexts as well as the development of spatially aware applications that are based on contextual local interactions among them are addressed. We consider two types of spatial context information relevant for this work, which are (i) the spatial properties position, direction and extension in space and (ii) spatial relations between the properties of two artifacts at a time, emphasize their importance for the autonomous adaptation to changing spatial situations and provide a classification in different categories in order to narrow down the scope of this thesis. According to most context-aware systems, a multi-layered architecture for spatial awareness is proposed, which is responsible for (i) acquiring spatial contexts from sensors and surrounding artifacts via wireless communication, (ii) recognizing, representing and reasoning about binary qualitative spatial relations and (iii) providing services for the development of spatially aware applications. In this regard, just sensors for quantitative static spatial properties with regard to a fixed external reference frame are considered. At the end of this chapter, re-
180
8 Conclusion
lated projects are surveyed and the present thesis is positioned against the most closely related ones in a comparing table. From our observations, spatial relations – and especially in their qualitatively abstracted form – are a little investigated topic in the pervasive computing field of research. Although several projects make use of them in either way, there seems to be none which provides flexible means for recognizing and representing arbitrary types of spatial relations as well as for exploiting those relations and their changes over time for the application development, which we have identified as research questions for our work. Accordingly, distinguishing features of this thesis include (i) the adaptive (i.e. as required by a running application) recognition and qualitative representation of spatial relations from shared context information and (ii) the proposed rule-based approach for reasoning about qualitative spatial and temporal relations, which have shown to facilitate the development of spatially aware applications in the evaluation. The representation of space is a precondition for processing it computationally, where we distinguish two types of spatial abstractions in Chapter 3. The first one is the quantitative representation of an artifact’s spatial properties with one or more of so-called Zones-of-Influence, which are – possibly changeable – explicitly defined geographic regions that are of interest for the application (e.g. the physical extension of an artifact or the area in which user interaction is possible) and exchanged among artifacts by using self-descriptions as described above. The second one is the qualitative representation of spatial relations between an artifact’s own and one of its received Zones-of-Influence at a time by means of discrete and human-readable symbols. We decided for qualitative abstractions mainly because (i) they facilitate the programmatic use of spatial relations by separating from irrelevant low-level details and providing them with application-specific semantics and in a more natural way that is closer to human concepts of space, and ii) for the same reasons allow for computationally less expensive processing and dealing with coarse spatial information. At the end of this chapter, a simple structure of self-descriptions for the ad-hoc exchange of both quantitative spatial properties and qualitative spatial relations is proposed and discussed. We regard the quantitative and qualitative abstractions of spatial contexts to be dependent on a particular spatially aware application, which is for reasons of flexibility and efficiency, and fundamentally influenced the spatial awareness architecture as well as its implementation with the Zones-of-Influence Framework described further below. This thesis restricts to binary relations which are defined between a reference and a primary Zone-of-Influence and characterized by their type (e.g. static orientation) and value (e.g. right), and they are distinguished by several aspects concerning their representation and recognition in Chapter 3. These include (i) the dimensionality of the represented space, whereas our focus is on 2-D representations
8.1 Summary
181
which are sufficient for many applications scenarios, (ii) the spatial primitives between which the relations are defined, where we use primarily points for positional and directional relations and spatial extensions for topological relations, and (iii) the reference frames with respect to which they are defined, where we distinguish between intrinsic and extrinsic ones. According to our focus on the spatial properties position, direction and spatial extension as well as their changes over time, we present a classification of qualitative relations in (i) positional (i.e. orientation and distance), (ii) directional and (iii) topological relations on the one hand, and in (i) static (i.e. given at a certain point in time) and (ii) dynamic (i.e. representing the degree of change at a point in time) relations on the other hand, and discuss them by means of exemplary relations with common semantics and granularity. A distinguishing feature of the present thesis is the consideration of time series of qualitative spatial relations that are given at certain points in time, which allows artifacts to take into account how their spatial situation is changing over a period of time. According to the qualitative abstraction of spatial relations, we decided for qualitative temporal relations between discrete time intervals in which they hold, which are again considered to be more natural for humans and helpful whenever exact temporal relations are not required or irrelevant for the application. By interpreting a temporal combination of spatial relations as a high-level spatiotemporal relation and assigning a respective semantically meaningful name to it as well as a time interval in which it holds, such a relationship combination can in turn be temporally combined with other spatial or spatiotemporal relations. Based on the representation of spatial relations discussed before, Chapter 4 investigates the inference of knowledge from binary relations between autonomous digital artifacts – more precisely, between their Zones-of-Influence – which are out of communication range. The challenge here is our limitation to qualitative relationship abstractions, namely without using any quantitative information in the reasoning process, which we consider not only useful for resource-constrained embedded systems but also for the case that no precise quantitative information is required or available. First, approaches for qualitative spatial reasoning are surveyed, where our scope is on (i) exploiting the properties reflexivity, symmetry and transitivity of binary relations and processing the respective closures, and on (ii) using composition tables for the inference of spatial relations between two artifacts via a third one at a time. While it is sufficient for the former to assign the respective properties to each relation value, a table which stores the possible alternative relations (i.e. referred to as compound relations) for each pair of base relations has to be developed for the latter. However, as for many relation types none of the three properties are fulfilled and closures can be processed for equal relation values only, our focus is on compositional reasoning with which these restrictions can be over-
182
8 Conclusion
come. Therefore, a composition table for positional relations which are given with respect to an extrinsic reference frame is proposed, as the composition of intrinsic relations leads to many disjunctions of possible base relations and is thus practically less relevant. Although composition tables allow for efficient reasoning by simple table lookups, it has to be noted that their construction is a time-consuming process and changes in the relation semantics require a reconstruction of the table. The main contribution of this chapter is an approach that provides a group of artifacts, which are either connected directly or via other artifacts, with an awareness about their relations to each other. A possible solution would be to exchange selfdescriptions between each pair of artifacts, for which we consider simple flooding most feasible as other variants mentioned in Chapter 4 either do not guarantee that all artifacts are reached or require information about the network topology. However, the broadcasting of self-descriptions leads to a high amount of traffic and relations, and hence requires more computational resources. The general idea of our approach is that artifacts recognize qualitative relations to others within communication range only by comparing their own with received quantitative spatial properties, and include them – in addition to the spatial properties – in their selfdescriptions, which in turn allows artifacts to infer relations to others out of range by using qualitative spatial reasoning techniques. The proposed approach exhibits self-organizing behavior in that artifacts autonomously, in an ad-hoc manner and by using locally perceived information only, establish an awareness about their spatial relations among each other. We present a novel algorithm for inferring and distributing qualitative spatial relations in Chapter 4, and demonstrate its function by means of examples. It is based on the following four steps: (i) an artifact broadcasts its self-description, whereupon (ii) each receiving artifact recognizes spatial relations to the broadcasting one, (iii) infers additional relations from already known and newly recognized or received ones by means of compositional reasoning, and (iv) again broadcasts its self-description – which also contains the inferred relations – if none has been received yet or new relations have been acquired. By delaying broadcasts, that is not performing them for every change upon receiving a self-description, the induced traffic can be significantly reduced. Another optimization concerns the acquisition of more accurate results, for which inferred compound relations are intersected with possibly known relations between the same artifacts. For evaluation purposes, we simulated three variants of this algorithm which differ in how the composition operation is implemented – i.e. using (i) separate composition tables for orientation and distance, (ii) a single composition table for combined orientation and distance relations as well as (iii) by processing the transitive closure of orientation relations – and compared them with flooding for the exchange of
8.1 Summary
183
self-descriptions between all artifacts. The four approaches are evaluated with five network topologies that are assumed to be static until the algorithm terminates, and with regard to their accuracy and the induced traffic, whereas neither wireless communication protocols including aspects such as delays or packet losses nor conflicting and inaccurate sensor data are considered. In comparison to the flooding approach, the simulation showed that (i) the traffic is in any case reduced by using qualitative reasoning, (ii) a combined composition of orientation and distance leads to more accurate results than their separate composition or by exploiting the transitivity property, (iii) using delayed broadcasts significantly reduces the induced traffic without having an impact on the achieved accuracy, (iv) the accuracy decreases with an increasing number of nodes, and (v) the percentage-wise reduction of traffic compared with flooding is in all simulated scenarios higher than the percentage-wise loss of accuracy. In Chapter 5, we propose a rule-based approach for reasoning about qualitative spatial relations and their changes over time. Rules allow for declarative programming, which provides a higher degree of flexibility and is considered more natural and intuitive for humans than imperative approaches. After a short survey on related work about the use of rules for defining the behavior of context-aware systems, we argue for using production rules which is because they enable the specification of actions which are to be executed whenever a certain condition becomes true due to an update on a database, and are thus suitable for maintaining and reasoning about a spatial model of relations. In this thesis, the conditions are logical combinations of both qualitative spatial relations stored in the repository and qualitative temporal relations between the intervals in which they hold. Resulting actions can be (i) the inference of new high-level relations and their insertion in the repository or (ii) the inference of application-level actions, and they are executed once the respective rule’s condition becomes true (i.e. a certain defined pattern on the history of relations is fulfilled). We distinguish two types of rules: (i) inference rules for inferring applicationlevel actions as well as relations which can be used in other rules, and (ii) maintenance rules for performing maintenance operations on the repository such as limiting the length of the stored history of relations. Inference rules, which combine qualitative spatial and temporal relations, are explained by means of several examples. In this regard, a classification for combinations of qualitative spatial and temporal relations, depending on whether static or dynamic relations are combined and if temporal interval relations are considered, is given. Finally, a functional proof-of-concept evaluation with an open-source rule engine as well as a motion tracking system and two wireless tracking devices for the primary and reference Zones-of-Influence is presented. Thereby, both static and dynamic orientation,
184
8 Conclusion
distance and directional relations in two dimensions are periodically recognized and qualitatively abstracted for an exemplary movement, and new relations are inferred from them and inserted in the repository; topological relations are not considered at this point. The evaluation showed the feasibility of recognizing different types of qualitative static and dynamic spatial relations from sensor data on the one hand, and inferring new qualitative spatiotemporal relations by using production rules on the other hand. A software architecture that provides spatial awareness to autonomous embedded systems has been developed and prototypically implemented with the flexible and modular Zones-of-Influence Framework presented in Chapter 6, thus making it possible to build and perform experiments with spatially aware applications. It is based on the multi-layered architecture proposed in Chapter 2, and implements the concepts for spatial awareness elaborated in previous chapters with a clear separation of concerns. Key requirements for the framework architecture were the acquisition, representation and exchange of quantitative spatial properties and qualitative spatial relations, as well as the rule-based inference of conclusions from them. The following components, which constitute the core of the architecture, have been implemented by the time of this writing: (i) the Digital Artifact Service for discovering artifacts in range and exchanging self-descriptions with them, which is available in prototypical versions for loading self-descriptions from files as well as for creating and manipulating them with a graphical user interface at the moment, (ii) the Zones-of-Influence Service for acquiring and maintaining an up-to-date model of the static and dynamic Zones-of-Influence of known artifacts, and (iii) the Relations Service which is responsible for recognizing, representing and reasoning about qualitative relations between Zones-of-Influence. A detailed description of these components including their interfaces and implementation-specific issues is presented afterwards. Notable features in this regard are the dynamic adaptability of (i) parsers for the contents included in selfdescriptions, (ii) sensor data accessors for accessing spatial properties of dynamic Zones-of-Influence and (iii) relation recognizers for recognizing certain types of relations according to the respective application needs, as well as the event-based architecture which allows to recognize relations just for changed or new Zonesof-Influence and update the repository with changed or new relations only. Performance tests, which are presented at the end of Chapter 6, showed that the framework runs stable and how much time it requires for (i) acquiring a model of Zones-of-Influence from a varying number of new or changed self-descriptions, (ii) recognizing different types of relations out of them and (iii) inserting the relations in the repository as well executing a varying number of rules which infer and insert new relations.
8.2 Outlook
185
Chapter 7 finally presents a scenario-based evaluation of the Zones-of-Influence Framework. First, a relation-centric design process for building spatially aware applications with the framework is proposed, which comprises the following five steps: (i) identification of application-relevant spatial relations, (ii) definition of Zones-of-Influence needed for their recognition, (iii) selection of sensors for determining the spatial properties of dynamic ZoIs, (iv) selection of relation recognizers and (v) modelling of the application behavior by using rules for example. Multiple application scenarios requiring different spatial contexts and abstractions have been realized with the design process, which has shown to be helpful for a more structured development. The application scenarios have been chosen to cover a wide spectrum of framework aspects, mainly concerning (i) their implementation, (ii) the used Zones-of-Influence, (iii) which relations are recognized and how as well as (iv) the rule-based inference of conclusions. The implemented applications have proven the feasibility of the framework in that the relationship recognition and the inference of new relations or applicationlevel actions have been performed correctly for all scenarios. Moreover, they have shown the adequacy of (i) quantitative and qualitative abstractions of spatial properties and relations and (ii) the proposed rule-based approach for the development of spatially aware applications which distinguish a finite number of spatial situations, the flexibility of the framework with regard to utilizing different applicationspecific spatial contexts and abstractions as well as the reusability of components implementing spatial abstractions, which in conjunction facilitates the application development. The use of spatial abstractions that are specific to a certain application not only eases the application development, but also improves the efficiency by dynamically adapting them to the application’s needs. We noticed that static topological and positional relations seem to be more important than others, including dynamic spatial and inferred spatiotemporal relations, and that both twodimensional relations and point abstractions of physical objects are sufficient for many applications. The declarative combination of qualitative relations has shown to provide a high degree of flexibility with respect to the application-specific inference of conclusions; however, an observed drawback of using rules is the fixed nature of spatial and temporal conditions, why the patterns which should be recognized must be precisely known and occur exactly as defined in the rules.
8.2 Outlook The present thesis contributes to the research field of context-aware computing. It presents concepts and an architecture for the spatial awareness of spontaneously
186
8 Conclusion
interacting autonomous computing systems that are embedded in everyday physical objects, whereas our main focus was on the investigation of how qualitatively abstracted spatial and temporal relations can be used at the application level. Proof-of-concept implementations showed the feasibility of the presented approach; in this regard, primarily software concepts for spatial awareness rather than related hardware technologies have been addressed. With regard to the quantitative and qualitative abstractions of spatial context information, cognitive and linguistic aspects have not been considered in detail. A weak point of the thesis is the scenario-based evaluation of our approach. First, although most of the developed concepts have been covered with the scenarios, the framework still does not run on an embedded computing system which is capable of wireless ad-hoc communication for exchanging self-descriptions. Hence, no evaluation has been conducted which takes into account (i) the limited processing and storage resources of embedded system – which may, for example, not allow to recognize topological or distance relations between complex freeform-shaped Zones-of-Influence – as well as (ii) aspects of wireless communication like delays and packet losses. Second, just scenarios involving a few different ZoIs, recognized or inferred relations and rules have been evaluated; in order to show the feasibility for large real-world applications however, we consider it necessary to build more complex scenarios with tens of ad-hoc networked digital artifacts running the Zones-of-Influence Framework. Third, in all evaluations with physical sensors throughout this thesis, just a single and highly accurate spatial sensor for position and direction has been used, which however is not feasible with regard to outdoor scenarios covering a bigger spatial area and thus having to cope with comparatively inaccurate and less precise sensors. Fourth, the inference and distribution of relations discussed in Chapter 4 has not been evaluated with the framework yet, but simulation has been used instead. Moreover, no changing topologies and conflicting spatial information received from different artifacts – which are common in real-world scenarios – have been taken into account in this regard. The fifth and last point concerns the general problem that each artifact is assumed to have a-priori knowledge about the exchanged spatial information in order to be able to parse and interpret spatial properties and relations contained therein. The presented concepts have been implemented with the Zones-of-Influence Framework, which has shown to facilitate the development of spatially aware applications for autonomous embedded systems. However, as our main concern was on evaluating the recognition, representation and reasoning about qualitative spatial relations, several implementation-specific issues have not been addressed yet. With regard to future work, we consider three of them particularly interesting:
8.2 Outlook
187
• Porting the framework to an embedded hardware/software platform as for example the one presented in [FHdSR+ 07], which – together with less accurate sensors – would allow for evaluating the spatial awareness concepts presented in this thesis under more realistic conditions and by using adhoc wireless communication for discovering artifacts and exchanging selfdescriptions between them. • The integration of spatial sensors that provide (i) relationship information directly, like e.g. that of [HKG+ 05], which we consider challenging primarily with respect to their accuracy and the provision of identity information, as well as (ii) dynamic spatial properties such as acceleration or velocity. A related issue is the conversion between the reference systems of different sensors, for example by using logical sensors for converting between two of them at a time. • An extension with standardized markup languages for representing spatial properties and relations; in this regard, it would be necessary to investigate which languages are suitable and how different ones could be used in combination. Besides software and hardware aspects regarding the Zones-of-Influence Framework, we identified the following issues concerning our approach and general concepts for spatial awareness worth to address, which of course also affect the framework implementation insofar as they may require corresponding extensions and modifications: • The exchange of time-series of relations with self-descriptions, in order to be able to take into account the relationship histories of newly discovered artifacts for reasoning purposes. However, as artifacts are autonomously operating units which do not have sense of a global time, a synchronization of their clocks would be necessary. A related issue is to investigate how well future relations could be predicted from a discrete time series of past and current relations, as for example by using the context prediction approaches proposed in [Fer03] and [May04], which would allow for a proactive adaptation to future spatial situations. • A support for representing further types of relations, in particular unary and ternary relations as well as relations between groups of Zones-of-Influence, which we consider challenging both with regard to their representation in the framework and the rule-based reasoning about them. The same holds for the use of compound relations, which are disjunctions of alternatively
188
8 Conclusion
possible relations, and multiple relations of the same type that exist between two Zones-of-Influence at the same point in time. • The investigation of combined reasoning about static and dynamic relations, which we expect to enable the inference of new conclusions. For example, knowing that a primary object is right and moving rightward implies that it is not moving toward the reference object, which means that the possible dynamic distance relations are constrained by the static and dynamic orientation relations. • In Section 3.2.2, we mentioned that point abstractions of Zones-of-Influence which are nearby and do not have a uniform size may not be suitable for recognizing positional (i.e. orientation and distance) relations between them. For future work, it would thus be worth to investigate in which context and how to take into account the reference artifact’s spatial extension and direction in order to recognize cognitively meaningful relations (cf. [CFH97, Her94]). • In the current implementation of the framework, the intervals of recognized relations contained in the repository are periodically extended every time the rules are executed. Although the relation intervals are kept up-to-date therewith, it consumes a considerable amount of processing power as each change of a relation’s interval causes this relation to be matched against the rules. With regard to future work, it would be worth to investigate alternative solutions which for example update qualitative relations in the repository just when they start or end. • As discussed in Section 2.3, the concepts and solutions for the acquisition and local exchange of spatial context information developed within this thesis are considered valuable for their self-organization in space. Based on the present work on spatial awareness, the next step would be to investigate means for self-organizing behavior which relies on qualitatively abstracted spatial and temporal relations among digital artifacts. For future computing systems, which are expected to be increasingly embedded in our everyday environment, operate more autonomously and with which humans will interact in a more natural and implicit way, we consider the utilization of spatial contexts – and in particular of spatial relations between co-located artifacts – to play a significant role. Advances in location and inertial sensing technologies as well as their recent integration in everyday objects like mobile phones and notebook computers contribute to this vision, and there is strong evidence that
8.2 Outlook
189
the technological progress will include even non-computational physical objects which become augmented with means for assessing their spatial situation in the future. The present thesis makes a contribution in this regard by focussing on the abstraction and use of spatial and spatiotemporal relations, and providing concepts and software solutions that enable and facilitate the development of spatially aware applications for autonomous embedded systems.
Bibliography [AAC+ 00]
Harold Abelson, Don Allen, Daniel Coore, Chris Hanson, George Homsy, Thomas F. Knight, Radhika Nagpal, Erik Rauch, Gerald J. Sussman, and Ron Weiss. Amorphous computing. Communications of the ACM, 43(5):74–82, 2000.
[ACH+ 01]
Mike Addlesee, Rupert W. Curwen, Steve Hodges, Joseph Newman, Pete Steggles, Andy Ward, and Andy Hopper. Implementing a sentient computing system. IEEE Computer, 34(8):50–56, 2001.
[AEG94]
Alia I. Abdelmoty and Baher A. El-Geresy. An intersection-based formalism for representing orientation relations in a geographic database. In Proceedings of the 2nd ACM Workshop on Advances In Geographic Information Systems, Workshop at CIKM 1994, pages 44–51. ACM Press, December 1994.
[AGH+ 02]
Larry Arnstein, Robert Grimm, Chia-Yang Hung, Jong Hee Kang, Anthony LaMarca, Gary Look, Stefan B. Sigurdsson, Jing Su, and Gaetano Borriello. Systems support for ubiquitous computing: A case study of two implementations of Labscape. In Proceedings of the 1st Int’l Conference on Pervasive Computing, Pervasive 2002, volume 2414 of LNCS, pages 30–44. Springer, August 2002.
[Aie02]
Marco Aiello. Spatial Reasoning – Theory and Practice. PhD thesis, University of Amsterdam, 2002.
[All83]
James F. Allen. Maintaining knowledge about temporal intervals. Communications of the ACM, 26(11):832–843, 1983.
[All05]
OSGi Alliance. OSGi service platform core specification, release 4. http://www.osgi.org/Specifications, 2005.
[All07]
OSGi Alliance. About the OSGi service platform. http://www.osgi.org/wiki/ uploads/Links/OSGiTechnicalWhitePaper.pdf, 2007.
[ASSC02]
Ian F. Akyildiz, Weilian Su, Yogesh Sankarasubramaniam, and Erdal Cayirci. Wireless sensor networks: A survey. Computer Networks, 38(4):393–422, 2002.
[BBB+ 07]
Bruno Berstel, Philippe Bonnard, François Bry, Michael Eckert, and PaulaLavinia Patranjan. Reactive rules on the web. In Proceedings of the 3rd Int’l Summer School on Reasoning Web, volume 4636 of LNCS, pages 183–239. Springer, September 2007.
192
Bibliography
[BBC97]
Peter J. Brown, John D. Bovey, and Xian Chen. Context-aware applications: From the laboratory to the marketplace. IEEE Personal Communications, 4(5):58–64, October 1997.
[BCFM03]
Wolfgang Beer, Volker Christian, Alois Ferscha, and Lars Mehrmann. Modeling context-aware behavior by interpreted ECA rules. In Proceedings of the 9th Int’l Conference on Parallel and Distributed Computing, Euro-Par 2003, volume 2790 of LNCS, pages 1064–1073. Springer, August 2003.
[BF93]
Steve Benford and Lennart Fahlen. A spatial model of interaction in largescale virtual environments. In Proceedings of the 3rd European Conference on Computer-Supported Cooperative Work, ECSCW 1993, pages 109–124. Kluwer Academic Publishers, September 1993.
[BF04]
John A. Bateman and Scott Farrar. Modelling models of robot navigation using formal spatial ontology. In Proceedings of the 4th Int’l Conference on Spatial Cognition 2004: Reasoning, Action, Interaction, volume 3343 of LNCS, pages 366–389. Springer, October 2004.
[BGS01]
Michael Beigl, Hans-Werner Gellersen, and Albrecht Schmidt. Mediacups: Experience with design and use of computer-augmented everyday artefacts. Computer Networks, 35(4):401–409, 2001.
[BIK+ 04]
Cristian Borcea, Chalermek Intanagonwiwat, Porlin Kang, Ulrich Kremer, and Liviu Iftode. Spatial programming using smart messages: Design and implementation. In Proceedings of the 24th Int’l Conference on Distributed Computing Systems, ICDCS 2004, pages 690–699. IEEE CS Press, March 2004.
[BKMS00]
Barry Brumitt, John Krumm, Brian Meyers, and Steven Shafer. Ubiquitous computing & the role of geometry. IEEE Personal Communications, 7(5):41–43, 2000.
[BMK+ 00]
Barry Brumitt, Brian Meyers, John Krumm, Amanda Kern, and Steven A. Shafer. EasyLiving: Technologies for intelligent environments. In Proceedings of the 2nd Int’l Symposium on Handheld and Ubiquitous Computing, HUC 2000, volume 1927 of LNCS, pages 12–29. Springer, September 2000.
[BSHK06]
Urs Bischoff, Martin Strohbach, Mike Hazas, and Gerd Kortuem. Constraintbased distance estimation in ad-hoc wireless sensor networks. In Proceedings of the 3rd European Workshop on Wireless Sensor Networks, EWSN 2006, volume 3868 of LNCS, pages 54–68. Springer, February 2006.
[But02]
Bill Butera. Programming a Paintable Computer. PhD thesis, MIT Media Laboratory, 2002.
[CBGG97]
Anthony G. Cohn, Brandon Bennett, John Gooday, and Nicholas M. Gotts. Representing and reasoning with qualitative spatial relations about regions. In Spatial and Temporal Reasoning, pages 97–134. Kluwer Academic Publishers, 1997.
Bibliography
193
[CDL+ 04]
Simon Cox, Paul Daisey, Ron Lake, Clemens Portele, and Arliss Whiteside. OpenGIS Geography Markup Language (GML) implementation specification, version 3.1.1. Open Geospatial Consortium, Inc., http://www.opengeospatial.org/standards/gml, 2004.
[CFH97]
Eliseo Clementini, Paolino Di Felice, and Daniel Hernández. Qualitative representation of positional information. Artificial Intelligence, 95(2):317– 356, 1997.
[CFS+ 01]
Scott Camazine, Nigel R. Franks, James Sneyd, Eric Bonabeau, Jean-Louis Deneubourg, and Guy Theraula. Self-Organization in Biological Systems. Princeton University Press, Princeton, NJ, USA, 2001.
[CH01]
Anthony G. Cohn and Shyamanta M. Hazarika. Qualitative spatial representation and reasoning: An overview. Fundamenta Informaticae, 46(1-2):1–29, 2001.
[CHH02]
Srdjan Capkun, Maher Hamdi, and Jean-Pierre Hubaux. GPS-free positioning in mobile ad hoc networks. Cluster Computing, 5(2):157–167, 2002.
[CK00]
Guanling Chen and David Kotz. A survey of context-aware mobile computing research. Technical Report TR2000-381, Dartmouth College, Computer Science, Hanover, NH, November 2000.
[Con03]
connectBlue Bluetooth serial port adapter OEMSPA33i. http://www.connectblue.com, 2003.
[Dan99]
Peter H. Dana. Coordinate systems overview. University of Texas, Department of Geography, USA, http://www.colorado.edu/geography/gcraft/notes/coordsys/coordsys.html, 1999.
[DB92]
Paul Dourish and Victoria Bellotti. Awareness and coordination in shared workspaces. In Proceedings of the ACM Conference on Computer-Supported Cooperative Work, CSCW 1992, The Power of Simple Shared Workspaces, pages 107–114, November 1992.
[DBK+ 04]
Christian Decker, Michael Beigl, Albert Krohn, Philip Robinson, and Uwe Kubach. eSeal - A system for enhanced electronic assertion of authenticity and integrity. In Proceedings of the 2nd Int’l Conference on Pervasive Computing, Pervasive 2004, volume 3001 of LNCS, pages 254–268. Springer, April 2004.
[Dey00]
Anind K. Dey. Providing Architectural Support for Building Context-Aware Applications. PhD thesis, Georgia Institute of Technology, November 2000.
[Dey01]
Anind Dey. Understanding and using context. Personal and Ubiquitous Computing, 5(1):4–7, 2001.
[DFWW06]
Frank Dylla, Lutz Frommberger, Jan O. Wallgrün, and Diedrich Wolter. SparQ: A toolbox for qualitative spatial representation and reasoning. In Proceedings of the Workshop on Qualitative Constraint Calculi: Application and Integration, Workshop at KI 2006, pages 79–90, June 2006.
194
Bibliography
[Dut88]
Soumitra Dutta. Approximate spatial reasoning. In Proceedings of the 1st Int’l Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems, IEA/AIE 1988, pages 126–140. ACM Press, June 1988.
[Ecl08]
Eclipse. Equinox implementation of the OSGi R4 core framework specification. http://www.eclipse.org/equinox, 2008.
[ECPS02]
Deborah Estrin, David Culler, Kris Pister, and Gaurav Sukhatme. Connecting the physical world with pervasive networks. IEEE Pervasive Computing, 1(1):59–69, 2002.
[Ege89]
Max J. Egenhofer. A formal definition of binary topological relationships. In Proceedings of the 3rd Int’l Conference on Foundations of Data Organization and Algorithms, FODO 1989, volume 367 of LNCS, pages 457–472. Springer, June 1989.
[FAOH03]
Christian Floerkemeier, Dipan Anarkat, Ted Osinski, and Mark Harrison. PML core specification 1.0, September 2003.
[FDE+ 08]
Alois Ferscha, Jakob Doppler, Bernadette Emsenhuber, Manfred Hechinger, Dominik Hochreiter, Clemens Holzmann, Andreas Riener, Doris Zachhuber, Marquart Franz, Cornel Klein, Marcos dos Santos Rocha, Jelena Mitic de Useche, and Andreas Zeidler. Flexible Autonomic Context Technologies (FACT), research project between the Johannes Kepler University Linz, Department of Pervasive Computing, and Siemens AG Germany, Corporate Technology, CT SE 2. http://www.pervasive.jku.at, 2007–2008.
[Fer03]
Alois Ferscha. Coordination in pervasive computing environments. In Proceedings of the 12th IEEE Int’l Workshops on Enabling Technologies, WETICE 2003, pages 3–9. IEEE CS Press, June 2003.
[Fer07a]
Alois Ferscha. Informative art display metaphors. In Proceedings of the 4th Int’l Conference on Universal Access in Human-Computer Interaction, UAHCI 2007, volume 4555 of LNCS, pages 82–92. Springer, July 2007.
[Fer07b]
Alois Ferscha. Spatial awareness of digital artefacts. In Proceedings of the 11th Int’l Symposium on Distributed Simulation and Real Time Applications, DS-RT 2007, pages 3–4. IEEE CS Press, October 2007.
[FER+ 08]
Alois Ferscha, Bernadette Emsenhuber, Andreas Riener, Clemens Holzmann, Manfred Hechinger, Dominik Hochreiter, Marquart Franz, Andreas Zeidler, Marcos dos Santos Rocha, and Cornel Klein. Vibro-tactile spaceawareness. In Proceedings of the 10th Int’l Conference on Ubiquitous Computing, Ubicomp 2008, video submission, to appear, September 2008.
[FEST06]
Alois Ferscha, Bernadette Emsenhuber, Heinrich Schmitzberger, and Petra Thon. Aesthetic awareness displays. In Adjunct Proceedings of the 4th Int’l Conference on Pervasive Computing, Pervasive 2006, volume 207. Austrian Computer Society (OCG), May 2006.
Bibliography
195
[FHdSR+ 07] Alois Ferscha, Manfred Hechinger, Marcos dos Santos Rocha, Rene Mayrhofer, Andreas Zeidler, Andreas Riener, and Marquart Franz. Building flexible manufacturing systems based on Peer-its. EURASIP Journal on Embedded Systems, 2007. [FHM+ 04]
Alois Ferscha, Manfred Hechinger, Rene Mayrhofer, Marcos dos Santos Rocha, Marquart Franz, and Roy Oberhauser. Digital aura. In Proceedings of the 4th Int’l Conference on Pervasive Computing, Pervasive 2004, Advances in Pervasive Computing, pages 405–410, April 2004.
[FHO04a]
Alois Ferscha, Clemens Holzmann, and Stefan Oppl. Context-awareness for group interaction support. In Proceedings of the 2nd Int’l Workshop on Mobility Management and Wireless Access, Workshop at MobiCom 2004, pages 88–97. ACM Press, October 2004.
[FHO04b]
Alois Ferscha, Clemens Holzmann, and Stefan Oppl. Team-awareness in personalized learning environments. In Proceedings of the 3rd Int’l Conference on Mobile Learning, MLEARN 2004, pages 67–72. LSDA, July 2004.
[FHR+ 06a]
Alois Ferscha, Manfred Hechinger, Andreas Riener, Heinrich Schmitzberger, Marquart Franz, Marcos dos Santos Rocha, and Andreas Zeidler. Contextaware profiles. In Proceedings of the 2nd Int’l Conference on Autonomic and Autonomous Systems, ICAS 2006. IEEE CS Press, July 2006.
[FHR06b]
Alois Ferscha, Clemens Holzmann, and Stefan Resmerita. The key knob. In Proceedings of the 6th IEEE Workshop on Smart Appliances and Wearable Computing, Workshop at ICDCS 2006, pages 62–71. IEEE CS Press, July 2006.
[FHR+ 08]
Alois Ferscha, Manfred Hechinger, Andreas Riener, Marcos dos Santos Rocha, Andreas Zeidler, Marquart Franz, and Rene Mayrhofer. Peer-it: Stick-on solutions for networks of things. Journal of Pervasive and Mobile Computing, 4(3):448–479, 2008.
[FIB95]
George W. Fitzmaurice, Hiroshi Ishii, and William Buxton. Bricks: Laying the foundations for graspable user interfaces. In Proceedings of the Conference on Human Factors in Computing Systems, CHI 1995, pages 442–449. ACM Press, May 1995.
[Fis04]
Kenneth P. Fishkin. A taxonomy for and analysis of tangible interfaces. Personal and Ubiquitous Computing, 8(5):347–358, 2004.
[Fit93]
George W. Fitzmaurice. Situated information spaces and spatially aware palmtop computers. Communications of the ACM, 36(7):38–49, 1993.
[For97]
Kenneth D. Forbus. Qualitative reasoning. In The Computer Science and Engineering Handbook, pages 715–733. 1997.
[FR93]
Christian Freksa and Ralf Röhrig. Dimensions of qualitative spatial reasoning. In Proceedings of the IMACS Workshop on Qualitative Reasoning in Decision Technologies, QUARDET 1993, pages 483–492, June 1993.
196
Bibliography
[Fra98]
Andrew U. Frank. Formal models for cognition - taxonomy of spatial location description and frames of reference. In Spatial Cognition - An Interdisciplinary Approach to Representing and Processing Spatial Knowledge, volume 1404 of LNCS, pages 293–312. Springer, 1998.
[Fre91]
Christian Freksa. Conceptual neighborhood and its role in temporal and spatial reasoning. In Proceedings of the IMACS Workshop on Decision Support Systems and Qualitative Reasoning, pages 181–187. Elsevier, March 1991.
[Fre92a]
Christian Freksa. Temporal reasoning based on semi-intervals. Artificial Intelligence, 54(1):199–227, 1992.
[Fre92b]
Christian Freksa. Using orientation information for qualitative spatial reasoning. In Proceedings of the Int’l Conference GIS - From Space to Territory: Theories and Methods of Spatio-Temporal Reasoning, volume 639 of LNCS, pages 162–178. Springer, September 1992.
[FRH06]
Alois Ferscha, Stefan Resmerita, and Clemens Holzmann. Human computer confluence. In Proceedings of the 9th ERCIM Workshop on User Interfaces for All, UI4All 2006, volume 4397 of LNCS, pages 14–27. Springer, September 2006.
[FRHR05]
Alois Ferscha, Stefan Resmerita, Clemens Holzmann, and Martin Reichör. Orientation sensing for gesture-based interaction with smart artifacts. Computer Communications, 28(13):1552–1563, 2005.
[Gar70]
Martin Gardner. Mathematical games: The fantastic combinations of John Conway’s new solitaire game ‘Life’. Scientific American, 223(4):120–123, October 1970.
[GGH06]
Björn Gottfried, Hans W. Guesgen, and Sebastian Hübner. Spatiotemporal reasoning for smart homes. In Designing Smart Homes, The Role of Artificial Intelligence, volume 4008 of LNCS, pages 16–34. Springer, 2006.
[Goo08]
Google, Inc. KML (Keyhole Markup Language) documentation. http://code.google.com/apis/kml/documentation, 2008.
[GSB02]
Hans-Werner Gellersen, Albrecht Schmidt, and Michael Beigl. Multi-sensor context-awareness in mobile devices and smart artefacts. Mobile Networks and Applications, 7(5):341–351, 2002.
[GSG07]
Dominique Guinard, Sara Streng, and Hans Gellersen. Extending mobile devices with spatially arranged gateways to pervasive services. In Proceedings of the 3rd Int’l Workshop on Pervasive Mobile Interaction Devices, Workshop at Pervasive 2007, May 2007.
[GSSS02]
David Garlan, Dan Siewiorek, Asim Smailagic, and Peter Steenkiste. Project aura: Towards distraction-free pervasive computing. IEEE Pervasive Computing, 21(2):22–31, 2002.
Bibliography
197
[HB00]
Felix Hupfeld and Michael Beigl. Spatially aware local communication in the RAUM system. In Proceedings of the 7th Int’l Workshop on Interactive Distributed Multimedia Systems and Telecommunication Services, IDMS 2000, volume 1905 of LNCS, pages 285–296. Springer, October 2000.
[HB01]
Jeffrey Hightower and Gaetano Borriello. A survey and taxonomy of location systems for ubiquitous computing. Technical report, August 2001. Extended paper from Computer, 34(8):57-66, 2001.
[HBB02]
Jeffrey Hightower, Barry Brumitt, and Gaetano Borriello. The location stack: A layered model for location in ubiquitous computing. In Proceedings of the 4th Workshop on Mobile Computing Systems & Applications, WMCSA 2002, pages 22–28. IEEE Computer Society, June 2002.
[HCF95]
Daniel Hernández, Eliseo Clementini, and Paolino Di Felice. Qualitative distances. In Proceedings of the Int’l Conference on Spatial Information Theory: A Theoretical Basis for GIS, COSIT 1995, volume 988 of LNCS, pages 45–57. Springer, September 1995.
[HE03]
David Hales and Bruce Edmonds. Evolving social rationality for MAS using “tags”. In Proceedings of the 2nd Int’l Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS 2003, pages 497–503, New York, NY, USA, July 2003. ACM Press.
[Her94]
Daniel Hernández. Qualitative Representation of Spatial Knowledge, volume 804 of LNCS. Springer, 1994.
[HF07]
Clemens Holzmann and Alois Ferscha. Towards collective spatial awareness using binary relations. In Proceedings of the 3rd Int’l Conference on Autonomic and Autonomous Systems, ICAS 2007. IEEE CS Press, June 2007.
[HFH+ 08]
Clemens Holzmann, Alois Ferscha, Manfred Hechinger, Andreas Zeidler, Marcos dos Santos Rocha, and Marquart Franz. Using spatial abstractions in industrial environments. In Proceedings of the 4th Int’l Conference on Wireless and Mobile Communications, ICWMC 2008, pages 108–117. IEEE CS Press, July 2008.
[HG03]
Francis Heylighen and Carlos Gershenson. The meaning of self-organization in computing. IEEE Intelligent Systems, 18(4):72–75, 2003.
[HHF08]
Clemens Holzmann, Manfred Hechinger, and Alois Ferscha. Relationcentric development of spatially-aware applications. In Proceedings of the 17th IEEE Int’l Workshops on Enabling Technologies, WETICE 2008, to appear. IEEE CS Press, June 2008.
[HHS+ 02]
Andy Harter, Andy Hopper, Pete Steggles, Andy Ward, and Paul Webster. The anatomy of a context-aware application. Wireless Networks, 8(2-3):187– 197, 2002.
198 [HKG+ 05]
Bibliography
Mike Hazas, Christian Kray, Hans-Werner Gellersen, Henoc Agbota, Gerd Kortuem, and Albert Krohn. A relative positioning system for co-located mobile devices. In Proceedings of the 3rd Int’l Conference on Mobile Systems, Applications, and Services, MobiSys 2005, pages 177–190. ACM Press, June 2005.
[HMEZ+ 05] Sumi Helal, William C. Mann, Hicham El-Zabadani, Jeffrey King, Youssef Kaddoura, and Erwin Jansen. The Gator Tech Smart House: A programmable pervasive space. IEEE Computer, 38(3):50–60, 2005. [HMS+ 01]
Lars Erik Holmquist, Friedemann Mattern, Bernt Schiele, Petteri Alahuhta, Michael Beigl, and Hans-Werner Gellersen. Smart-Its Friends: A technique for users to easily establish connections between smart artefacts. In Proceedings of the 3rd Int’l Conference on Ubiquitous Computing, Ubicomp 2001, volume 2201 of LNCS, pages 116–122. Springer, September 2001.
[HN02]
Jerry R. Hobbs and Srini Narayanan. Spatial representation and reasoning. In Encyclopedia of Cognitive Science. MacMillan, London, UK, 2002.
[Hol07a]
Clemens Holzmann. Inferring and distributing spatial context. In Proceedings of the 2nd European Conference on Smart Sensing and Context, EuroSSC 2007, volume 4793 of LNCS, pages 77–92. Springer, October 2007.
[Hol07b]
Clemens Holzmann. Rule-based reasoning about qualitative spatiotemporal relations. In Proceedings of the 5th Int’l Workshop on Middleware for Pervasive and Ad-Hoc Computing, Workshop at Middleware 2007, pages 49–54. ACM Press, November 2007.
[Hor01]
Paul Horn. Autonomic computing: IBM’s perspective on the state of information technology. Technical report, Int’l Business Machines Corporation (IBM), New Orchard Road, Armonk, NY 10504, USA, October 2001.
[HR85]
Frederick Hayes-Roth. Rule-based systems. Communications of the ACM, 28(9):921–932, 1985.
[HRLF06]
Clemens Holzmann, Stefan Resmerita, Michael Leitner, and Alois Ferscha. A paradigm for orientation-based universal remote control. In Proceedings of the 3rd Int’l Workshop on the Tangible Space Initiative, Workshop at Pervasive 2006, pages 425–432, May 2006.
[Int08]
InterSense, Inc. IS-900 precision motion tracker. http://www.isense.com, 2008.
[IoD00]
National Imagery and Mapping Agency: Department of Defense. World geodetic system 1984, its definition and relationships with local geodetic systems, third edition, national geospatial-intelligence agency. http://earthinfo.nga.mil/GandG/publications/tr8350.2/wgs84fin.pdf, 2000.
[IU97]
Hiroshi Ishii and Brygg Ullmer. Tangible bits: Towards seamless interfaces between people, bits and atoms. In Proceedings of the Conference on Hu-
Bibliography
199
man Factors in Computing Systems, CHI 1997, pages 234–241. ACM Press, March 1997. [JBL+ 06]
Márk Jelasity, Özalp Babaoglu, Robert Laddaga, Radhika Nagpal, Franco Zambonelli, Emin Gün Sirer, Hakima Chaouchi, and Mikhail I. Smirnov. Interdisciplinary research: Roles for self-organization. IEEE Intelligent Systems, 21(2):50–58, 2006.
[JS02]
Changhao Jiang and Peter Steenkiste. A hybrid location model with a computable location identifier for ubiquitous computing. In Proceedings of the 4th Int’l Conference on Ubiquitous Computing, Ubicomp 2002, volume 2498 of LNCS, pages 246–263. Springer, September 2002.
[KB99]
Christian Kray and Anselm Blocher. Modeling the basic meanings of path relations. In Proceedings of the 16th Int’l Joint Conference on Artificial Intelligence, IJCAI 1999, pages 384–393. Morgan Kaufmann, July 1999.
[KBM+ 02]
Tim Kindberg, John J. Barton, Jeff Morgan, Gene Becker, Debbie Caswell, Philippe Debaty, Gita Gopal, Marcos Frid, Venky Krishnan, Howard Morris, John Schettino, Bill Serra, and Mirjana Spasojevic. People, places, things: Web presence for the real world. Mobile Networks and Applications, 7(5):365–376, 2002.
[KC03]
Jeffrey O. Kephart and David M. Chess. The vision of autonomic computing. IEEE Computer, 36(1):41–50, 2003.
[KE01]
James Kennedy and Russell C. Eberhart. Swarm Intelligence. The Morgan Kaufmann Series in Evolutionary Computation. Morgan Kaufmann, San Francisco, CA, USA, March 2001.
[KH04]
John Krumm and Ken Hinckley. The NearMe wireless proximity server. In Proceedings of the 6th Int’l Conference on Ubiquitous Computing, Ubicomp 2004, volume 3205 of LNCS, pages 283–300. Springer, September 2004.
[KHM+ 00]
John Krumm, Steve Harris, Brian Meyers, Barry Brumitt, Michael Hale, and Steve Shafer. Multi-camera multi-person tracking for EasyLiving. In Proceedings of the 3rd Int’l Workshop on Visual Surveillance, VS 2000, pages 3–10, July 2000.
[KKG05]
Gerd Kortuem, Christian Kray, and Hans Gellersen. Sensing and visualizing spatial relations of mobile devices. In Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology, UIST 2005, pages 93–102. ACM Press, October 2005.
[KKP99]
Joseph M. Kahn, Randy H. Katz, and Kristofer S.J. Pister. Next century challenges: Mobile networking for "smart dust". In Proceedings of the 5th Annual ACM/IEEE Int’l Conference on Mobile Computing and Networking, MobiCom 1999, pages 271–278. ACM Press, August 1999.
200
Bibliography
[Kla98]
Roberta L. Klatzky. Allocentric and egocentric spatial representations: Definitions, distinctions, and interconnections. In Spatial Cognition - An Interdisciplinary Approach to Representing and Processing Spatial Knowledge, volume 1404 of LNCS, pages 1–18. Springer, 1998.
[Köh03]
Christian Köhler. A proposal for a spatio-temporal qualitative query language based on primitive geometry for cognitive vision systems. Albert-Ludwigs-University Freiburg, Chair for Foundations of Artificial Intelligence, Germany, http://www.informatik.uni-freiburg.de/cogvisys/papers/proposal.pdf, 2003.
[KOY+ 05]
Naohiko Kohtake, Ryo Ohsawa, Takuro Yonezawa, Yuki Matsukura, Masayuki Iwai, Kazunori Takashio, and Hideyuki Tokuda. u-Texture: Selforganizable universal panels for creating smart surroundings. In Proceedings of the 4th Int’l Conference on Ubiquitous Computing, Ubicomp 2005, volume 3660 of LNCS, pages 19–36. Springer, September 2005.
[LCA98]
Gottfried Wilhelm Leibniz, Samuel Clarke, and Henry G. Alexander. The Leibniz-Clarke Correspondence: Together With Extracts from Newton’s Principia and Opticks. Manchester University Press, 1998.
[LCC+ 05]
Anthony LaMarca, Yatin Chawathe, Sunny Consolvo, Jeffrey Hightower, Ian E. Smith, James Scott, Timothy Sohn, James Howard, Jeff Hughes, Fred Potter, Jason Tabert, Pauline Powledge, Gaetano Borriello, and Bill N. Schilit. Place Lab: Device positioning using radio beacons in the wild. In Proceedings of the 3rd Int’l Conference on Pervasive Computing, Pervasive 2005, volume 3468 of LNCS, pages 116–133. Springer, May 2005.
[Leo98]
Ulf Leonhardt. Supporting Location-Awareness in Open Distributed Systems. PhD thesis, Imperial College London, May 1998.
[Lev96]
Stephen Levinson. Frames of reference and Molyneux’s question: Crosslinguistic evidence. In Language and Space, pages 109–169. MIT Press, Cambridge, MA, 1996.
[LM98]
Ulf Leonhardt and Jeff Magee. Multi-sensor location tracking. In Proceedings of the 4th annual ACM/IEEE Int’l Conference on Mobile Computing and Networking, MobiCom 1998, pages 203–214. ACM Press, October 1998.
[LR03]
Koen Langendoen and Niels Reijers. Distributed localization in wireless sensor networks: A quantitative comparison. Computer Networks, 43(4):500– 518, 2003.
[LSBP02]
Joshua Lifton, Deva Seetharam, Michael Broxton, and Joseph A. Paradiso. Pushpin computing system overview: A platform for distributed, embedded, ubiquitous sensor networks. In Proceedings of the 1st Int’l Conference on Pervasive Computing, Pervasive 2002, volume 2414 of LNCS, pages 139– 151. Springer, August 2002.
Bibliography
201
[LSG02]
Kristof Van Laerhoven, Albrecht Schmidt, and Hans-Werner Gellersen. Pin&Play: Networking objects through pins. In Proceedings of the 4th Int’l Conference on Ubiquitous Computing, Ubicomp 2002, volume 2498 of LNCS, pages 219–228. Springer, September 2002.
[May04]
Rene Mayrhofer. An Architecture for Context Prediction. PhD thesis, Johannes Kepler University Linz, 2004.
[May06]
R. Mayrhofer. A context authentication proxy for IPSec using spatial reference. In Proceedings of the 1st Int’l Workshop on Trustworthy Ubiquitous Computing, TwUC 2006. Austrian Computer Society (OCG), December 2006.
[MDF05]
Reinhard Moratz, Frank Dylla, and Lutz Frommberger. A relative orientation algebra with adjustable granularity. In Proceedings of the Workshop on Agents in Real-Time and Dynamic Environments, Workshop at IJCAI 2005, July 2005.
[MF00]
Reinhard Moratz and Kerstin Fischer. Cognitively adequate modelling of spatial reference in human-robot interaction. In Proceedings of the 12th IEEE Int’l Conference on Tools with Artificial Intelligence, ICTAI 2000. IEEE CS Press, November 2000.
[MFH+ 04]
Masateru Minami, Yasuhiro Fukuju, Kazuki Hirasawa, Shigeaki Yokoyama, Moriyuki Mizumachi, Hiroyuki Morikawa, and Tomonori Aoyama. DOLPHIN: A practical approach for implementing a fully distributed indoor ultrasonic positioning system. In Proceedings of the 6th Int’l Conference on Ubiquitous Computing, Ubicomp 2004, volume 3205 of LNCS, pages 347– 365. Springer, September 2004.
[Mil07]
Kevin L. Mills. A brief survey of self-organization in wireless sensor networks. Wireless Communications and Mobile Computing, 7(7):823–834, 2007.
[MMTZ06]
Marco Mamei, Ronaldo Menezes, Robert Tolksdorf, and Franco Zambonelli. Case studies for self-organization in computer science. Journal of Systems Architecture, 52(8-9):442–460, 2006.
[MOFH03]
Rene Mayrhofer, Florian Ortner, Alois Ferscha, and Manfred Hechinger. Securing passive objects in mobile ad-hoc peer-to-peer networks. In Proceedings of the 1st Int’l Workshop on Security Issues in Coordination Models, Languages, and Systems, SecCo 2003, volume 85. Elsevier, June 2003.
[Mor04]
Reinhard Moratz. Qualitative spatial reasoning about oriented points. Technical Report SFB/TR 8 Report No. 003-10/2004, University of Bremen, Bremen, Germany, October 2004.
[MRW00]
Reinhard Moratz, Jochen Renz, and Diedrich Wolter. Qualitative spatial reasoning about line segments. In Proceedings of the 14th European Conference
202
Bibliography
on Artificial Intelligence, ECAI 2000, pages 234–238. IOS Press, August 2000. [MSE+ 98]
Alexandra Musto, Klaus Stein, Andreas Eisenkolb, Kerstin Schill, and Wilfried Brauer. Generalization, segmentation, and classification of qualitative motion data. In Proceedings of the 13th European Conference on Artificial Intelligence, ECAI 1998, pages 180–184. John Wiley & Sons, August 1998.
[MSER99]
Alexandra Musto, Klaus Stein, Andreas Eisenkolb, and Thomas Röfer. Qualitative and quantitative representations of locomotion and their application in robot navigation. In IJCAI, pages 1067–1073. Morgan Kaufmann, July 1999.
[Mus99]
Alexandra Musto. On spatial reference frames in qualitative motion representation. Technical Report FKI-230-99, Institut für Informatik, Technische Universität München, 1999.
[Mus00]
Alexandra Musto. Qualitative Repräsentation von Bewegungsverläufen. PhD thesis, Technical University Munich, 2000.
[MZ03]
Marco Mamei and Franco Zambonelli. Self-organization in multi agent systems: A middleware approach. In Proceedings of the Int’l Workshop on Engineering Self-Organising Applications, Workshop at AAMAS 2003, volume 2977 of LNCS, pages 233–248. Springer, July 2003.
[MZ04]
Marco Mamei and Franco Zambonelli. Programming pervasive and mobile computing applications with the TOTA middleware. In Proceedings of the 2nd Int’l Conference on Pervasive Computing and Communications, PerCom 2004, pages 263–276. IEEE CS Press, March 2004.
[MZ05]
Marco Mamei and Franco Zambonelli. Spatial computing: the TOTA approach. In Proceedings of the Int’l Workshop on Self-Star Properties in Complex Information Systems, volume 3460 of LNCS, pages 307–324. Springer, 2005.
[MZL03]
Marco Mamei, Franco Zambonelli, and Letizia Leonardi. Tuples on the air: A middleware for context-aware computing in dynamic networks. In Proceedings of the 2nd Int’l Workshop on Mobile Computing Middleware, Workshop at ICDCS 2003. IEEE CS Press, May 2003.
[NS02]
Bernhard Nebel and Alexander Scivos. Formal properties of constraint calculi for qualitative spatial reasoning. KI, 16(4):14–18, 2002.
[PMBT01]
Nissanka B. Priyantha, Allen K.L. Miu, Hari Balakrishnan, and Seth J. Teller. The Cricket compass for context-aware mobile applications. In Proceedings of the 7th Int’l Conference on Mobile Computing and Networking, MobiCom 2001, pages 1–14. ACM Press, July 2001.
[PNF+ 08]
Mark Proctor, Michael Neale, Michael Frandsen, Sam Griffith, Edson Tirelli, Fernando Meyer, and Kris Verlaenen. Drools documentation, version 4.0.4. http://labs.jboss.com, 2008.
Bibliography
[Pol08]
203
Polulu Robotics and Electronics. Polulu IR Beacon. http://www.pololu.com, 2008.
[RAMC+ 04] Anand Ranganathan, Jalal Al-Muhtadi, Shiva Chetan, Roy H. Campbell, and M. Dennis Mickunas. Middlewhere: A middleware for location awareness in ubiquitous computing applications. In Proceedings of the 5th Int’l Middleware Conference, Middleware 2004, volume 3231 of LNCS, pages 397–416. Springer, October 2004. [RCC92]
David A. Randell, Zhan Cui, and Anthony G. Cohn. A spatial logic based on regions and connection. In Proceedings of the 3rd Int’l Conference on Principles of Knowledge Representation and Reasoning, KR 1992, pages 165–176. Morgan Kaufmann, October 1992.
[RM04]
Jochen Renz and Debasis Mitra. Qualitative direction calculi with arbitrary granularity. In Proceedings of the 8th Pacific Rim Int’l Conference on Artificial Intelligence, PRICAI 2004, volume 3157 of LNCS, pages 65–74. Springer, August 2004.
[Rnc08]
RN-Control microcontroller board 1.4 Mega32. http://www.robotikhardware.de, 2008.
[Sat01]
Mahadev Satyanarayanan. Pervasive computing: Vision and challenges. IEEE Personal Communications, 8(4):10–17, August 2001.
[SAW94]
Bill Schilit, Norman Adams, and Roy Want. Context-aware computing applications. In Proceedings of the 1st Workshop on Mobile Computing Systems and Applications, WMCSA 1994, pages 85–94. IEEE CS Press, December 1994.
[SBG99]
Albrecht Schmidt, Michael Beigl, and Hans-Werner Gellersen. There is more to context than location. Computers & Graphics, 23(6):893–901, 1999.
[Sch95]
William Noah Schilit. A System Architecture for Context-Aware Computing. PhD thesis, Columbia University, May 1995.
[Sch02]
Albrecht Schmidt. Ubiquitous Computing - Computing in Context. PhD thesis, Lancaster University, November 2002.
[SDA99]
Daniel Salber, Anind K. Dey, and Gregory D. Abowd. The context toolkit: Aiding the development of context-enabled applications. In Proceedings of the Conference on Human factors in computing systems, CHI 1999, pages 434–441. ACM Press, May 1999.
[SFH+ 03]
Giovanna Di Marzo Serugendo, Noria Foukia, Salima Hassas, Anthony Karageorgos, Soraya Kouadri Mostéfaoui, Omer F. Rana, Mihaela Ulieru, Paul Valckenaers, and Chris van Aart. Self-organisation: Paradigms and applications. In Proceedings of the Workshop on Engineering Self-Organising Applications, ESOA 2003, volume 2977 of LNCS, pages 1–19. Springer, July 2003.
204
[SG02]
[SGKK04]
[Sha96]
[SKGK04]
[SLB+ 03]
[SN04]
[Ten05]
[TM03]
[TTH+ 04]
[TY04]
Bibliography
João Pedro Sousa and David Garlan. Aura: An architectural framework for user mobility in ubiquitous computing environments. In Proceedings of the 3rd Working IEEE/IFIP Conference on Software Architecture, WICSA 2002, volume 224, pages 29–43. Kluwer Academic Publishers, August 2002. Martin Strohbach, Hans-Werner Gellersen, Gerd Kortuem, and Christian Kray. Cooperative artefacts: Assessing real world situations with embedded technology. In Proceedings of the 6th Int’l Conference on Ubiquitous Computing, Ubicomp 2004, volume 3205 of LNCS, pages 250–267. Springer, September 2004. Jayant Sharman. Integrated Spatial Reasoning in Geographic Information Systems: Combining Topology and Direction. PhD thesis, University of Maine, May 1996. Martin Strohbach, Gerd Kortuem, Hans-Werner Gellersen, and Christian Kray. Using cooperative artefacts as basis for activity recognition. In Proceedings of the 2nd European Symposium on Ambient Intelligence, EUSAI 2004, volume 3295 of LNCS, pages 49–60. Springer, November 2004. Bill N. Schilit, Anthony LaMarca, Gaetano Borriello, William G. Griswold, David W. McDonald, Edward D. Lazowska, Anand Balachandran, Jason I. Hong, and Vaughn Iverson. Challenge: Ubiquitous location-aware computing and the “Place Lab” initiative. In Proceedings of the 1st Int’l Workshop on Wireless Mobile Applications and Services on WLAN Hotspots, WMASH 2003, pages 29–35. ACM Press, September 2003. Alexander Scivos and Bernhard Nebel. The finest of its class: The natural point-based ternary calculus for qualitative spatial reasoning. In Proceedings of the 4th Int’l Conference on Spatial Cognition 2004: Reasoning, Action, Interaction, volume 3343 of LNCS, pages 283–303. Springer, October 2004. Thora Tenbrink. Semantics and application of spatial dimensional terms in English and German. Technical Report SFB/TR 8 Report No. 004-03/2005, University of Bremen, Germany, March 2005. Thora Tenbrink and Reinhard Moratz. Group-based spatial reference in linguistic human-robot interaction. In Proceedings of the European Cognitive Science Conference, EuroCogSci 2003, pages 325–330, 2003. Tsutomu Terada, Masahiko Tsukamoto, Keisuke Hayakawa, Tomoki Yoshihisa, Yasue Kishino, Atsushi Kashitani, and Shojiro Nishio. Ubiquitous chip: A rule-based I/O control device for ubiquitous computing. In Proceedings of the 2nd Int’l Conference on Pervasive Computing, Pervasive 2004, volume 3001 of LNCS, pages 238–253. Springer, April 2004. Koji Tsukada and Michiaki Yasumura. Activebelt: Belt-type wearable tactile display for directional navigation. In Proceedings of the 6th Int’l Conference on Ubiquitous Computing, Ubicomp 2004, volume 3205 of LNCS, pages 384–399. Springer, September 2004.
Bibliography
205
[Uni04]
University of Illinois at Urbana-Champaign. sim.org, 2004.
J-Sim 1.3.
http://www.j-
[VK86]
Marc Vilain and Henry Kautz. Constraint propagation algorithms for temporal reasoning. In Proceedings of the 5th National Conference on Artificial Intelligence, AAAI 1986, pages 377–382. Morgan Kaufmann, August 1986.
[Vog02]
Simon Vogl. Coordination of Users and Services via Wall Interfaces. PhD thesis, Johannes Kepler University Linz, 2002.
[VPKdL08]
Alex Varshavsky, Denis Pankratov, John Krumm, and Eyal de Lara. Calibree: Calibration-free localization using relative distance estimations. In Proceedings of the 6th Int’l Conference on Pervasive Computing, Pervasive 2008, LNCS. Springer, May 2008.
[Wan04]
Yingxu Wang. On autonomous computing and cognitive processes. In Proceedings of the 3rd Int’l Conference on Cognitive Informatics, ICCI 2004, pages 3–4. IEEE CS Press, August 2004.
[WC02]
Brad Williams and Tracy Camp. Comparison of broadcasting techniques for mobile ad hoc networks. In Proceedings of the 3rd ACM Int’l Symposium on Mobile Ad Hoc Networking and Computing, MobiHoc 2002, pages 194–205. ACM Press, 2002.
[Wei91]
Mark Weiser. The computer for the 21st century. Scientific American, 265(3):94–104, 1991.
[WFDW07]
Jan Oliver Wallgrün, Lutz Frommberger, Frank Dylla, and Diedrich Wolter. SparQ user manual v0.7. Technical report, University of Bremen, Germany, July 2007.
[WH04]
Tom De Wolf and Tom Holvoet. Emergence versus self-organisation: Different concepts but promising when combined. In Proceedings of the Workshop on Engineering Self-Organising Applications, ESOA 2004, volume 3464 of LNCS, pages 1–15. Springer, July 2004.
[WHFG92]
Roy Want, Andy Hopper, Veronica Falcao, and Jonathan Gibbons. The active badge location system. ACM Transactions on Information Systems, 10(1):91–102, January 1992.
[WLLP01]
Brett Warneke, Matt Last, Brian Liebowitz, and Kristofer S.J. Pister. Smart dust: Communicating with a cubic-millimeter computer. IEEE Computer, 34(1):44–51, 2001.
[WSA+ 95]
Roy Want, Bill Schilit, Norman Adams, Rich Gold, Karin Petersen, John Ellis, David Goldberg, and Mark Weiser. The PARCTAB ubiquitous computing experiment. Technical Report CSL-95-1, Xerox Palo Alto Research Center, March 1995.
[Zam04]
Franco Zambonelli. Spatial computing and self-organization. In Proceedings of the 13th IEEE Int’l Workshops on Enabling Technologies, WETICE 2004, pages 4–8. IEEE CS Press, June 2004.
206
Bibliography
[ZF96]
Kai Zimmermann and Christian Freksa. Qualitative spatial reasoning using orientation, distance, and path knowledge. Applied Intelligence, 6(1):49–58, 1996.
[ZGMT04]
Franco Zambonelli, Marie Pierre Gleizes, Marco Mamei, and Robert Tolksdorf. Spray computers: Frontiers of self-organization for pervasive computing. In Proceedings of the 13th IEEE Int’l Workshops on Enabling Technologies, WETICE 2004, pages 403–408. IEEE CS Press, June 2004.
[ZM04]
Franco Zambonelli and Marco Mamei. Spatial computing: An emerging paradigm for autonomic computing and communication. In Proceedings of the 1st Int’l IFIP Workshop on Autonomic Communication, WAC 2004, volume 3457 of LNCS, pages 44–57. Springer, October 2004.
